Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extensible Kafka KeyOrValueSpecs #4346

Closed
cpwright opened this issue Aug 21, 2023 · 2 comments · Fixed by #4648
Closed

Extensible Kafka KeyOrValueSpecs #4346

cpwright opened this issue Aug 21, 2023 · 2 comments · Fixed by #4648
Assignees
@rcaudy @devinrsmith
Labels
feature request New feature or request kafka
Milestone
Backlog

Comments

@cpwright
Copy link
Contributor

As a data administrator/developer, I want to be able to substitute my own key or value specifications instead of ones that Deephaven has built so that I can customize my ingestion for new formats or as a developer experiment with alternative processors for existing formats.

The current architecture has switch statements that are not conducive to extending the ingester framework at runtime.
@cpwright cpwright added feature request New feature or request triage labels Aug 21, 2023
@devinrsmith devinrsmith self-assigned this Aug 25, 2023
@devinrsmith devinrsmith added this to the Backlog milestone Aug 25, 2023
@devinrsmith
Copy link
Member

Agreed. It's rather cumbersome the way it is now. During recent refactorings, the switch statements were removed, and the implementations were made cleaner. That said, it's really an internal interface still, while it doesn't need to be.

As part of the work in #4375, we'll be a lot closer to achieving this reality.

@cpwright
Copy link
Contributor Author

I think the change in #4248 might be close enough in terms of code. The protobuf spec was added without changing post-refactor. However, the io.deephaven.kafka.KafkaTools.Consume.KeyOrValueSpec is undocumented and the SchemaProviderProvider interface is private (and undocumented) which likely makes it hard to understand how one is meant to implement a KeyOrValueSpec that extends the SchemaProviderProvider.

@rcaudy rcaudy added kafka and removed triage labels Aug 30, 2023
devinrsmith added a commit to devinrsmith/deephaven-core that referenced this issue Oct 16, 2023
@devinrsmith
Add KafkaTools.Consume#objectProcessorSpec for custom KeyOrValueSpecs
b8bb1f3 
This introduces a new public interface, ObjectProcessor, that provides a means for end-users to create custom KeyOrValueSpec for use with Kafka ingestion. The interface isn't specific to kafka though, and should be extensible to other domains in the future.

The existing KeyOrValue specs are left unchanged; but arguably, there may be value in porting our existing implementations over to it in the future.

Fixes deephaven#4346
@devinrsmith devinrsmith mentioned this issue Oct 16, 2023
Merged
devinrsmith added a commit that referenced this issue Oct 18, 2023
@devinrsmith
Add KafkaTools.Consume#objectProcessorSpec for custom KeyOrValueSpecs (…
e0e968e 
…#4648)

This introduces a new public interface, ObjectProcessor, that provides a means for end-users to create custom KeyOrValueSpec for use with Kafka ingestion. The interface isn't specific to kafka though, and should be extensible to other domains in the future.

The existing KeyOrValue specs are left unchanged; but arguably, there may be value in porting our existing implementations over to it in the future.

Fixes #4346
@devinrsmith devinrsmith mentioned this issue Mar 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rcaudy rcaudy

@devinrsmith devinrsmith

Labels
feature request New feature or request kafka
Projects
None yet
Milestone
Backlog
Development

Successfully merging a pull request may close this issue.

Add KafkaTools.Consume#objectProcessorSpec for custom KeyOrValueSpecs devinrsmith/deephaven-core
3 participants
@rcaudy @devinrsmith @cpwright