Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add raw protobuf kafka support #4527

Closed
devinrsmith opened this issue Sep 20, 2023 · 2 comments
Closed

Add raw protobuf kafka support #4527

devinrsmith opened this issue Sep 20, 2023 · 2 comments
Assignees
Labels
Milestone

Comments

@devinrsmith
Copy link
Member

The existing kafka protobuf feature relies on the schema registry and the kafka protobuf serdes protocol.

Some use cases dictate the parsing of raw protobuf from a topic - ie, a topic that does not have the magic byte, or the 4 byte schema id. It should be possible to add efficient DH support for raw parsing based on a Descriptor:

protobuf_spec(
    ...
    schema_message_name = "com.example.MyMessage",
    type = "local" # or "serdes" (the default), or "registry"
)

serdes would be the default, and would mean the schema exists on the registry and the topic is the "serdes" protocol
local would mean the schema exists on the classpath and the topic is a "raw" protocol
registry would mean the schema exists on the registry and the topic is a "raw" protocol

@devinrsmith devinrsmith added feature request New feature or request kafka labels Sep 20, 2023
@devinrsmith devinrsmith added this to the Backlog milestone Sep 20, 2023
@devinrsmith devinrsmith self-assigned this Sep 20, 2023
@devinrsmith
Copy link
Member Author

It might also be reasonable to add "local" support, or point-in-time "registry" support, even if the protocol is really serdes. This might allow DH to be more efficient and skip the confluent io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer impl.

@devinrsmith
Copy link
Member Author

Fixed by #4585

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant