Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka protobuf consumer support #4375

Merged
merged 32 commits into from
Sep 7, 2023
Merged

Commits on Aug 24, 2023

  1. Kafka protobuf consumer support

    This adds support for kafka serdes-protobuf deserialization via KafkaTools#protobufSpec.
    
    There is an extensive amount of unit testing, both in the form of "how does this message structure gets parsed into
    column types" and "how do schema changes get adapted into the same table".
    
    While this feature is primarily targetted for use via KafkaTools, the underlying extensions-protobuf library is
    generalized to work in other cases as well. For example, it would work just as well at parsing protobuf messages via
    gRPC or websockets (at which point, it could easily be adapted into a blink table).
    
    The implementation is underpinned by a set a functional interfaces for extracting primitives or Objects from a
    generic type; in the case of protobuf, the generic type is com.google.protobuf.Message. Coupled with functional
    composition, these interfaces provide useful primitives for fluently expressing and executing parsing expressions in
    a reusable way.
    
    See https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/serdes-protobuf.html
    
    Fixes deephaven#3832
    devinrsmith committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    543da37 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e64b976 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2023

  1. Add schema_message_name

    devinrsmith committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    d641a30 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. Configuration menu
    Copy the full SHA
    911189a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6f0d04b View commit details
    Browse the repository at this point in the history
  3. f

    devinrsmith committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    e4c9ab9 View commit details
    Browse the repository at this point in the history
  4. Rename to ToX function

    devinrsmith committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    513e7cf View commit details
    Browse the repository at this point in the history
  5. Rename mapToX

    devinrsmith committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    1abe406 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    0a5687a View commit details
    Browse the repository at this point in the history
  7. Remove equals/hashcode

    devinrsmith committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    73c0624 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    99ba1bf View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    0ddbda0 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    4114db4 View commit details
    Browse the repository at this point in the history
  11. Update javadocs

    devinrsmith committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    05dc323 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    8c6f86c View commit details
    Browse the repository at this point in the history
  13. remove unused

    devinrsmith committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    011032d View commit details
    Browse the repository at this point in the history
  14. javadoc

    devinrsmith committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    f538b08 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. box

    devinrsmith committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    5e9d7b9 View commit details
    Browse the repository at this point in the history
  2. more fixes

    devinrsmith committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    0a9182d View commit details
    Browse the repository at this point in the history
  3. Add canonical parsers

    devinrsmith committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    dc72866 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f569254 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    1c4b1cd View commit details
    Browse the repository at this point in the history
  6. Comments

    devinrsmith committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    0e9fce2 View commit details
    Browse the repository at this point in the history
  7. python comment changes

    devinrsmith committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    04f09b7 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    c45a57d View commit details
    Browse the repository at this point in the history
  9. ToXFunction testing

    devinrsmith committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    b7a1538 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    4a975f2 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    56b713d View commit details
    Browse the repository at this point in the history
  12. final review points

    devinrsmith committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    3302e23 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Kafka consume test

    devinrsmith committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    bd93a6c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    66bdec8 View commit details
    Browse the repository at this point in the history
  3. formatting

    devinrsmith committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    7baad9f View commit details
    Browse the repository at this point in the history