Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Apoursam/spark kafka input #1272

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Apoursam/spark kafka input #1272

wants to merge 12 commits into from

Commits on Nov 13, 2024

  1. [vpj] Kafka ingestion into rows of a data frame via Spark Table API

    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    2a2f4c3 View commit details
    Browse the repository at this point in the history

  2. Spotbug fix1

    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    09cdd59 View commit details
    Browse the repository at this point in the history

  3. The use of VeniceProperties as the method for passing configuration i… 

    …s deeply entrenched in the system, adopting this approach for KIF repush.

formatting fixes / datatype fixes / spotbug fixes
    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    56cac6d View commit details
    Browse the repository at this point in the history

  4. Unified the usage of property bags across the rest of the code.

    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    4fc913e View commit details
    Browse the repository at this point in the history

  5. Protect the system from NPEs resulting from calling addAll on a consu… 

    …merBugger.get that comes back as null .
    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    072ee4e View commit details
    Browse the repository at this point in the history

  6. Better offset math to deal with off-by-ones happening throughout .

    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b1870d9 View commit details
    Browse the repository at this point in the history

  7. Rearranged pubsub test.

    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    68cf01c View commit details
    Browse the repository at this point in the history

  8. Removed un-needed clients they are now embedded in the partition read… 

    …er and ...
    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    09f4b01 View commit details
    Browse the repository at this point in the history

  9. Added counter for delivered records, can help with debugging future m… 

    …issing records situations.
    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    1d7bebd View commit details
    Browse the repository at this point in the history

  10. Removed un-needed tests and comments.

    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    9f72cbb View commit details
    Browse the repository at this point in the history

  11. Explicit imports per venice coding style.

    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    2bc56ee View commit details
    Browse the repository at this point in the history

  12. Added method to expose stats from the record reader and related tests… 

    …. Polished logging.
    @eldernewborn
    eldernewborn committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    57606ec View commit details
    Browse the repository at this point in the history