Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: New ingestion consumer #27668

Draft
wants to merge 20 commits into
base: feat/producer-refactor
Choose a base branch
from

Conversation

benjackwhite
Copy link
Contributor

@benjackwhite benjackwhite commented Jan 18, 2025

Problem

This PR is the first in a few steps to start refactoring the plugin server to be much cleaner and more minimal

Changes

  • New consumer based of the same class based consumer for cdp
  • Essentially does the same thing as the old each-batch-ingestion process but cleaner and removing a lot of complex code paths
  • Changes to be much more config driven - this allows us to run multiple version of exactly the same consumer just with different source / destination topics and consumer group IDs
  • Adds a new mode for local running (not default) so that we can test all ingestion consumers this way - this will become the default once we release and remove the older options
  • Tests to cover most of this

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

@benjackwhite benjackwhite changed the base branch from master to feat/producer-refactor January 19, 2025 10:46
@benjackwhite benjackwhite changed the title wip: New ingestion consumer feat: New ingestion consumer Jan 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant