Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create rc-processing-transaction #79200

Merged
merged 67 commits into from
Oct 22, 2024
Merged

create rc-processing-transaction #79200

merged 67 commits into from
Oct 22, 2024

Conversation

kneeyo1
Copy link
Contributor

@kneeyo1 kneeyo1 commented Oct 16, 2024

This creates rc processing transactions redis cluster specifically for transactions. For now as part of the rollout, this only initializes the new eventstore and refactors a lot of the code so that most of the processing and post processing is aware of the event type.

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 16, 2024
@lynnagara
Copy link
Member

lynnagara commented Oct 16, 2024

there's a lot of places in code where you need to implement the dual writing, have you considered putting the dual writing logic in one place only inside the transaction_processing_store implementation, so that none of the callsites need to change when rolling this out?

Co-authored-by: Lyn Nagara <1779792+lynnagara@users.noreply.github.com>
Co-authored-by: Lyn Nagara <1779792+lynnagara@users.noreply.github.com>
@lynnagara
Copy link
Member

i think there are a few more places in the codebase where this is needed.. e.g.

unprocessed = event_processing_store.get(
cache_key_for_event({"project": event.project_id, "event_id": event.event_id}),
unprocessed=True,
)

can you check the whole codebase?

@kneeyo1
Copy link
Contributor Author

kneeyo1 commented Oct 17, 2024

i think there are a few more places in the codebase where this is needed.. e.g.

unprocessed = event_processing_store.get(
cache_key_for_event({"project": event.project_id, "event_id": event.event_id}),
unprocessed=True,
)

can you check the whole codebase?

that is from the _nodestore_save_many function which checks first to ensure the event type is not a transaction

if event.get_event_type() not in ("transaction", "generic") and job["groups"]:

The rest of the references in the codebase are test cases (and reprocessing, which I'm not sure if it needs to be updated) but I think they don't need to know about the split

@kneeyo1 kneeyo1 removed the request for review from a team October 22, 2024 03:47
src/sentry/eventstream/base.py Outdated Show resolved Hide resolved
src/sentry/eventstream/kafka/backend.py Outdated Show resolved Hide resolved
src/sentry/eventstream/snuba.py Outdated Show resolved Hide resolved
@@ -0,0 +1,8 @@
class EventStreamEventType:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the reason was related to celery task pickling. We could put the string version of the enum values into task messages though.

@kneeyo1 kneeyo1 merged commit cb14959 into master Oct 22, 2024
49 checks passed
@kneeyo1 kneeyo1 deleted the rc-processing-split-pt1 branch October 22, 2024 18:41
@bmckerry bmckerry added the Trigger: Revert Add to a merged PR to revert it (skips CI) label Oct 22, 2024
@getsentry-bot
Copy link
Contributor

PR reverted: 56b7027

getsentry-bot added a commit that referenced this pull request Oct 22, 2024
This reverts commit cb14959.

Co-authored-by: bmckerry <110857332+bmckerry@users.noreply.github.com>
cmanallen pushed a commit that referenced this pull request Oct 23, 2024
This creates rc processing transactions redis cluster specifically for
transactions. For now as part of the rollout, this only initializes the
new eventstore and refactors a lot of the code so that most of the
processing and post processing is aware of the event type.
cmanallen pushed a commit that referenced this pull request Oct 23, 2024
This reverts commit cb14959.

Co-authored-by: bmckerry <110857332+bmckerry@users.noreply.github.com>
kneeyo1 added a commit that referenced this pull request Oct 23, 2024
repeat of #79200  
This splits up that PR so that we change `save_event` first before the
kafka consumers.

---------

Co-authored-by: Mark Story <mark@mark-story.com>
kneeyo1 added a commit that referenced this pull request Oct 24, 2024
Part of splitting up the event_processing store. see #79200
kneeyo1 added a commit that referenced this pull request Oct 24, 2024
Post Process is unaware of the event type, this makes picking the
correct eventstore harder. This is part of #79200
@github-actions github-actions bot locked and limited conversation to collaborators Nov 7, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Scope: Backend Automatically applied to PRs that change backend components Scope: Frontend Automatically applied to PRs that change frontend components Trigger: Revert Add to a merged PR to revert it (skips CI)
Projects
None yet
Development

Successfully merging this pull request may close these issues.