Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(escalating-v2): Emit escalating metrics #52774

Merged
merged 5 commits into from
Jul 20, 2023
Merged

feat(escalating-v2): Emit escalating metrics #52774

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4d9ab22
Emit escalating metrics
snigdhas Jul 12, 2023
40329fe
Merge branch 'master' into snigdha/metrics-emit
snigdhas Jul 13, 2023
58ae535
Merge branch 'master' into snigdha/metrics-emit
snigdhas Jul 18, 2023
5999cf8
Set tag value as string
snigdhas Jul 19, 2023
c0ef368
Close metrics backend
snigdhas Jul 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions src/sentry/tasks/post_process.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
from sentry.issues.grouptype import GroupCategory
from sentry.issues.issue_occurrence import IssueOccurrence
from sentry.killswitches import killswitch_matches_context
from sentry.sentry_metrics.kafka import KafkaMetricsBackend
Copy link
Contributor

@onewland onewland Jul 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ayirr7 I think we should hide which backend gets used from the caller.

it should be something like

from sentry.sentry_metrics import get_sentry_metrics_backend

# today this returns a KafkaMetricsBackend, but later it
# should be configurable from somewhere outside the importing file
metrics_backend = get_sentry_metrics_backend()

This also makes it easier to only have one connection (pool) per process

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sentry has the service delegator / LazyServiceWrapper abstraction which is commonly used for this

from sentry.sentry_metrics.use_case_id_registry import UseCaseID
from sentry.signals import event_processed, issue_unignored, transaction_processed
from sentry.tasks.base import instrumented_task
from sentry.utils import metrics
Expand Down Expand Up @@ -115,6 +117,23 @@ def _capture_event_stats(event: Event) -> None:
metrics.timing("events.size.data", event.size, tags=tags)


def _update_escalating_metrics(event: Event) -> None:
"""
Update metrics for escalating issues when an event is processed.
"""
metrics_backend = KafkaMetricsBackend()
metrics_backend.counter(
UseCaseID.ESCALATING_ISSUES,
org_id=event.project.organization_id,
project_id=event.project.id,
metric_name="event_ingested",
value=1,
Copy link
Member Author

@snigdhas snigdhas Jul 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ayirr7 this should get aggregated as part of the counter right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that's correct

tags={"group": str(event.group_id)},
unit=None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is unit?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's the mri_unit in this template ":/@". I've mostly seen units of time (millisecond, second) here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ayirr7 what should this be set to for our needs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unit for the metric you are emitting (whatever you are measuring). If there is no unit then None is fine

)
metrics_backend.close()


def _capture_group_stats(job: PostProcessJob) -> None:
event = job["event"]
if not job["group_state"]["is_new"] or not should_write_event_stats(event):
Expand Down Expand Up @@ -571,6 +590,7 @@ def get_event_raise_exception() -> Event:
update_event_groups(event, group_states)
bind_organization_context(event.project.organization)
_capture_event_stats(event)
_update_escalating_metrics(event)

group_events: Mapping[int, GroupEvent] = {
ge.group_id: ge for ge in list(event.build_group_events())
Expand Down
Loading