Inhibition rules and "spammy" alerts

[PabloPie]

Hey,

This is not not really a request but more a topic I'd like to have some opinions on.

So basically, Pyrra right now creates 4 Prometheus alerts for ErrorBudgetBurn with different criticalities and exhaustions. Apart from the fact that the lack of the exhaustion label (coming hopefully in v0.7) makes the alerts with same criticality confusing to understand when they are firing, the fact that all 4 alerts can be firing at the same time can make the notification channels spammy (especially with a receiver like Slack).

My thought process is that, if a critical ErrorBudgetBurn with a short exhaustion is firing, any other alert (critical or warning) is pointless until that one is resolved. For that reason, we put in place some inhibitRules, silencing less critical alerts when more important ones are firing. If my logic is right, that's maybe something Pyrra itself could create together with the PrometheusRules.

I was just wondering if anyone else had "problems" with spammy alerts and notifications and how they deal with it, especially in the context of Slack. I want to provide clear, and concise alerts for our developers, and I feel like I am failing a bit at that 😅