[Suggestion] Extending rabbit_alarms and displaying in Management Plugin UI

[aaron-seo]

RabbitMQ series

3.13.x

Operating system (distribution) used

Linux

How is RabbitMQ deployed?

Other

What would you like to suggest for a future version of RabbitMQ?

Hi team,

We're interested in contributing improvements to observability for limits (connection_max, channel_max_per_node, etc), and one idea I had was to extend the rabbit_alarms module beyond FD, memory, and disk space alarms and use this as a unified alarm module for limit breaches as well (as of now, these are mostly error messages).

On top of that, I think it'd be useful to display these alarm states in the Management Plugin UI to encourage observability of these limits. I'm thinking it'd be similar to the FD, memory, and disk space UI we already have, but dedicate a separate section (in the Overview page or elsewhere) for all the limit breach alarms as well.

Would appreciate any thoughts on this idea before I dig deeper into it!

This can probably be configured with Prometheus alarms, but wanted to build a built-in, no-dependency option, too (well, other than the management plugin).

[michaelklishin]

Alarms in RabbitMQ are currently resource alarms, that is, it is a big deal when you run out of disk space, memory or file handles. So connections that publish messages or new connections are temporarily blocked.

Exceeding a limit such as connection_max is not a completely comparable situation, although it can be about as disruptive.

We potentially can extend the idea to two types of alarms: resource alarms and limit alarms. Resource
alarms then will work as they work today, at least for now. Limit alarms will be tracked, logged
and displayed as UI warnings. They should also be exposed via the Prometheus API endpoint to support different alerting options.

[aaron-seo]

Ah okay I didn't consider that rabbit_alarm also does this behavior: "So connections that publish messages or new connections are temporarily blocked."

In that case, I agree we should separate the idea to two types of alarms as suggested: resource and limit alarms.

[lhoguin]

See the alarms option in Ranch for a way to handle some of that: https://ninenines.eu/docs/en/ranch/2.1/manual/ranch/

(just noticed the treshold typo, whoops!)