Non-unique metrics exposed on per-object metrics endpoint

[vCra]

Describe the bug

When scraping the /metrics/per-object endpoint, we occasionally see duplicated metrics returned from RMQ

These metrics sometimes have the correct labels, and sometimes do not. This issue doesn't affect the entire cluster - it's usually just one node

This is causing issues with ingestion into prometheus, as it sees these identical metrics as out-of-order, which generated errors

Metrics which seem to be affected

• rabbitmq_raft_term_total
• rabbitmq_raft_log_snapshot_index
• rabbitmq_raft_log_last_applied_index
• rabbitmq_raft_log_commit_index
• rabbitmq_raft_log_last_written_index
• rabbitmq_raft_entry_commit_latency_seconds

Reproduction steps

We are struggling to reproduce this

Expected behavior

All the metrics should have unique labels

Additional context

No response

[michaelklishin]

@vCra how do you define "an incorrect label"? I can be a race condition between the table that stores individual metric values, e.g. it does not necessarily require the keys to be unique.

But that would not result in any "incorrect" labels.

[michaelklishin]

Another hypothesis we have: can those queues be gone by now but remain in the metrics table? Do you experience much quorum queue churn in your system?

[vCra]

We do have an MQTT listener enabled, but we don't use it at all - we just use AMQP
We don't automatically churn queues or anything - maybe manually remove 1-2 a year at most

[dbereza-machinify]

Hello, was wondering if there are any updates on this?

We are also seeing a similar issue crop up on RabbitMQ 3.13.6 for the /metrics endpoint where duplicate metrics are being outputted, the same six listed in the initial post:

rabbitmq_raft_term_total
rabbitmq_raft_log_snapshot_index
rabbitmq_raft_log_last_applied_index
rabbitmq_raft_log_commit_index
rabbitmq_raft_log_last_written_index
rabbitmq_raft_entry_commit_latency_seconds

Example duplicate output:

$ curl http://172.60.18.168:15692/metrics | grep "rabbitmq_raft_term_total "
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# TYPE rabbitmq_raft_term_total counter
# HELP rabbitmq_raft_term_total Current Raft term number
rabbitmq_raft_term_total 259
rabbitmq_raft_term_total 40

Similarly, we don't have the MQTT plugin enabled nor have any real queue churn, although we do pretty much exclusively use quorum and stream queues.

[michaelklishin]

There are no updates except for one: we know when this duplicate metric can be returned by the HTTP API: when a quorum queue is in the process of transferring leadership, so for a short period of time both nodes will have local leader metrics.

I don't see how this can affect the Prometheus /metrics endpoint which returns data from a single node only.

Those who are interested in getting to the bottom of this are welcome to investigate. This is open source software after all.

[vCra]

We just ended up dropping raft related metrics whilst scraping, as we didn't have any need for them

          "metricRelabelings" = [
            {
              "sourceLabels" = [
                "__name__"
              ]
              "action" = "drop"
              "regex"  = "rabbitmq_raft_term_total|rabbitmq_raft_log_snapshot_index|rabbitmq_raft_log_last_applied_index|rabbitmq_raft_log_commit_index|rabbitmq_raft_log_last_written_index|rabbitmq_raft_entry_commit_latency_seconds"
            }
          ]