Classic mirrored queues repeatedly log errors

[NickRandal]

Describe the bug

My log is filling Gigabytes worth of text with

2023-07-26 17:12:13.235000-04:00 [error] <0.4832.0> Discarding message {'$gen_call',{<0.4832.0>,#Ref<0.3650537411.2880438274.115712>},stat} from <0.4832.0> to <0.28261.1102> in an old incarnation (1687742798) of this node (1690395957)

At some point something bad happened and certain queues became unsynchronised. The problem is that I cannot affect any change on them either through web management or API. I have tried every option but all those operations either hang the UI or timeout through the API with no change.

• ErlangFullVersion | Erlang/OTP 23 [erts-11.2.2.10] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]
• 3.9.13
• 3 nodes with classic mirror queues, most are functional

Reproduction steps

I don't know what caused it in the first place

Expected behavior

Some way to resolve the issue. Resynching the queues would be preferable but deleting them would be acceptable at this point.

Additional context

No response

[michaelklishin]

Please use Discussions for questions.

There is rabbitmqctl delete_queue which, at least in 3.11 and 3.12, handles CMQs without an electable leader.

Classic mirrored queues are deprecated and scheduled to be removed in early 2024 in RabbitMQ 4.0.
Their flaws are unfixable and it has been announced a few years ago. Use Quorum queues and/or streams. There are migration guidelines for 3.11 and 3.12. Or use non-mirrored classic queues where appropriate.

RabbitMQ 3.9 goes completely EOL in a few days. Please upgrade to 3.12 (3.9 => latest 3.9 => latest 3.10 => latest 3.11 => 3.12, enabling all feature flags in the process, or using a Blue/Green deployment).

There are many more reasons to upgrade.

[NickRandal]

Thank you!!! This worked for 3.9. The only hitch was that the documentation for this command did not list --vhost= which took some time searching for.

[michaelklishin]

--vhost is a global option accepted by many commands. It is documented in the help message at the bottom.

[NickRandal]

I thought you might say that. I will say that it is nowhere to be found on https://www.rabbitmq.com/rabbitmqctl.8.html and was not reported as an option by the command when I ran it. It only said queue not found under vhost "/"

[michaelklishin]

It's mentioned under the General Options below but fair enough, #10633 adds a more explicit usage description for delete_queue.