HA mirroring policy was not applied after federation was turned off

[timur-ND]

Describe the bug

Hi. We are facing a problem with the federation. There is a cluster of 5 nodes and another rabbitmq instance was connected through the federation. After turning off the federation (deleting Federation Upstreams and policy), the queues (type of queues is classic) that were transferred through the federation were unable to apply the mirroring (ha-all) policy between the cluster nodes. The queue had one master and 0 active replicas, and after the startup of one of the nodes (not the master), it appeared in the not syncronised status. Command rabbitmqctl sync_queue -p <vhost> <queue> returned that queue not mirrored. But policy is applied for all queues in vhost.
In the log, I found messages only like this:

2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> [1,<0.148187310.1>,<0.148188328.1>,<0.148187310.1>, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> <<"ip:39599 -> another_ip:5671">>, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> rabbit_framing_amqp_0_9_1, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> {user,<<"events">>,[], 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> [{rabbit_auth_backend_internal, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> #Fun<rabbit_auth_backend_internal.3.35421723>}]}, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> <<"events">>, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> [{<<"consumer_cancel_notify">>,bool,true}, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> {<<"connection.blocked">>,bool,true}, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> {<<"authentication_failure_close">>,bool,true}], 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> <0.148188888.1>,<0.148188415.1>]}}, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> {restart_type,transient}, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> {significant,true}, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> {shutdown,70000}, 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1> {child_type,worker}] 2024-01-23 08:29:37.812711+00:00 [error] <0.148185383.1>

Cluster info:

`rabbitmq-diagnostics status
Status of node rabbit@node_cluster1 ...
Runtime

OS PID: 327821
OS: Linux
Uptime (seconds): 12788383
Is under maintenance?: false
RabbitMQ version: 3.12.4
RabbitMQ release series support status: supported
Node name: rabbit@node_cluster1
Erlang configuration: Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 1.1.1k FIPS 25 Mar 2021
Erlang processes: 18435 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

• rabbitmq_prometheus
• rabbitmq_federation_management
• rabbitmq_federation
• rabbitmq_auth_mechanism_ssl
• accept
• prometheus
• rabbitmq_management
• rabbitmq_management_agent
• rabbitmq_web_dispatch
• amqp_client
• cowboy
• cowlib

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@node_cluster1
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@node_cluster1/quorum/rabbit@node_cluster1

Config files

• /etc/rabbitmq/rabbitmq.config

Log file(s)

• /var/log/rabbitmq/rabbit@node_cluster1.log

Alarms

(none)

Memory

Total memory used: 2.4136 gb
Calculation strategy: rss
Memory high watermark setting: 0.8 of available memory, computed to: 13.4053 gb

allocated_unused: 0.8563 gb (35.48 %)
quorum_ets: 0.5464 gb (22.64 %)
other_proc: 0.2776 gb (11.5 %)
reserved_unallocated: 0.2081 gb (8.62 %)
mgmt_db: 0.1047 gb (4.34 %)
connection_other: 0.0648 gb (2.69 %)
plugins: 0.0581 gb (2.41 %)
queue_procs: 0.0567 gb (2.35 %)
binary: 0.0414 gb (1.71 %)
mnesia: 0.0402 gb (1.67 %)
code: 0.0378 gb (1.57 %)
other_system: 0.0338 gb (1.4 %)
queue_slave_procs: 0.0249 gb (1.03 %)
other_ets: 0.0212 gb (0.88 %)
connection_readers: 0.0158 gb (0.65 %)
connection_channels: 0.0068 gb (0.28 %)
metrics: 0.0065 gb (0.27 %)
msg_index: 0.0051 gb (0.21 %)
quorum_queue_procs: 0.0036 gb (0.15 %)
atom: 0.002 gb (0.08 %)
connection_writers: 0.0018 gb (0.08 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 1505, limit: 16287
Sockets: 491, limit: 14656

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 95.8187 gb

Totals

Connection count: 491
Queue count: 2803
Virtual host count: 26

Listeners

Interface: [::], port: 15671, protocol: https, purpose: HTTP API over TLS (HTTPS)
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS`

All other nodes have the same version on config.
the node with which the federation was configured:
Status of node rabbit@node_old ... [{pid,6244}, {running_applications, [{rabbitmq_federation_management,"RabbitMQ Federation Management", "3.7.8"}, {rabbitmq_federation,"RabbitMQ Federation","3.7.8"}, {rabbitmq_auth_mechanism_ssl, "RabbitMQ SSL authentication (SASL EXTERNAL)","3.7.8"}, {rabbitmq_management,"RabbitMQ Management Console","3.7.8"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.8"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.8"}, {rabbit,"RabbitMQ","3.7.8"}, {amqp_client,"RabbitMQ AMQP Client","3.7.8"}, {rabbit_common, "Modules shared by rabbitmq-server and rabbitmq-erlang-client", "3.7.8"}, {os_mon,"CPO CXC 138 46","2.4.7"}, {ranch_proxy_protocol,"Ranch Proxy Protocol Transport","1.5.0"}, {cowboy,"Small, fast, modern HTTP server.","2.2.2"}, {ranch,"Socket acceptor pool for TCP protocols.","1.5.0"}, {ssl,"Erlang/OTP SSL application","9.2.3.5"}, {public_key,"Public key infrastructure","1.6.6.1"}, {asn1,"The Erlang ASN1 compiler version 5.0.8","5.0.8"}, {cowlib,"Support library for manipulating Web protocols.","2.1.0"}, {jsx,"a streaming, evented json parsing toolkit","2.8.2"}, {mnesia,"MNESIA CXC 138 12","4.15.6"}, {recon,"Diagnostic tools for production use","2.3.2"}, {inets,"INETS CXC 138 49","7.0.7.1"}, {crypto,"CRYPTO","4.4.2"}, {xmerl,"XML parser","1.3.20.1"}, {lager,"Erlang logging framework","3.6.3"}, {goldrush,"Erlang event stream processor","0.1.9"}, {compiler,"ERTS CXC 138 10","7.3.2"}, {syntax_tools,"Syntax tools","2.1.7.1"}, {syslog,"An RFC 3164 and RFC 5424 compliant logging framework.","3.4.3"}, {sasl,"SASL CXC 138 11","3.3"}, {stdlib,"ERTS CXC 138 10","3.8.2.2"}, {kernel,"ERTS CXC 138 10","6.3.1.3"}]}, {os,{unix,linux}}, {erlang_version, "Erlang/OTP 21 [erts-10.3.5.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:128] [hipe]\n"}, {memory, [{connection_readers,0}, {connection_writers,0}, {connection_channels,0}, {connection_other,1942412}, {queue_procs,72376476}, {queue_slave_procs,0}, {plugins,201032572}, {other_proc,0}, {metrics,8539564}, {mgmt_db,182927944}, {mnesia,28986568}, {other_ets,33408808}, {binary,34915040}, {msg_index,956376}, {code,23979300}, {atom,1172689}, {other_system,16710379}, {allocated_unused,2087831024}, {reserved_unallocated,0}, {strategy,rss}, {total,[{erlang,577071632},{rss,2605199360},{allocated,2664902656}]}]}, {alarms,[]}, {listeners, [{clustering,25672,"::"}, {amqp,5672,"::"}, {'amqp/ssl',5671,"::"}, {https,15671,"::"}]}, {vm_memory_calculation_strategy,rss}, {vm_memory_high_watermark,0.8}, {vm_memory_limit,6545634099}, {disk_free_limit,50000000}, {disk_free,22473158656}, {file_descriptors, [{total_limit,16284}, {total_used,22}, {sockets_limit,14653}, {sockets_used,0}]}, {processes,[{limit,1048576},{used,8634}]}, {run_queue,1}, {uptime,34710057}, {kernel,{net_ticktime,60}}]

To fix that, I restarted master node for failed queues rabbitmqctl stop_app and rabbitmqctl start_app . Master was migrated , a policy was applied and mirroring between all nodes was began .

Reproduction steps

1. Create federation
2. Remove policies and federation upstreams from cluster and separate rabbitmq node
3. Queues still has policy which was created for federation
4. Restart rabbitmq node
5. Master was migrated for failed queues, policy is applied

Expected behavior

Policy for ha mirroring must be applied after removing federation policy and federation upstream.

Additional context

No response

[michaelklishin]

There isn't enough information to conclude that this is a bug. No logs, no policy priority details.
Your node could end up without an electable leader, we have absolutely no way of confirming this. In that case, no operations on it will succeed until you restart some replicas (and even then it is not guaranteed to help, depending on the mirroring settings).

More importantly, we will not be investigating anything related to classic mirrored queues. They will be REMOVED from RabbitMQ within the next 2-3 months. They have been deprecated for over two years now.

Quorum queues have been around for close to six years.

It's the end of the road for classic mirrored queues, regardless of what exactly happened in your cluster.