Unable to upgrade from 3.12.13 (erlang 26.2.5) to 3.13.3 - incompatible_feature_flags

[pmcevoy]

I'm trying to do a rolling upgrade of a 3.12.13 cluster to 3.13.3 (both with erlang 26.2.5) but new 3.13.3 nodes are failing to join with the error incompatible_feature_flags

My existing 3 node cluster has the following versions with all flags enabled

Running Nodes

rabbit@10.80.10.240
rabbit@10.80.8.246
rabbit@10.80.9.139

Versions

rabbit@10.80.10.240: RabbitMQ 3.12.14 on Erlang 26.2.5
rabbit@10.80.8.246: RabbitMQ 3.12.14 on Erlang 26.2.5
rabbit@10.80.9.139: RabbitMQ 3.12.14 on Erlang 26.2.5

... 

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: direct_exchange_routing_v2, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: feature_flags_v2, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: listener_records_in_ets, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: restart_streams, state: enabled
Flag: stream_queue, state: enabled
Flag: stream_sac_coordinator_unblock_group, state: enabled
Flag: stream_single_active_consumer, state: enabled
Flag: tracking_records_in_ets, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Note that as per recommendation, all feature flags are enabled.

The new 3.13.3 node has the following feature flags out-of-the-box from the RPM from github download - I did nothing to enable extra features.

Running Nodes

rabbit@10.80.8.116

Versions

rabbit@10.80.8.116: RabbitMQ 3.13.3 on Erlang 26.2.5
...
Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: detailed_queues_endpoint, state: enabled
Flag: direct_exchange_routing_v2, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: feature_flags_v2, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: khepri_db, state: disabled
Flag: listener_records_in_ets, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: message_containers, state: enabled
Flag: message_containers_deaths_v2, state: enabled
Flag: quorum_queue, state: enabled
Flag: quorum_queue_non_voters, state: enabled
Flag: restart_streams, state: enabled
Flag: stream_filtering, state: enabled
Flag: stream_queue, state: enabled
Flag: stream_sac_coordinator_unblock_group, state: enabled
Flag: stream_single_active_consumer, state: enabled
Flag: stream_update_config_command, state: enabled
Flag: tracking_records_in_ets, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Relevant rabbitmq log on the node failing to join:

2024-06-12 16:10:44.514418+00:00 [info] <0.255.0> Peer discovery: will try to unlock with peer discovery backend rabbit_peer_discovery_aws
2024-06-12 16:10:46.006409+00:00 [info] <0.255.0> Peer discovery: node 'rabbit@10.80.8.246' selected for auto-clustering
2024-06-12 16:10:46.006498+00:00 [info] <0.255.0> Peer discovery: will try to lock with peer discovery backend rabbit_peer_discovery_aws
2024-06-12 16:10:46.006581+00:00 [info] <0.255.0> Will try to lock connecting to nodes ['rabbit@10.80.8.116',
2024-06-12 16:10:46.006581+00:00 [info] <0.255.0>                                       'rabbit@10.80.8.246']
2024-06-12 16:10:46.007416+00:00 [info] <0.255.0> DB: checking if `rabbit@10.80.8.116` can join cluster using remote node `rabbit@10.80.8.246`
2024-06-12 16:10:46.007493+00:00 [notice] <0.255.0> Feature flags: checking nodes `rabbit@10.80.8.116` and `rabbit@10.80.8.246` compatibility...
2024-06-12 16:10:46.012214+00:00 [warning] <0.255.0> Feature flags: nodes `rabbit@10.80.8.116` and `rabbit@10.80.8.246` are incompatible
2024-06-12 16:10:46.012312+00:00 [warning] <0.255.0> Peer discovery: could not auto-cluster with node 'rabbit@10.80.8.246': {error,incompatible_feature_flags}

I'm at a loss how to proceed or resolve. I have successfully managed to incrementally upgrade all the way from rabbit 3.8.2/erlang 22.2.7 to rabbit 3.12.14/erlang 26.2.5 without (much) issues, but this last jump has me baffled.

Sincerely
Pete

[lukebakken]

I verified that upgrading from 3.12.13 to the latest 3.x RabbitMQ works correctly:

https://github.com/lukebakken/rabbitmq-server-11441

See upgrade.sh for steps I used to upgrade this docker compose environment.

To see it in action:

direnv allow # optional
make up
./upgrade.sh