Crash during 3.13 -> 4.0 upgrade when there is an enabled feature flag of a disabled plugin

[gomoripeti]

Describe the bug

When a plugin is enabled its feature flags are discovered however when the same plugin is disabled its feature flags remain discovered. It is even possible to enable such feature flag leading to various problems, crashes, and nodes refusing to start.

We have originally faced this issue with the MQTT plugin and the rabbit_mqtt_qos0_queue feature flag. On a new 3.13.7 cluster mqtt plugin was enabled then disabled. Then all feature flags were enabled enabling rabbit_mqtt_qos0_queue. Finally during a rolling upgrade to 4.0.4 the bellow crash was seen preventing the first node to start:

boot crash

2024-12-16 12:39:58.423260+00:00 [error] <0.267.0> BOOT FAILED
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0> ===========
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0> Exception during startup:
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0> error:{badmatch,#{feature_flags => #{classic_mirrored_queue_version => #{name => classic_mirrored_queue_version,desc => "Support setting version for classic mirrored queues",stability => required,provided_by => rabbit},classic_queue_type_delivery_support => #{name => classic_queue_type_delivery_support,desc => "Bug fix for classic queue deliveries using mixed versions",stability => required,doc_url => "https://github.com/rabbitmq/rabbitmq-server/issues/5931",depends_on => [stream_queue],provided_by => rabbit},detailed_queues_endpoint => #{name => detailed_queues_endpoint,desc => "Add a detailed queues HTTP API endpoint. Reduce number of metrics in the default endpoint.",stability => stable,depends_on => [feature_flags_v2],provided_by => rabbitmq_management},direct_exchange_routing_v2 => #{name => direct_exchange_routing_v2,desc => "v2 direct exchange routing implementation",stability => required,depends_on => [feature_flags_v2,implicit_default_bindings],provided_by => rabbit},drop_unroutable_metric => #{name => drop_unroutable_metric,desc => "Count unroutable publishes to be dropped in stats",stability => required,provided_by => rabbitmq_management_agent},empty_basic_get_metric => #{name => empty_basic_get_metric,desc => "Count AMQP `basic.get` on empty queues in stats",stability => required,provided_by => rabbitmq_management_agent},feature_flags_v2 => #{name => feature_flags_v2,desc => "Feature flags subsystem V2",stability => required,provided_by => rabbit},implicit_default_bindings => #{name => implicit_default_bindings,desc => "Default bindings are now implicit, instead of being stored in the database",stability => required,provided_by => rabbit},listener_records_in_ets => #{name => listener_records_in_ets,desc => "Store listener records in ETS instead of Mnesia",stability => required,depends_on => [feature_flags_v2],provided_by => rabbit},maintenance_mode_status => #{name => maintenance_mode_status,desc => "Maintenance mode status",stability => required,provided_by => rabbit},message_containers => #{name => message_containers,desc => "Message containers.",stability => stable,depends_on => [feature_flags_v2],provided_by => rabbit},message_containers_deaths_v2 => #{name => message_containers_deaths_v2,desc => "Bug fix for dead letter cycle detection",stability => stable,doc_url => "https://github.com/rabbitmq/rabbitmq-server/issues/11159",depends_on => [message_containers],provided_by => rabbit},quorum_queue => #{name => quorum_queue,desc => "Support queues of type `quorum`",stability => required,doc_url => "https://www.rabbitmq.com/quorum-queues.html",provided_by => rabbit},quorum_queue_non_voters => #{name => quorum_queue_non_voters,desc => "Allows new quorum queue members to be added as non voters initially.",stability => stable,depends_on => [quorum_queue],provided_by => rabbit},restart_streams => #{name => restart_streams,desc => "Support for restarting streams with optional preferred next leader argument.Used to implement stream leader rebalancing",stability => stable,depends_on => [stream_queue],provided_by => rabbit},stream_filtering => #{name => stream_filtering,desc => "Support for stream filtering.",stability => stable,depends_on => [stream_queue],provided_by => rabbit},stream_queue => #{name => stream_queue,desc => "Support queues of type `stream`",stability => required,doc_url => "https://www.rabbitmq.com/stream.html",depends_on => [quorum_queue],provided_by => rabbit},stream_sac_coordinator_unblock_group => #{name => stream_sac_coordinator_unblock_group,desc => "Bug fix to unblock a group of consumers in a super stream partition",stability => stable,doc_url => "https://github.com/rabbitmq/rabbitmq-server/issues/7743",depends_on => [stream_single_active_consumer],provided_by => rabbit},stream_single_active_consumer => #{name => stream_single_active_consumer,desc => "Single active consumer for streams",stability => required,doc_url => "https://www.rabbitmq.com/stream.html",depends_on => [stream_queue],provided_by => rabbit},stream_update_config_command => #{name => stream_update_config_command,desc => "A new internal command that is used to update streams as part of a policy.",stability => stable,depends_on => [stream_queue],provided_by => rabbit},tracking_records_in_ets => #{name => tracking_records_in_ets,desc => "Store tracking records in ETS instead of Mnesia",stability => required,depends_on => [feature_flags_v2],provided_by => rabbit},user_limits => #{name => user_limits,desc => "Configure connection and channel limits for a user",stability => required,provided_by => rabbit},virtual_host_metadata => #{name => virtual_host_metadata,desc => "Virtual host metadata (description, tags, etc)",stability => required,provided_by => rabbit},transient_nonexcl_queues => #{messages => #{when_permitted => "Feature `transient_nonexcl_queues` is deprecated.\nBy default, this feature can still be used for now.\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.transient_nonexcl_queues = true\"\nTo test RabbitMQ as if the feature was removed, set this in your configuration:\n    \"deprecated_features.permit.transient_nonexcl_queues = false\"",when_denied => "Feature `transient_nonexcl_queues` is deprecated.\nIts use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.transient_nonexcl_queues = false\"\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.transient_nonexcl_queues = true\""},name => transient_nonexcl_queues,deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-transient-non-exclusive-queues",provided_by => rabbit},global_qos => #{messages => #{when_permitted => "Feature `global_qos` is deprecated.\nBy default, this feature can still be used for now.\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.global_qos = true\"\nTo test RabbitMQ as if the feature was removed, set this in your configuration:\n    \"deprecated_features.permit.global_qos = false\"",when_denied => "Feature `global_qos` is deprecated.\nIts use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.global_qos = false\"\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.global_qos = true\""},name => global_qos,deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-global-qos",provided_by => rabbit},khepri_db => #{name => khepri_db,desc => "Use the new Khepri Raft-based metadata store",stability => experimental,doc_url => [],depends_on => [feature_flags_v2,direct_exchange_routing_v2,maintenance_mode_status,user_limits,virtual_host_metadata,tracking_records_in_ets,listener_records_in_ets,classic_queue_mirroring,ram_node_type],provided_by => rabbit},classic_queue_mirroring => #{messages => #{when_permitted => "Classic mirrored queues are deprecated.\nBy default, they can still be used for now.\nTheir use will not be permitted by default in the next minorRabbitMQ version (if any) and they will be removed from RabbitMQ 4.0.0.\nTo continue using classic mirrored queues when they are not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.classic_queue_mirroring = true\"\nTo test RabbitMQ as if they were removed, set this in your configuration:\n    \"deprecated_features.permit.classic_queue_mirroring = false\"",when_denied => "Classic mirrored queues are deprecated.\nTheir use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.classic_queue_mirroring = false\"\nTheir use will not be permitted by default in the next minor RabbitMQ version (if any) and they will be removed from RabbitMQ 4.0.0.\nTo continue using classic mirrored queues when they are not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.classic_queue_mirroring = true\""},name => classic_queue_mirroring,callbacks => #{enable => {rabbit_deprecated_features,enable_underlying_feature_flag_cb},is_feature_used => {rabbit_mirror_queue_misc,are_cmqs_used}},deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-classic-queue-mirroring",provided_by => rabbit},ram_node_type => #{messages => #{when_permitted => "Feature `ram_node_type` is deprecated.\nBy default, this feature can still be used for now.\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.ram_node_type = true\"\nTo test RabbitMQ as if the feature was removed, set this in your configuration:\n    \"deprecated_features.permit.ram_node_type = false\"",when_denied => "Feature `ram_node_type` is deprecated.\nIts use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.ram_node_type = false\"\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.ram_node_type = true\""},name => ram_node_type,deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-ram-nodes",provided_by => rabbit},management_metrics_collection => #{messages => #{when_permitted => "Feature `management_metrics_collection` is deprecated.\nBy default, this feature can still be used for now.\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.management_metrics_collection = true\"\nTo test RabbitMQ as if the feature was removed, set this in your configuration:\n    \"deprecated_features.permit.management_metrics_collection = false\"",when_denied => "Feature `management_metrics_collection` is deprecated.\nIts use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.management_metrics_collection = false\"\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.management_metrics_collection = true\""},name => management_metrics_collection,deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#disable-metrics-delivery-via-the-management-api--ui",provided_by => rabbitmq_management_agent}},states_per_node => #{'rabbit@healthy-cobalt-greyhound-03' => #{rabbit_mqtt_qos0_queue => true,classic_mirrored_queue_version => true,classic_queue_type_delivery_support => true,detailed_queues_endpoint => true,direct_exchange_routing_v2 => true,drop_unroutable_metric => true,empty_basic_get_metric => true,feature_flags_v2 => true,implicit_default_bindings => true,listener_records_in_ets => true,maintenance_mode_status => true,message_containers => true,message_containers_deaths_v2 => true,quorum_queue => true,quorum_queue_non_voters => true,restart_streams => true,stream_filtering => true,stream_queue => true,stream_sac_coordinator_unblock_group => true,stream_single_active_consumer => true,stream_update_config_command => true,tracking_records_in_ets => true,user_limits => true,virtual_host_metadata => true,transient_nonexcl_queues => false,global_qos => false,khepri_db => false,classic_queue_mirroring => false,ram_node_type => false,management_metrics_collection => false,delete_ra_cluster_mqtt_node => false,mqtt_v5 => false},'rabbit@healthy-cobalt-greyhound-02' => #{rabbit_mqtt_qos0_queue => true,classic_mirrored_queue_version => true,classic_queue_type_delivery_support => true,detailed_queues_endpoint => true,direct_exchange_routing_v2 => true,drop_unroutable_metric => true,empty_basic_get_metric => true,feature_flags_v2 => true,implicit_default_bindings => true,listener_records_in_ets => true,maintenance_mode_status => true,message_containers => true,message_containers_deaths_v2 => true,quorum_queue => true,quorum_queue_non_voters => true,restart_streams => true,stream_filtering => true,stream_queue => true,stream_sac_coordinator_unblock_group => true,stream_single_active_consumer => true,stream_update_config_command => true,tracking_records_in_ets => true,user_limits => true,virtual_host_metadata => true,transient_nonexcl_queues => false,global_qos => false,khepri_db => false,classic_queue_mirroring => false,ram_node_type => false,management_metrics_collection => false,delete_ra_cluster_mqtt_node => false,mqtt_v5 => false}},applications_per_node => #{'rabbit@healthy-cobalt-greyhound-03' => [rabbit,rabbit_common,rabbitmq_consistent_hash_exchange,rabbitmq_federation,rabbitmq_federation_management,rabbitmq_management,rabbitmq_management_agent,rabbitmq_prelaunch,rabbitmq_prometheus,rabbitmq_shovel,rabbitmq_shovel_management,rabbitmq_web_dispatch],'rabbit@healthy-cobalt-greyhound-02' => [rabbit,rabbit_common,rabbitmq_consistent_hash_exchange,rabbitmq_federation,rabbitmq_federation_management,rabbitmq_management,rabbitmq_management_agent,rabbitmq_prelaunch,rabbitmq_prometheus,rabbitmq_shovel,rabbitmq_shovel_management,rabbitmq_web_dispatch]}}}
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_ff_controller:-check_one_way_compatibility/2-fun-0-/3, line 514
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     lists:all_1/2, line 1520
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_ff_controller:are_compatible/2, line 496
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_ff_controller:check_node_compatibility_task1/4, line 437
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_db_cluster:check_compatibility/1, line 376
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_mnesia:check_cluster_consistency/2, line 687
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     lists:foldl/3, line 1594
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_mnesia:check_cluster_consistency/0, line 648

But similar issues can be reproduced on new 4.0.4 cluster with the rabbitmq_management plugin and detailed_queues_endpoint feature flag. See repro steps.

It is possible that feature flag handling changed between 3.13.7 and 4.0 so that this crash that happened during upgrade couldn't happen for later upgrades (eg 4.0.x -> 4.1.x)

Reproduction steps

1. Start local 3-node 4.0.x cluster from git repo

   % make start-cluster TEST_TMPDIR=/tmp/rabbitmq RABBITMQ_ENABLED_PLUGINS=
   

2. Plugins are not enabled and the detailed_queues_endpoint feature flag is not available

   % for i in 1 2 3; do sbin/rabbitmq-plugins list -q rabbitmq_management --node rabbit-$i; done
   [  ] rabbitmq_management            4.0.3+112.g78602ec
   [  ] rabbitmq_management_agent      4.0.3+112.g78602ec
   [  ] rabbitmq_management            4.0.3+112.g78602ec
   [  ] rabbitmq_management_agent      4.0.3+112.g78602ec
   [  ] rabbitmq_management            4.0.3+112.g78602ec
   [  ] rabbitmq_management_agent      4.0.3+112.g78602ec
   
   % for i in 1 2 3; do sbin/rabbitmqctl list_feature_flags --node rabbit-$i | grep detailed ; done
   <empty response>
   

3. Enable management plugin

   % sbin/rabbitmq-plugins enable rabbitmq_management --node rabbit-1
   Enabling plugins on node rabbit-1@Peters-MBP:
   rabbitmq_management
   The following plugins have been configured:
     rabbitmq_management
     rabbitmq_management_agent
     rabbitmq_web_dispatch
   Applying plugin configuration to rabbit-1@Peters-MBP...
   The following plugins have been enabled:
     rabbitmq_management
     rabbitmq_management_agent
     rabbitmq_web_dispatch
   
   started 3 plugins.
   

4. The feature flag is listed on node-1 as disabled. This is already unexpected. I would expect it to be "unavailable" as it is not available on all nodes

   % for i in 1 2 3; do echo rabbit-$i; sbin/rabbitmqctl list_feature_flags --node rabbit-$i | grep detailed ; done
   rabbit-1
   detailed_queues_endpoint	disabled
   rabbit-2
   rabbit-3
   

5. Disable the management plugin

   % sbin/rabbitmq-plugins disable rabbitmq_management --node rabbit-1
   Disabling plugins on node rabbit-1@Peters-MBP:
   rabbitmq_management
   All plugins have been disabled.
   Applying plugin configuration to rabbit-1@Peters-MBP...
   The following plugins have been disabled:
     rabbitmq_web_dispatch
     rabbitmq_management_agent
     rabbitmq_management
   
   stopped 3 plugins.
   

6. Feature flag still listed on node-1

   % for i in 1 2 3; do echo rabbit-$i; sbin/rabbitmqctl list_feature_flags --node rabbit-$i | grep detailed ; done
   rabbit-1
   detailed_queues_endpoint	disabled
   rabbit-2
   rabbit-3
   

7. Trying to enable the plugin on node-2 leads to a crash and node-2 goes down

   % sbin/rabbitmqctl enable_feature_flag detailed_queues_endpoint --node rabbit-2
   Enabling feature flag "detailed_queues_endpoint" ...
   Error:
   {{{:badmap, :undefined}, [{:rabbit_ff_controller, :run_callback, 5, []}, {:rabbit_ff_controller, :do_enable, 3, [file: 'rabbit_ff_controller.erl', line: 1043]}, {:rabbit_ff_controller, :update_feature_state_and_enable, 2, [file: 'rabbit_ff_controller.erl', line: 980]}, {:rabbit_ff_controller, :enable_with_registry_locked, 2, [file: 'rabbit_ff_controller.erl', line: 851]}, {:rabbit_ff_controller, :lock_registry_and_enable, 2, [file: 'rabbit_ff_controller.erl', line: 819]}, {:rabbit_ff_controller, :enable_many, 2, [file: 'rabbit_ff_controller.erl', line: 776]}, {:rabbit_ff_controller, :updating_feature_flag_states, 3, [file: 'rabbit_ff_controller.erl', line: 276]}, {:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1395]}]}, {:gen_statem, :call, [:rabbit_ff_controller, {:enable, [:detailed_queues_endpoint]}, :infinity]}}
   

Expected behavior

• if the plugin is only loaded on part of the cluster nodes: list it as unavailable
• it is not possible to enable a feature flag of a disabled plugin

Additional context

No response

[gomoripeti]

One difference between 3.13.7 and mqtt plugin and 4.0.x and management plugin is that on 3.13.7 with a disabled mqtt plugin I got the following error printout

sudo rabbitmqctl enable_feature_flag all 
 Enabling all feature flags ... 
 Error: 
 This feature flag is not supported by node rabbit@node-01 

And out of the 3 feature flags defined by the mqtt plugin (rabbit_mqtt_qos0_queue, delete_ra_cluster_mqtt_node and mqtt_v5) only the first one got enabled.

[michaelklishin]

Such exotic scenarios where not all nodes have all the plugins enabled, or they are being disabled and re-enabled during an upgrade… are they a really good use of our time?

I highly doubt any feature flag- or upgrades-related integration suites enable plugins "mid-upgrade".

Feature flags are considered and documented as cluster-wide state. This means that the enabled set of plugins must be stable during an upgrade, or else the state is mutated by something the feature flag controller has no knowledge of, or control over.

[michaelklishin]

@gomoripeti I have a fundamental problem with our small team's resources being spent on highly exotic scenarios like this.

Disabling and re-enabling plugins during a version upgrade is not a good idea. During a routine restart where the list of plugin changes feature flags should play no role if the version is stable.

In a hypothetical scenario where one of the enabled plugin automatically needs to enable feature flags, it will not be able to do that until all nodes are compatible.

We can spend a few person-weeks on this or we can spend it on improvements such as Ra 2.16 which deliver double digit efficiency improvements on workflows that affect most QQ users.

[michaelklishin]

if the plugin is only loaded on part of the cluster nodes: list it as unavailable

Enabled plugins is a node's state. Feature flags is a cluster-wide state.

It would likely raise a lot of eyebrows if a plugin was listed as "disabled" or unavailable due to feature flag state.

it is not possible to enable a feature flag of a disabled plugin

This would imply that when a plugin is disabled, so should be its feature flags, and a feature flag cannot be disabled once enabled, or at least it won't be possible while other nodes have the plugin and the FF enabled.

Instead of trying to change RabbitMQ, reduce the number of changes are allowed during a version upgrade in your system. A version upgrade is not the time when plugins should be enabled or disabled, but they can be enabled or disabled once all nodes are on the same version, in which case you won't run into any surprises around feature flags.

[Zerpet]

I echo Michael's thoughts. I think this scenario is quite unusual, perhaps even artificial. That being said, I think we should print a friendly error, as we did in 3.13.x.

[dumbbell]

This scenario is already supported. This is clearly a bug: the feature flag controller isn’t supposed to run a callback on a node that doesn’t know the feature flag. The list of nodes is already filtered, so there is an issue with the filtering or the input of that filtering.

The feature flag is listed on node-1 as disabled. This is already unexpected. I would expect it to be "unavailable" as it is not available on all nodes

This is expected because a node can have a plugin enabled while other nodes in the cluster don’t have it enabled or don’t have it at all. When that plugin is enabled elsewhere, it is supposed to pick up the feature state from the rest of the cluster. The sync is the same as when a node joins a cluster or is restarted.

[dumbbell]

I submitted a patch in #12966. Could you please give it a try?

Unrelated to this, but in main (future 4.1.x), I tried to improve the user experience around feature flags. If you think something still needs improvements, please share!

[gomoripeti]

Sorry for the misleading and bad repro steps using 4.0.x and the management plugin. I tried to come up with an example on a recent version but I failed.

I'd like to clarify that there is no change happening during the upgrade. The real world issue what we faced is the following (that I also described and attached the logs for)

• On a new 3.13.7 cluster mqtt plugin was enabled then disabled 2 months ago
• Then (a few days ago) a 3.13.7 -> 4.0.4 upgrade was initiated, we enable all feature flags before starting that process.
• First problem: enabling all feature flags returned an error but out of the 3 feature flags defined by the mqtt plugin (rabbit_mqtt_qos0_queue, delete_ra_cluster_mqtt_node and mqtt_v5) the first one still got enabled.

  sudo rabbitmqctl enable_feature_flag all 
   Enabling all feature flags ... 
   Error: 
   This feature flag is not supported by node rabbit@node-01 
  

• Second problem: during a rolling upgrade to 4.0.4 a big crash was seen preventing the first node to start (crash included in the original post)

...while typing the core team already provided a potential patch :) thank you very much, I am giving it a try

[dumbbell]

I merged the bug fix in #12966 and will backport it to 4.0.x in a few days.

Otherwise, main and 4.0.x look ok w.r.t. the initial crash. I don’t plan to backport #10988 because my rule is to only backport small bugfixes in the Feature flag subsystem, never a behavior change or a new feature. Anyway, 3.13.x is not maintained anymore.

I will stop working on this now. @gomoripeti: do you have any observations?

[gomoripeti]

thank you for your time on this topic! (I did not expect anything to be backported to 3.13.x)

What I dont fully understand is that in my use case the crash happens on a 4.0.x node (the first node in a rolling upgrade) So I thought that "something" can be fixed in 4.0.x to avoid this. Do I understand correctly then that this crash does not happen because is_enabled() returns true or false, but because it returns a different value on different nodes in a 3.13.x+4.0.x mixed version cluster? (by crash I mean error:{badmatch, in rabbit_ff_controller:-check_one_way_compatibility/2-fun-0-/3, line 514)

[michaelklishin]

rabbit_ff_controller can be updated in v4.0.x to be more defensive, e.g. by using a case with more heads rather than an if with just two https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_ff_controller.erl#L514. If that alone makes the upgrade scenario proceed, I'd say we should do that.

[michaelklishin]

@gomoripeti would you like to verify the above hypothesis, assuming that you have a way to reproduce this behavior reliably?

[gomoripeti]

I am happy to verify things or even experiment with some changes - however unfortunately holiday season starts, so I can only do it after.

(just noting that line 514 on v4.0.4 is a different peace of code https://github.com/rabbitmq/rabbitmq-server/blob/v4.0.4/deps/rabbit/src/rabbit_ff_controller.erl#L514. dumbbell already provided a rewrite of this part in #12963 (reply in thread) but as far as I understand it changes behaviour of the FF subsystem and such changes are usually not back ported to previous minor versions)

let me get back to this topic in the new year. until then happy holidays!