Cluster formation: feature flag node compatibility check runs into an erpc no connection error #12877

Hussain-f · 2024-12-02T19:59:49Z

Hussain-f
Dec 2, 2024

Community Support Policy

I have read RabbitMQ's Community Support Policy
I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.3

Erlang version used

26.2.x

Operating system (distribution) used

Linux amazon

How is RabbitMQ deployed?

RPM package

Logs from node 1 (with sensitive values edited out)

2024-12-02 19:08:21.431319+00:00 [info] <0.762.0>  Starting RabbitMQ 4.0.4 on Erlang 27.1.2 [jit]
2024-12-02 19:08:21.431319+00:00 [info] <0.762.0>  Copyright (c) 2007-2024 Broadcom Inc and/or its subsidiaries
2024-12-02 19:08:21.431319+00:00 [info] <0.762.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
2024-12-02 19:08:21.432744+00:00 [info] <0.762.0> 
2024-12-02 19:08:21.432744+00:00 [info] <0.762.0>  node           : node-01@ip-x-x-x-x.eu-west-1.compute.internal
2024-12-02 19:08:21.432744+00:00 [info] <0.762.0>  home dir       : /var/lib/rabbitmq
2024-12-02 19:08:21.432744+00:00 [info] <0.762.0>  config file(s) : (none)
2024-12-02 19:08:21.432744+00:00 [info] <0.762.0>  cookie hash    : akPKEN*****************xig==
2024-12-02 19:08:21.432744+00:00 [info] <0.762.0>  log(s)         : /var/log/rabbitmq/node-01@ip-10-0-x-x.eu-west-1.compute.internal.log
2024-12-02 19:08:21.432744+00:00 [info] <0.762.0>                 : <stdout>
2024-12-02 19:08:21.432744+00:00 [info] <0.762.0>  data dir       : /var/lib/rabbitmq/mnesia/node-01@ip-10-0-x-x.eu-west-1.compute.internal
2024-12-02 19:08:21.480374+00:00 [info] <0.762.0> Running boot step pre_boot defined by app rabbit
2024-12-02 19:08:21.480445+00:00 [info] <0.762.0> Running boot step rabbit_global_counters defined by app rabbit
2024-12-02 19:08:21.482660+00:00 [info] <0.762.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2024-12-02 19:08:21.482887+00:00 [info] <0.762.0> Running boot step rabbit_core_metrics defined by app rabbit
2024-12-02 19:08:21.483611+00:00 [info] <0.762.0> Running boot step rabbit_alarm defined by app rabbit
2024-12-02 19:08:21.491531+00:00 [info] <0.828.0> Memory high watermark set to 1140 MiB (1196283494 bytes) of 1901 MiB (1993805824 bytes) total
2024-12-02 19:08:21.497202+00:00 [info] <0.830.0> Enabling free disk space monitoring (disk free space: 30197551104, total memory: 1993805824)
2024-12-02 19:08:21.497297+00:00 [info] <0.830.0> Disk free limit set to 50MB
2024-12-02 19:08:21.500670+00:00 [info] <0.762.0> Running boot step code_server_cache defined by app rabbit
2024-12-02 19:08:21.500812+00:00 [info] <0.762.0> Running boot step file_handle_cache defined by app rabbit
2024-12-02 19:08:21.501025+00:00 [info] <0.833.0> Limiting to approx 32671 file handles (29401 sockets)
2024-12-02 19:08:21.501198+00:00 [info] <0.834.0> FHC read buffering: OFF
2024-12-02 19:08:21.501261+00:00 [info] <0.834.0> FHC write buffering: ON
2024-12-02 19:08:21.501854+00:00 [info] <0.762.0> Running boot step worker_pool defined by app rabbit
2024-12-02 19:08:21.501976+00:00 [info] <0.815.0> Will use 2 processes for default worker pool
2024-12-02 19:08:21.502035+00:00 [info] <0.815.0> Starting worker pool 'worker_pool' with 2 processes in it
2024-12-02 19:08:21.502368+00:00 [info] <0.762.0> Running boot step database defined by app rabbit
2024-12-02 19:08:21.502859+00:00 [info] <0.762.0> Peer discovery: configured backend: rabbit_peer_discovery_classic_config
2024-12-02 19:08:21.504355+00:00 [notice] <0.816.0> Feature flags: attempt to enable `detailed_queues_endpoint`...
2024-12-02 19:08:21.515012+00:00 [notice] <0.816.0> Feature flags: `detailed_queues_endpoint` enabled
2024-12-02 19:08:21.515129+00:00 [notice] <0.816.0> Feature flags: attempt to enable `rabbit_exchange_type_local_random`...
2024-12-02 19:08:21.527327+00:00 [notice] <0.816.0> Feature flags: `rabbit_exchange_type_local_random` enabled
2024-12-02 19:08:21.527514+00:00 [notice] <0.816.0> Feature flags: attempt to enable `message_containers_deaths_v2`...
2024-12-02 19:08:21.539107+00:00 [notice] <0.816.0> Feature flags: `message_containers_deaths_v2` enabled
2024-12-02 19:08:21.539290+00:00 [notice] <0.816.0> Feature flags: attempt to enable `quorum_queue_non_voters`...
2024-12-02 19:08:21.550710+00:00 [notice] <0.816.0> Feature flags: `quorum_queue_non_voters` enabled
2024-12-02 19:08:21.550918+00:00 [notice] <0.816.0> Feature flags: attempt to enable `rabbitmq_4.0.0`...
2024-12-02 19:08:21.560063+00:00 [notice] <0.816.0> Feature flags: `rabbitmq_4.0.0` enabled
2024-12-02 19:08:21.560251+00:00 [info] <0.762.0> DB: virgin node -> run peer discovery
2024-12-02 19:08:21.560303+00:00 [warning] <0.762.0> Classic peer discovery backend: list of nodes does not contain the local node []
2024-12-02 19:08:21.564681+00:00 [notice] <0.45.0> Application mnesia exited with reason: stopped
2024-12-02 19:08:21.661600+00:00 [info] <0.762.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2024-12-02 19:08:21.661766+00:00 [info] <0.762.0> Successfully synced tables from a peer
2024-12-02 19:08:21.661872+00:00 [info] <0.762.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2024-12-02 19:08:21.662007+00:00 [info] <0.762.0> Successfully synced tables from a peer
2024-12-02 19:08:21.668667+00:00 [info] <0.762.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2024-12-02 19:08:21.668802+00:00 [info] <0.762.0> Successfully synced tables from a peer
2024-12-02 19:08:21.668945+00:00 [info] <0.762.0> Running boot step tracking_metadata_store defined by app rabbit
2024-12-02 19:08:21.669028+00:00 [info] <0.1021.0> Setting up a table for connection tracking on this node: tracked_connection
2024-12-02 19:08:21.669077+00:00 [info] <0.1021.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
2024-12-02 19:08:21.669172+00:00 [info] <0.1021.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
2024-12-02 19:08:21.669263+00:00 [info] <0.1021.0> Setting up a table for channel tracking on this node: tracked_channel
2024-12-02 19:08:21.669344+00:00 [info] <0.1021.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
2024-12-02 19:08:21.669434+00:00 [info] <0.762.0> Running boot step networking_metadata_store defined by app rabbit
2024-12-02 19:08:21.669505+00:00 [info] <0.762.0> Running boot step feature_flags defined by app rabbit
2024-12-02 19:08:21.669704+00:00 [info] <0.762.0> Running boot step codec_correctness_check defined by app rabbit
2024-12-02 19:08:21.669737+00:00 [info] <0.762.0> Running boot step external_infrastructure defined by app rabbit
2024-12-02 19:08:21.669795+00:00 [info] <0.762.0> Running boot step rabbit_event defined by app rabbit
2024-12-02 19:08:21.669924+00:00 [info] <0.762.0> Running boot step rabbit_registry defined by app rabbit
2024-12-02 19:08:21.669975+00:00 [info] <0.762.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2024-12-02 19:08:21.670050+00:00 [info] <0.762.0> Running boot step rabbit_auth_mechanism_anonymous defined by app rabbit
2024-12-02 19:08:21.670101+00:00 [info] <0.762.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2024-12-02 19:08:21.670162+00:00 [info] <0.762.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2024-12-02 19:08:21.670215+00:00 [info] <0.762.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2024-12-02 19:08:21.670269+00:00 [info] <0.762.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2024-12-02 19:08:21.670326+00:00 [info] <0.762.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2024-12-02 19:08:21.670387+00:00 [info] <0.762.0> Running boot step rabbit_exchange_type_local_random defined by app rabbit
2024-12-02 19:08:21.670467+00:00 [info] <0.762.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2024-12-02 19:08:21.670519+00:00 [info] <0.762.0> Running boot step rabbit_priority_queue defined by app rabbit
2024-12-02 19:08:21.670565+00:00 [info] <0.762.0> Running boot step kernel_ready defined by app rabbit
2024-12-02 19:08:21.670594+00:00 [info] <0.762.0> Running boot step pg_local_amqp_connection defined by app rabbit
2024-12-02 19:08:21.670705+00:00 [info] <0.762.0> Running boot step pg_local_amqp_session defined by app rabbit
2024-12-02 19:08:21.670828+00:00 [info] <0.762.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2024-12-02 19:08:21.670978+00:00 [info] <0.762.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2024-12-02 19:08:21.671794+00:00 [info] <0.1031.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2024-12-02 19:08:21.671895+00:00 [info] <0.762.0> Running boot step guid_generator defined by app rabbit
2024-12-02 19:08:21.674338+00:00 [info] <0.762.0> Running boot step rabbit_node_monitor defined by app rabbit
2024-12-02 19:08:21.674568+00:00 [info] <0.1035.0> Starting rabbit_node_monitor (in ignore mode)
2024-12-02 19:08:21.674662+00:00 [info] <0.762.0> Running boot step delegate_sup defined by app rabbit
2024-12-02 19:08:21.675024+00:00 [info] <0.762.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
2024-12-02 19:08:21.675087+00:00 [info] <0.762.0> Running boot step core_initialized defined by app rabbit
2024-12-02 19:08:21.675123+00:00 [info] <0.762.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2024-12-02 19:08:21.675180+00:00 [info] <0.762.0> Running boot step rabbit_classic_queue defined by app rabbit
2024-12-02 19:08:21.675256+00:00 [info] <0.762.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2024-12-02 19:08:21.675319+00:00 [info] <0.762.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2024-12-02 19:08:21.675382+00:00 [info] <0.762.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2024-12-02 19:08:21.805998+00:00 [info] <0.762.0> Running boot step rabbit_policies defined by app rabbit
2024-12-02 19:08:21.806501+00:00 [info] <0.762.0> Running boot step rabbit_policy defined by app rabbit
2024-12-02 19:08:21.806814+00:00 [info] <0.762.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2024-12-02 19:08:21.806903+00:00 [info] <0.762.0> Running boot step rabbit_quorum_queue defined by app rabbit
2024-12-02 19:08:21.807028+00:00 [info] <0.762.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2024-12-02 19:08:21.807177+00:00 [info] <0.762.0> Running boot step rabbit_vhost_limit defined by app rabbit
2024-12-02 19:08:21.807259+00:00 [info] <0.762.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2024-12-02 19:08:21.807327+00:00 [info] <0.762.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2024-12-02 19:08:21.807372+00:00 [info] <0.762.0> Management plugin: using rates mode 'basic'
2024-12-02 19:08:21.807603+00:00 [info] <0.762.0> Running boot step recovery defined by app rabbit
2024-12-02 19:08:21.808187+00:00 [info] <0.762.0> Running boot step empty_db_check defined by app rabbit
2024-12-02 19:08:21.808238+00:00 [info] <0.762.0> Will seed default virtual host and user...
2024-12-02 19:08:21.808366+00:00 [info] <0.762.0> Adding vhost '/' (description: 'Default virtual host', tags: [])
2024-12-02 19:08:21.826859+00:00 [info] <0.1078.0> Making sure data directory '/var/lib/rabbitmq/mnesia/node-01@ip-10-0-134-178.eu-west-1.compute.internal/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2024-12-02 19:08:21.827671+00:00 [info] <0.1078.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
2024-12-02 19:08:21.831305+00:00 [info] <0.1078.0> Starting message stores for vhost '/'
2024-12-02 19:08:21.832349+00:00 [info] <0.1078.0> Started message store of type transient for vhost '/'
2024-12-02 19:08:21.832812+00:00 [warning] <0.1092.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2024-12-02 19:08:21.833510+00:00 [info] <0.1078.0> Started message store of type persistent for vhost '/'
2024-12-02 19:08:21.833639+00:00 [info] <0.1078.0> Recovering 0 queues of type rabbit_classic_queue took 5ms
2024-12-02 19:08:21.833676+00:00 [info] <0.1078.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2024-12-02 19:08:21.833723+00:00 [info] <0.1078.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2024-12-02 19:08:21.836950+00:00 [info] <0.762.0> Created user 'guest'
2024-12-02 19:08:21.839120+00:00 [info] <0.762.0> Successfully set user tags for user 'guest' to [administrator]
2024-12-02 19:08:21.841384+00:00 [info] <0.762.0> Successfully set permissions for user 'guest' in virtual host '/' to '.*', '.*', '.*'
2024-12-02 19:08:21.841442+00:00 [info] <0.762.0> Running boot step rabbit_observer_cli defined by app rabbit
2024-12-02 19:08:21.841606+00:00 [info] <0.762.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2024-12-02 19:08:21.841842+00:00 [info] <0.762.0> Running boot step background_gc defined by app rabbit
2024-12-02 19:08:21.842032+00:00 [info] <0.762.0> Running boot step routing_ready defined by app rabbit
2024-12-02 19:08:21.842077+00:00 [info] <0.762.0> Running boot step pre_flight defined by app rabbit
2024-12-02 19:08:21.842166+00:00 [info] <0.762.0> Running boot step notify_cluster defined by app rabbit
2024-12-02 19:08:21.842254+00:00 [info] <0.762.0> Running boot step networking defined by app rabbit
2024-12-02 19:08:21.842304+00:00 [info] <0.762.0> Running boot step rabbit_quorum_queue_periodic_membership_reconciliation defined by app rabbit
2024-12-02 19:08:21.842556+00:00 [info] <0.762.0> Running boot step definition_import_worker_pool defined by app rabbit
2024-12-02 19:08:21.842656+00:00 [info] <0.815.0> Starting worker pool 'definition_import_pool' with 2 processes in it
2024-12-02 19:08:21.842953+00:00 [info] <0.762.0> Running boot step cluster_name defined by app rabbit
2024-12-02 19:08:21.843042+00:00 [info] <0.762.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-yiWcNWgkcVRfo8BepYIhTw'
2024-12-02 19:08:21.845255+00:00 [info] <0.762.0> Running boot step virtual_host_reconciliation defined by app rabbit
2024-12-02 19:08:21.845437+00:00 [info] <0.762.0> Running boot step direct_client defined by app rabbit
2024-12-02 19:08:21.845524+00:00 [info] <0.762.0> Running boot step cluster_tags defined by app rabbit
2024-12-02 19:08:21.847869+00:00 [info] <0.762.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2024-12-02 19:08:21.848009+00:00 [info] <0.1127.0> Resetting node maintenance status
2024-12-02 19:08:21.902990+00:00 [info] <0.1184.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2024-12-02 19:08:21.903181+00:00 [info] <0.1212.0> Statistics database started.
2024-12-02 19:08:21.903284+00:00 [info] <0.1211.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2024-12-02 19:08:21.903711+00:00 [info] <0.1127.0> Ready to start client connection listeners
2024-12-02 19:08:21.905204+00:00 [info] <0.1236.0> started TCP listener on [::]:5672
2024-12-02 19:08:21.982903+00:00 [info] <0.1127.0> Server startup complete; 3 plugins started.
2024-12-02 19:08:21.982903+00:00 [info] <0.1127.0>  * rabbitmq_management
2024-12-02 19:08:21.982903+00:00 [info] <0.1127.0>  * rabbitmq_management_agent
2024-12-02 19:08:21.982903+00:00 [info] <0.1127.0>  * rabbitmq_web_dispatch
2024-12-02 19:08:22.048507+00:00 [info] <0.721.0> Time to start RabbitMQ: 682 ms

Logs from node 2 (if applicable, with sensitive values edited out)

2024-12-02 19:18:12.161266+00:00 [info] <0.805.0> DB: checking if `node-02@ip-10-0-156-210.eu-west-1.compute.internal` can join cluster using remote node `node-01@ip-10-0-134-178.eu-west-1.compute.internal`
2024-12-02 19:18:12.161408+00:00 [notice] <0.805.0> Feature flags: checking nodes `node-02@ip-10-0-156-210.eu-west-1.compute.internal` and `node-01@ip-10-0-134-178.eu-west-1.compute.internal` compatibility...
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags: error while running:
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:   rabbit_ff_controller:running_nodes[]
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags: on node `node-01@ip-10-0-134-178.eu-west-1.compute.internal`:
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:   exception error: {erpc,noconnection}
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:     in function  erpc:call/5 (erpc.erl, line 1376)
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:     in call from rabbit_ff_controller:rpc_call/5 (rabbit_ff_controller.erl, line 1394)
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:     in call from rabbit_ff_controller:list_nodes_clustered_with/1 (rabbit_ff_controller.erl, line 486)
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:     in call from rabbit_ff_controller:check_node_compatibility_task/2 (rabbit_ff_controller.erl, line 398)
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:     in call from rabbit_db_cluster:can_join/1 (rabbit_db_cluster.erl, line 60)
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:     in call from rabbit_db_cluster:join/2 (rabbit_db_cluster.erl, line 92)
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> Feature flags:     in call from erpc:execute_call/4 (erpc.erl, line 1250)
2024-12-02 19:18:19.175451+00:00 [error] <0.805.0> 
2024-12-02 19:18:19.176119+00:00 [warning] <0.805.0> Feature flags: error while querying cluster members from node `node-01@ip-10-0-134-178.eu-west-1.compute.internal`:
2024-12-02 19:18:19.176119+00:00 [warning] <0.805.0> {error,{erpc,noconnection}}

Logs from node 3 (if applicable, with sensitive values edited out)

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

# not found / not set

Steps to deploy RabbitMQ cluster

Follow https://www.rabbitmq.com/docs/install-rpm to install rabbitmq on 3 nodes. Do not start rabbitmq on any nodes, just install and then leave alone.

On one of the nodes, choose randomly, do the following

run cat /etc/hosts/.erlang.cookie and save output somewhere
run sudo nano /etc/rabbitmq/rabbitmq-env.conf and place the following in the file:
NODENAME=node-01
USE_LONGNAME=true

Continue on node-01 machine and run
run systemctl restart rabbitmq-server
run rabbitmqctl start_app
run rabbitmqctl cluster_status and make a note of the fully qualified node name

On the other 2 machines/nodes do the following
run /etc/hosts/.erlang.cookie and replace the value with the erlang.cookie you saved from node-01 above
run sudo nano /etc/rabbitmq/rabbitmq-env.conf and insert the following content into the file:
NODENAME=<node-02/node-03>
USE_LONGNAME=true

run systemctl restart rabbitmq-server
run rabbitmqctl stop_app
run rabbitmqctl reset
run cat /etc/hosts/.erlang.cookie and confirm one last time the value is the same as node-01 erlang cookie from the first step above

finally, if the erlang cookie is the same run
rabbitmqctl join_cluster

Steps to reproduce the behavior in question

Same steps as above section. The problem occurs after normal installation. On a fresh install with no changes except replacing the erlang cookie with the same from node-01. And also adding a rabbitmq-env.conf file with a node name and set use-longname to true.

What problem are you trying to solve?

During a fresh install on ec2 machines following the rabbitmq rpm install guide online, with version 4.0.4 on all machines (erlang v27) i am getting an error when joining cluster. From the logs pasted above it seems an exception is thrown during the feature flag compatibility check. The logs are available above.

Basically running node-02 join_cluster node-03 errors out.
All nodes were setup at the same time with the same versions.
Security groups are all configured correctly.
Rabbitmq installed correctly on all nodes and can be started independantly with no issues.

The issue occurs when trying to cluster. Here is an output of the join cluster command immediately followed the the last output of logs showing the issue.

Answered by Hussain-f

Dec 3, 2024

Hi both.
I profusely apologise for having wasted your time. I can confirm the issue is resolved and clustering has been achieved.

Port 25672 was not opened.

Again, my sincere apologies.

View full answer

lukebakken · 2024-12-02T20:44:36Z

lukebakken
Dec 2, 2024
Maintainer

My guess is that, since you are using long node names, the value of NODENAME must be the long name, i.e. node-01@ip-x-x-x-x.eu-west-1.compute.internal. Since that name contains the node IP, it's not reliable.

Rather than form your clusters manually, use the AWS peer discovery plugin:
https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-aws

1 reply

Hussain-f Dec 2, 2024
Author

Hi
your suggested solution unfortunately does not work and results in the same output.
I have ensured all nodes have the fully qualified node name in the rabbitmq-env.conf file. I then restart the servers too using systemctl restart rabbitmq-server.

The logs on node-02 have one exrtra line this time though. The extra line indicates an infinite loop.

I am aware of the peer discovery mechanisms however I would still like to get this working before considering that.

lukebakken · 2024-12-02T21:42:45Z

lukebakken
Dec 2, 2024
Maintainer

The solution is to use the AWS peer discovery plugin.

You haven't indicated what network troubleshooting steps you've taken - https://www.rabbitmq.com/docs/troubleshooting-networking

I'll re-open this issue long enough to ensure that long name clustering works with 4.0.4 in my local test environment.

5 replies

Hussain-f Dec 2, 2024
Author

I wasn't aware long name clustering was not an option, and therefore I am not sure if it is worth even performing network troubleshooting. I'll move over to AWS peer discovery.
Thanks for your time much appreciated.

lukebakken Dec 2, 2024
Maintainer

I wasn't aware long name clustering was not an option, and therefore I am not sure if it is worth even performing network troubleshooting.

Yes, of course you can cluster using long names. This is why I said I'm going to verify that something hasn't broken with long name clustering using version 4.0.4.

There is a slim chance you have found a bug, however, the issue probably lies in your environment.

Hussain-f Dec 2, 2024
Author

I have started looking into the network troubleshooting. Immediately I noticed I do not have any config files in place. When running cluster_status the config directory is empty. Maybe this could be the issue. Ill give it a go. Do I have to ensure each node in the cluster has the same content in their respective config files? I am going to use classic config to try get cluster long name working before moving onto AWS peer discovery.

cluster_formation.peer_discovery_backend = classic_config

# the backend can also be specified using its module name
# cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config

cluster_formation.classic_config.nodes.1 = rabbit@hostname1.eng.example.local
cluster_formation.classic_config.nodes.2 = rabbit@hostname2.eng.example.local

With it being the latest version, I couldn't find anything related online, and I found through the rabbitmq docs that V4 has some feature flags enabled. The log output showed an error during a compatibility feature flag check so I thought maybe this could be related to the new feature flags in V4.

lukebakken Dec 2, 2024
Maintainer

I created the following project to demonstrate forming a cluster using version 4.0.4 (compiled from source):

https://github.com/lukebakken/rabbitmq-server-12877

It is based on this project, which uses peer discovery:

https://github.com/lukebakken/rabbitmq-server-11534

Both projects work correctly.

I'm not sure what's going on in your environment. You should only need an /etc/rabbitmq/rabbitmq-env.conf file on each node, with content like this:

USE_LONGNAME=true
NODENAME=node-0X.a.b.c.d.internal

NOTE: each node MUST be able to resolve the other two node names using DNS.

Then, ensure the /var/lib/rabbitmq/.erlang.cookie file is identical on all nodes. Use sha256sum to be certain.

Then, start all three RabbitMQ nodes from scratch (i.e., rm -rf /var/lib/rabbitmq/* first`).

Then, on node-02, run the following command:

rabbitmqctl stop_app
rabbitmqctl join_cluster node-01.a.b.c.d.internal
rabbitmqctl start_app

Then, on node-03, run the same command as above. There is no need to stop/start RabbitMQ using systemctl.

michaelklishin Dec 2, 2024
Maintainer

I just want to add that clustering requirements are documented in [1][2], and according to user feedback, nothing obvious is missing there.

Hussain-f · 2024-12-03T13:06:57Z

Hussain-f
Dec 3, 2024
Author

Thanks for your help however I completely ripped up my ec2 instances and started from scratch following the instructions to the letter, and I am getting the exact same error.

@lukebakken you mentioned each node only needs a rabbitmq-env.conf file (this indicates a config file is not needed). According to the cluster formation and peer discovery guide, a config file containing the AWS access/secret key is needed as well as the cluster_formation.peer_discovery_backend variable needing set. Can you confirm if the config file is not needed because the rabbitmq docs suggest it is.

@lukebakken @michaelklishin The erpc no connection error, would you happen to know what this is related to? Below is the output if rabbitmq-diagnostics status on node1

Status of node node-01@ip-10-0-142-252.eu-west-1.compute.internal ...
[]
Runtime

OS PID: 30017
OS: Linux
Uptime (seconds): 338
Is under maintenance?: false
RabbitMQ version: 4.0.4
RabbitMQ release series support status: see https://www.rabbitmq.com/release-information
Node name: node-01@ip-10-0-142-252.eu-west-1.compute.internal
Erlang configuration: Erlang/OTP 27 [erts-15.1.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.0.8 7 Feb 2023
Erlang processes: 302 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_peer_discovery_aws
 * rabbitmq_peer_discovery_common
 * rabbitmq_aws

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/node-01@ip-10-0-142-252.eu-west-1.compute.internal
Raft data directory: /var/lib/rabbitmq/mnesia/node-01@ip-10-0-142-252.eu-west-1.compute.internal/quorum/node-01@ip-10-0-142-252.eu-west-1.compute.internal

Config files

 * /etc/rabbitmq/rabbitmq.conf

Log file(s)

 * /var/log/rabbitmq/node-01@ip-10-0-142-252.eu-west-1.compute.internal.log
 * <stdout>

Alarms

(none)

Tags

(none)

Memory

Total memory used: 0.1117 gb
Calculation strategy: rss
Memory high watermark setting: 0.6 of available memory, computed to: 1.1963 gb

reserved_unallocated: 0.0528 gb (47.29 %)
code: 0.023 gb (20.59 %)
other_system: 0.0177 gb (15.89 %)
other_proc: 0.0157 gb (14.07 %)
other_ets: 0.0025 gb (2.21 %)
atom: 0.0011 gb (1.02 %)
metrics: 0.0006 gb (0.57 %)
binary: 0.0003 gb (0.27 %)
msg_index: 0.0001 gb (0.11 %)
metadata_store: 0.0001 gb (0.1 %)
mnesia: 0.0001 gb (0.07 %)
plugins: 0.0 gb (0.02 %)
quorum_ets: 0.0 gb (0.02 %)
metadata_store_ets: 0.0 gb (0.01 %)
quorum_queue_procs: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_other: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)
mgmt_db: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)

File Descriptors

Total: 0, limit: 32671

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 30.2209 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

I have done a fresh install as per the rabbitmq RPM install docs.
I ensured erlang cookie is same on all nodes
I added rabbitmq-env.conf to /etc/rabbitmq/ containing NODENAME, CONFIG_FILE=/etc/rabbitmq/rabbitmq.conf and USE_LONGNAME=true
I enabled the aws peer discovery plugin on all nodes
I added a config file at /etc/rabbitmq/rabbitmq.conf on each node with the below:

cluster_formation.peer_discovery_backend = aws
cluster_formation.aws.region = eu-west-1
cluster_formation.aws.use_autoscaling_group = true # in order to make sure our cluster is grouped under autoscaling group we will enable this.
cluster_formation.discovery_retry_limit = 10
cluster_formation.discovery_retry_interval = 10000

cluster_formation.aws.access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
cluster_formation.aws.secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

All nodes can start independantly but when calling rabbitmq join_cluster node-01@x.x.x.x.x.internal I get the same error I have been getting since the beginning.

6 replies

Hussain-f Dec 3, 2024
Author

below is the ports that have been opened for the machine in AWS. Note some are for internal use, but as you can see all rabbitmq related ports have been accounted for.

michaelklishin Dec 3, 2024
Maintainer

At no point I see EC2 security groups and opening up the ports necessary for inter-node communication mentioned.

Starting with https://www.rabbitmq.com/docs/networking#ports and specifically the epmd and the inter-node communication port.
See https://www.rabbitmq.com/docs/troubleshooting-networking and take a traffic capture when in doubt (on both nodes), and then inspect it with Wireshark.

Our community policy clearly states that we will not troubleshoot networking for non-paying, non-contributing users.

Despite this, you have received well more than a list of relevant links.

michaelklishin Dec 3, 2024
Maintainer

Nodes and CLI tools discover each other's port via 4369 but then communicate on the discovered port, which is by default, and almost always in practice, is 25672. If that port is blocked — and it is not on the list above — nodes won't be able to communicate.

Hussain-f Dec 3, 2024
Author

Hi both.
I profusely apologise for having wasted your time. I can confirm the issue is resolved and clustering has been achieved.

Port 25672 was not opened.

Again, my sincere apologies.

Answer selected by michaelklishin

michaelklishin Dec 3, 2024
Maintainer

Glad you were able to figure it out. I guess the docs need a very visible remark about the ports somewhere in the clustering pre-requisites section.

Hussain-f Dec 3, 2024
Author

tbf it was me just not looking at the basics. I thought I had got all the ports down, but I guess I missed one. Your docs are quite clear.
Much obliged.

lukebakken · 2024-12-03T16:52:48Z

lukebakken
Dec 3, 2024
Maintainer

you mentioned each node only needs a rabbitmq-env.conf file (this indicates a config file is not needed). According to the cluster formation and peer discovery guide, a config file containing the AWS access/secret key is needed as well as the cluster_formation.peer_discovery_backend variable needing set.

I thought you were still doing "plain jane" clustering, and NOT using the AWS peer discovery backend.

When asking for support, it's really important to be crystal clear what you are attempting, and, if you move on to something else, let the people giving you free support know.

I'm glad you figured the issue out. It turns out my original suggestion, to do the network troubleshooting, was correct.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cluster formation: feature flag node compatibility check runs into an erpc no connection error #12877

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 12 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Cluster formation: feature flag node compatibility check runs into an erpc no connection error #12877

Hussain-f Dec 2, 2024

Community Support Policy

RabbitMQ version used

Erlang version used

Operating system (distribution) used

How is RabbitMQ deployed?

Logs from node 1 (with sensitive values edited out)

Logs from node 2 (if applicable, with sensitive values edited out)

Logs from node 3 (if applicable, with sensitive values edited out)

rabbitmq.conf

Steps to deploy RabbitMQ cluster

Steps to reproduce the behavior in question

What problem are you trying to solve?

Replies: 4 comments · 12 replies

lukebakken Dec 2, 2024 Maintainer

Hussain-f Dec 2, 2024 Author

lukebakken Dec 2, 2024 Maintainer

Hussain-f Dec 2, 2024 Author

lukebakken Dec 2, 2024 Maintainer

Hussain-f Dec 2, 2024 Author

lukebakken Dec 2, 2024 Maintainer

michaelklishin Dec 2, 2024 Maintainer

Hussain-f Dec 3, 2024 Author

Hussain-f Dec 3, 2024 Author

michaelklishin Dec 3, 2024 Maintainer

michaelklishin Dec 3, 2024 Maintainer

Hussain-f Dec 3, 2024 Author

michaelklishin Dec 3, 2024 Maintainer

Hussain-f Dec 3, 2024 Author

lukebakken Dec 3, 2024 Maintainer

Hussain-f
Dec 2, 2024

Replies: 4 comments 12 replies

lukebakken
Dec 2, 2024
Maintainer

Hussain-f Dec 2, 2024
Author

lukebakken
Dec 2, 2024
Maintainer

Hussain-f Dec 2, 2024
Author

lukebakken Dec 2, 2024
Maintainer

Hussain-f Dec 2, 2024
Author

lukebakken Dec 2, 2024
Maintainer

michaelklishin Dec 2, 2024
Maintainer

Hussain-f
Dec 3, 2024
Author

Hussain-f Dec 3, 2024
Author

michaelklishin Dec 3, 2024
Maintainer

michaelklishin Dec 3, 2024
Maintainer

Hussain-f Dec 3, 2024
Author

michaelklishin Dec 3, 2024
Maintainer

Hussain-f Dec 3, 2024
Author

lukebakken
Dec 3, 2024
Maintainer