Management Plugin Web UI shows an error when accessing connection details view

[eisendle]

Describe the bug

When a RabbitMQ cluster is configured with management.disable_stats = true, trying to access connection details in the Web UI causes a Management API returned status code 500 - to be shown.

Reproduction steps

1. Enable rabbitmq_management plugin
2. Add management.disable_stats = true to the cluster configuration
3. Access the management web interface connection overview at http://[host]:15671/connections
4. Click a connection to show connection details.
5. A Management API returned status code 500 - is shown

Error message from RabbitMQ log:

2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> crasher:
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> initial call: cowboy_stream_h:request_process/3
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> pid: <0.50215.0>
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> registered_name: []
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> exception error: no case clause matching
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> {badrpc,
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> {'EXIT',
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> {undef,
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> [{rabbit_connection_tracking,lookup,
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> [<<"x.x.x.x:17638 -> x.x.x.x:5671">>,
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> ['rabbit@rmq-cluster-server-4.xxx']],
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> []}]}}}
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> in function rabbit_connection_tracking:lookup/2 (rabbit_connection_tracking.erl, line 235)
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> in call from rabbit_mgmt_wm_connection:conn/1 (rabbit_mgmt_wm_connection.erl, line 76)
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> in call from rabbit_mgmt_wm_connection:is_authorized/2 (rabbit_mgmt_wm_connection.erl, line 63)
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> in call from cowboy_rest:call/3 (src/cowboy_rest.erl, line 1590)
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> in call from cowboy_rest:is_authorized/2 (src/cowboy_rest.erl, line 368)
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> in call from cowboy_rest:upgrade/4 (src/cowboy_rest.erl, line 284)
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> in call from cowboy_stream_h:execute/3 (src/cowboy_stream_h.erl, line 306)
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> in call from cowboy_stream_h:request_process/3 (src/cowboy_stream_h.erl, line 295)
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> ancestors: [<0.50214.0>,<0.11454.0>,<0.11446.0>,<0.11445.0>,<0.11443.0>,
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> rabbit_web_dispatch_sup,<0.11192.0>]
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> message_queue_len: 0
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> messages: []
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> links: [<0.50214.0>]
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> dictionary: []
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> trap_exit: false
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> status: running
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> heap_size: 1598
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> stack_size: 28
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> reductions: 2115
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0> neighbours:
2024-08-07 07:41:09.343841+00:00 [error] <0.50215.0>
2024-08-07 07:41:09.344444+00:00 [error] <0.50214.0> Ranch listener {acceptor,{0,0,0,0,0,0,0,0},15671}, connection process <0.50214.0>, stream 1 had its request process <0.50215.0> exit with reason {case_clause,{badrpc,{'EXIT',{undef,[{rabbit_connection_tracking,lookup,[<<"x.x.x.x:17638 -> x.x.x.x:5671">>,['rabbit@rmq-cluster-server-4.xxx']],[]}]}}}} and stacktrace [{rabbit_connection_tracking,lookup,2,[{file,"rabbit_connection_tracking.erl"},{line,235}]},{rabbit_mgmt_wm_connection,conn,1,[{file,"rabbit_mgmt_wm_connection.erl"},{line,76}]},{rabbit_mgmt_wm_connection,is_authorized,2,[{file,"rabbit_mgmt_wm_connection.erl"},{line,63}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,1590}]},{cowboy_rest,is_authorized,2,[{file,"src/cowboy_rest.erl"},{line,368}]},{cowboy_rest,upgrade,4,[{file,"src/cowboy_rest.erl"},{line,284}]},{cowboy_stream_h,execute,3,[{file,"src/cowboy_stream_h.erl"},{line,306}]},{cowboy_stream_h,request_process,3,[{file,"src/cowboy_...
2024-08-07 07:41:09.344444+00:00 [error] <0.50214.0>

Expected behavior

Detail information about the connection is shown (e.g client properties).

Additional context

Deployment: 5 node RabbitMQ cluster deployed in Openshift
Version: 3.13.6

[michaelklishin]

I cannot reproduce. According to the exception, rabbit_connection_tracking:lookup/2 is a non-existing function but it surely does exist in 3.13.6.

[michaelklishin]

OK, the issue comes down to the fact that the node your browser is contacting cannot contact a peer that hosts the connection according to the connection tracking data. Something in your cluster prevents (or prevented in that particular moment) at least two nodes from communicating with each other.

While this error can be handled better, the server would at best return a 404 and I don't think it would be a massive improvement. I'm afraid our team has much larger priorities.

[eisendle]

I've tested this again, this time the cluster was completely idle to rule out any RPC timeouts due to network utilization. if management.disable_stats = true is set, the error occurs, if management.disable_stats = false everything works as expected.

I don't see any other inter-node communication related errors in the log files. As my company is eligible for enterprise support, I'll open a Broadcom ticket.

[michaelklishin]

Sorry, I only trust exceptions in the logs, not users' gut feelings.

If you disable stats then certain kinds of inter-node communication won't happen at all.

[michaelklishin]

@eisendle you are welcome to open a ticket but assuming that you don't have any new logs or metrics, in response you will be told to investigate what's going on with inter-node communication in this cluster. The exception is as specific as a single snippet from the logs can be.

[eisendle]

Sorry, I only trust exceptions in the logs, not users' gut feelings.

If you disable stats then certain kinds of inter-node communication won't happen at all.

Does this mean when management.disable_stats = true, connection details are not supposed to work for a multi-node cluster? I've tested this on a single node cluster on Windows and the connection details are working fine. From a code perspective this should work (correct me if I'm wrong, I'm not fluent with Erlang) as the code distinguishes whether disable_stats is true or false.

@eisendle you are welcome to open a ticket but assuming that you don't have any new logs or metrics, in response you will be told to investigate what's going on with inter-node communication in this cluster. The exception is as specific as a single snippet from the logs can be.

This comment suggests that it's a transient problem with our cluster setup.

I would appreciate a clear explanation regarding whether connection details are intended to function in a multi-node cluster when stats are disabled.

[michaelklishin]

I did not say that with management.disable_stats = true connection tracking is disabled. I only said that management.disable_stats = true disables metric collection and thus certain kinds of inter-node communication will not happen.

Accessing connection details on the local node, and using a single node in general, is not a comparable scenario to using a cluster of N nodes and looking up details of connection on remote node.

What node are you trying to list this connection on, and what node is this client connected to? (it should be possible to tell from the connection list and the IP address of the node, if anything, using CLI tools rabbitmq-diagnostics list_connections on every node.