CLI: ctl check_if_any_deprecated_features_are_used runs into an exception

[metron2]

Describe the bug

We had some mirroring queues when I upgraded to rabbitmq 3.13.1, so I deleted those policies but now this api call fails. We have a banner that says "Management API returned status code 500 -"

Running rabbitmqctl check_if_any_deprecated_features_are_used also returns a crash dump.

Reproduction steps

The feature was working right after the upgrade, correctly listing that we were using mirroring. I deleted those policies and that seems to have introduced this problem.

Expected behavior

The api and command should continue to work.

Additional context

rabbitmqctl check_if_any_deprecated_features_are_used
Checking if any deprecated features are used ...
Error:
{:badarg, [{:lists, :keymember, ["ha-mode", 1, %{"message-ttl" => 604800000}], [error_info: %{module: :erl_stdlib_errors}]}, {:rabbit_mirror_queue_misc, :does_policy_configure_cmq, 1, [file: ~c"rabbit_mirror_queue_misc.erl", line: 1014]}, {:lists, :any_1, 2, [file: ~c"lists.erl", line: 1543]}, {:rabbit_mirror_queue_misc, :are_cmqs_used1, 0, [file: ~c"rabbit_mirror_queue_misc.erl", line: 997]}, {:rabbit_mirror_queue_misc, :are_cmqs_used, 1, []}]}

[metron2]

We're using bitnami/rabbitmq:3.13.1-debian-12-r0 in kubernetes with the bitnami helm chart.

[michaelklishin]

It should not matter whether you run on Kubernetes or not, and whether you use Bitnami, although I'd personally highly recommend against Bitnami over very questionable decisions like bitnami/charts#16081.

In order to be able to reproduce this — so far I cannot — we need exported definitions without the sensitive values. For example, feel free to remove the users key and rename virtual hosts (consistently).

[lukebakken]

Hello, thanks for using RabbitMQ and taking the time to report this.

Could you please provide more information so I can reproduce this issue?

• What version of RabbitMQ and Erlang were you using prior to the upgrade?
• Please provide a definitions export that includes your mirroring policies that (probably) triggered this error. I could guess but it would be best to work with as close an environment as yours.

[michaelklishin]

I could not reproduce with the smallest possible CMQ policy.

[michaelklishin]

{:lists, :keymember, ["ha-mode", 1, %{"message-ttl" => 604800000} means that a function used by this operation tries to look up ha-mode is a policy definition that only has "message-ttl" => 604800000.

This was not reported elsewhere (that I've seen), so while this function can simply be made
safer:

does_policy_configure_cmq(KeyList) ->
    lists:keymember(<<"ha-mode">>, 1, KeyList).

it would be great to have a way to reproduce this with your actual policies and queues (see our recommendations above).

[michaelklishin]

Hold on, I understand what the problem is.

does_policy_configure_cmq(KeyList) ->
    lists:keymember(<<"ha-mode">>, 1, KeyList).

works perfectly fine with missing keys as long as KeyList is a proplist but in the exception, it is a map.

[metron2]

Export appears to be broken. We only get this partial response. I voted for the operator if that helps haha.

{
  "rabbit_version": "3.13.2",
  "parameters": [],
  "policies": [],
  "queues": [
    {
      "name": "log-messages",
      "durable": true,
      "auto_delete": false,
      "arguments": {}
    }
  ],
  "exchanges": [
    {
      "name": "ExGrayLogIncoming",
      "type": "x-modulus-hash",
      "durable": true,
      "auto_delete": false,
      "internal": false,
      "arguments": {}
    },
    {
      "name": "ExSpans",
      "type": "x-modulus-hash",
      "durable": true,
      "auto_delete": false,
      "internal": false,
      "arguments": {}
    }
  ],
  "bindings": [
    {
      "source": "ExGrayLogIncoming",
      "destination": "log-messages",
      "destination_type": "queue",
      "routing_key": "#",
      "arguments": {}
    }
  ]
}

[michaelklishin]

@metron2 thank you for upvoting (for others reading this: see Discord and community Slack).

Very likely you have exported definitions of a single virtual host.

You can export definitions of a single virtual host or the entire cluster, there is a switch in the UI at the top right corner. IIRC in 3.13 the export form also has a select.

I understand what the problem is and working on a PR.

[michaelklishin]

#11193 should avoid the exception.

[michaelklishin]

@metron2 can you give this OCI image tag a try? It includes #11193.

[michaelklishin]

Unfortunately no matter what I try, I cannot reproduce this exception. #11193 is not a controversial change and we can merge it anyway but without HTTP API exception logs I can only guess if it will help with both cases or just the CLI command.

[michaelklishin]

Some examples of what I have tried. Declare a policy with CMQ keys:

bazel run rabbitmqctl -- set_policy cmq "^cq\." '{"ha-mode": "all"}'

bazel run rabbitmqctl -- list_policies --formatter=pretty_table
%% ...
%% => Listing policies for vhost "/" ...
%% => ┌───────┬────────┬──────────┬───────────────┬────────────────────────────────┬──────────┐
%% => │ vhost │ name   │ pattern  │ apply-to      │ definition                     │ priority │
%% => ├───────┼────────┼──────────┼───────────────┼────────────────────────────────┼──────────┤
%% => │ /     │ cmq    │ ^cq\.    │ all           │ {"ha-mode": "all"}.            │ 0        │
%% => └───────┴────────┴──────────┴───────────────┴────────────────────────────────┴──────────┘

Run bazel run rabbitmqctl check_if_any_deprecated_features_are_used:

bazel run rabbitmqctl check_if_any_deprecated_features_are_used
%% => Checking if any deprecated features are used ...
%% => classic_queue_mirroring

Change the policy so that the CMQ keys are no longer present:

bazel run rabbitmqctl -- set_policy cmq "^cq\." '{"message-ttl": 10000000}'

bazel run rabbitmqctl -- list_policies --formatter=pretty_table

%% => Listing policies for vhost "/" ...
%% => ┌───────┬────────┬──────────┬───────────────┬────────────────────────────────┬──────────┐
%% => │ vhost │ name   │ pattern  │ apply-to      │ definition                     │ priority │
%% => ├───────┼────────┼──────────┼───────────────┼────────────────────────────────┼──────────┤
%% => │ /     │ cmq    │ ^cq\.    │ all           │ {"message-ttl":10000000}       │ 0        │
%% => └───────┴────────┴──────────┴───────────────┴────────────────────────────────┴──────────┘

Run bazel run rabbitmqctl check_if_any_deprecated_features_are_used:

bazel run rabbitmqctl check_if_any_deprecated_features_are_used
%% => Checking if any deprecated features are used ...
%% => Cluster reported no deprecated features in use

(this is on the branch of #11193)

[metron2]

Deleting the policies fixes the issue.

  "policies": [
    {
      "vhost": "dev-be-wn-encrypted",
      "name": "encrypted-limit",
      "pattern": ".*",
      "apply-to": "queues",
      "definition": {
        "max-length": 1500
      },
      "priority": 0
    },
    {
      "vhost": "dev-be-wn-encrypted",
      "name": "RetryPolicy",
      "pattern": "com.hbsi.be.decrypt.*",
      "apply-to": "queues",
      "definition": {
        "message-ttl": 600000,
        "queue-version": 2
      },
      "priority": 10
    },
    {
      "vhost": "dev-be-wn-decrypted",
      "name": "be-wn-decrypted",
      "pattern": ".*",
      "apply-to": "queues",
      "definition": {
        "max-length": 1000
      },
      "priority": 0
    },
    {
      "vhost": "dev-db-wn-encrypted",
      "name": "RetryPolicy",
      "pattern": "com.hbsi.be.decrypt.*",
      "apply-to": "queues",
      "definition": {
        "max-length": 3000,
        "message-ttl": 600000
      },
      "priority": 10
    }
  ],

[metron2]

rabbit_export_formatted.json

[metron2]

Might be the operator policy? I deleted that too

[michaelklishin]

If there also were operator policies then they also had an effect. And #11193 should help equally in those cases, see above.