Conversion getting stuck between cqv1 and cqv2

[johanrhodin]

This discussion is to collect evidence of circumstances of queues getting stuck in the conversion from classic queues version one to version two (cqv1->cqv2).

When this conversion gets stuck you'd see something like this in the observer, the message queue for the queue process (in this case two of them) would collect a MsgQueue that doesn't get shorter (fast), increases or stays the same, over a longer period of time.

The stack trace for the process(es) would look something like this:

In the log you'd see something along the lines of:

2024-02-29 14:51:41.144702+00:00 [info] <0.7098.0> Converting running queue myqueue in vhost myvhost from v1 to v2
2024-02-29 14:51:41.148906+00:00 [info] <0.7098.0> Queue myqueue in vhost myvhost converted 3 messages from v1 to v2
2024-02-29 14:51:41.149445+00:00 [info] <0.7098.0> Queue myqueue in vhost myvhost converted 3 total messages from v1 to v2

In this case above it was Erlang 25.3.2.7 and Rabbitmq 3.12.10, queues were short (less than 100 messages) and had a policy of "ha-mode":"exactly", "ha-params":1 before conversion.

We'll add to this discussion when we find additional cases of stuck conversions, and would appreciate if other users would also add their cases.

[michaelklishin]

From 3.13.0 release notes:

Upgrading

…

classic queues to CQv2 at boot time using the configuration file setting can be
potentially unsafe in environments where deprecated classic mirrored queues still exist.

Classic mirrored queues cannot be safely upgraded to CQv2. One replica may be upgraded but nothing
prevents the rest of trying to go back to CQv1 in certain scenarios. Which is why 3.13.0
release notes explicitly say "non-mirrored classic queues" in most places where CQv2 are mentioned.

We won't spend time on classic mirrored queues in general. They are on their way out later this year.

[michaelklishin]

By the way, the original plan for 3.13.0 was to make CQv2 the default. But after observing similar behavior we have reverted the default and only recommend upgrading to CQv2 for non-mirrored CQs and after all cluster nodes have been upgraded to the same version (that is, not using rabbitmq.conf at first unless it's a new cluster).

In 4.0 this won't be an issue because classic queue mirroring will not exist. Well, neither will CQv1.