QQ gets into inconsistent state by add_member in pre_vote state

[gomoripeti]

A quorum queue crashes and gets into an inconsistent state regarding its members when a member change is attempted while there is no quorum and no elected leader.

RabbitMQ version main at ed7f5c2 with ra v2.4.1. (Was seen with 3.10.x versions as well)

Steps to reproduce: 3 node cluster with 1 quorum queue leader on node 2

1. stop_app on a follower 3
2. delete_member on leader node 2
3. add_member same node 2 (this hangs indefinitely waiting for a leader),
4. start_app on node 2 -> hanging add_member call crashes, node 2 still not member

shell log

$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r2@localhost              │ leader     │ 2         │ 2            │ undefined      │ 1    │ 3               │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ follower   │ 2         │ 2            │ undefined      │ 1    │ 3               │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ follower   │ 2         │ 2            │ undefined      │ 1    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘
$ sbin/rabbitmqctl --node r3 stop_app
Stopping rabbit application on node r3@localhost...
$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r2@localhost              │ leader     │ 2         │ 2            │ undefined      │ 1    │ 3               │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ noproc     │           │              │                │      │                 │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ follower   │ 2         │ 2            │ undefined      │ 1    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘
$ ./sbin/rabbitmq-queues delete_member qq4 r2@localhost
Removing a replica of queue qq4 on node r2@localhost...
$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ noproc     │           │              │                │      │                 │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ pre_vote   │ 3         │ 2            │ undefined      │ 1    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘
$ ./sbin/rabbitmq-queues add_member qq4 r2@localhost
Adding a replica for queue qq4 on node r2@localhost...

<...hangs until start_app...>

Error:
{{{:badkey, :previous_cluster}, [{:erlang, :map_get, [:previous_cluster, %{aux_state:

in another shell:

$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ noproc     │           │              │                │      │                 │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ pre_vote   │ 3         │ 2            │ undefined      │ 1    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘
$ sbin/rabbitmqctl --node r3 start_app
Starting node r3@localhost...
$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost ...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ leader     │ 3         │ 3            │ undefined      │ 2    │ 3               │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ follower   │ 3         │ 3            │ undefined      │ 2    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

At this point the ra server processes are up on all 3 nodes but the queue entry in rabbit_queue table only knows about node 1 and 3.

trying add member again fails.

./sbin/rabbitmq-queues add_member qq4 r2@localhost
Adding a replica for queue qq4 on node r2@localhost...
Error:
{:shutdown, {:failed_to_start_child, :"%2F_qq4", {:already_started, #PID<12602.1338.0>}}}

node 1 log: qq_add_member_crash.txt

[lukebakken]

I'm curious if this represents a real-world issue or just some testing. It seems like a Bad Idea to stop a raft member and then do membership changes. I'm sure @kjnilsson will have more to ask/say.

[gomoripeti]

I tried to come up with a reproducible list of steps. The real world issue would be a race condition between a raft membership change and a node crashing and restarting meanwhile. (If I try to check if a raft cluster/quorum queue is healthy before I do a membership change operation, a failure can still happen in between.)

The crash seems to be similar to rabbitmq/ra#33 but that should be fixed long ago.

[michaelklishin]

If there is no quorum and elected leader you cannot perform cluster membership changes since they go through Raft log.

Ra intentionally limits cluster membership changes to one at a time (so do some other Raft implementations).

So this is at least in part by design.

[gomoripeti]

sorry I was not clear about the expected behaviour

It is totally fine and expected that when there is no quorum/no leader cluster membership changes are either rejected or pending until later (the later is observed in this case)

However I would expect that when quorum is restored, the pending change is either rejected cleanly or applied. In this case it causes a crash and although it looks like one node is not a member, it is not possible to add it.

[michaelklishin]

Accumulating membership change operations until after the quorum is restored violates two ideas in Ra:

• that clusters without a quorum won't support any write operations, including membership changes
• that only one change is applied at a time

The only exception here, which at some point will be addressed, is a forced boot of a single node so that at a cluster can be "restored" by replacing all nodes but one from the original cluster.

[gomoripeti]

I can fully accept those main ideas but in that case operations that violate those principles should be rejected. It would be very helpful if the rabbitmq-queues commands (add_member, delete_member, grow, shrink) would return an error immediately. (I intentionally mention the rabbitmq cli, and not lower levels of abstraction like the Ra library, as I am not familiar with the implementation. But as an operator who manually executes those commands or from a script an error return would be very helpful)

[kjnilsson]

I think the issue we see here is one of unclear sources of truth. One may think that mnesia is the source of truth in RabbitMQ and for classic queues that is the case mostly but for quorum queues and streams mnesia is more of a view of the source of truth. In this case the source of truth of the actual QQ membership is the quorum queue itself, not mnesia and in this instance, because the membership change did not complete in the QQ the two have diverged.

that is why we have functions like this one where we periodically check if the actual leader has diverged from what is in mnesia and then updates mnesia as necessary. I think we need to also check that the configured nodes are in sync in order to repair a diverged situation.

[gomoripeti]

absolutely.

I also see repair_amqqueue_nodes, but I don't see it called from anywhere (still handy for debugging)

I'd like to mention that we've also seen discrepancies between membership in the quorum queue and on which nodes the ra_server processes are actually running. ( between ra:start_server and ra:add_member). For example a server was running but not a member, resulting in a {:failed_to_start_child, :..., {:already_started, #PID<12354.17656.0>}}} error.

First I thought that a fix would prevent such inconsistencies from happening, but maybe a more fool-proof approach is to make rabbit_quorum_queue:add_member/delete_member more idempotent (after the source of truth is clarified)

[kjnilsson]

There may well be stuff we can do to make member changes better in Ra itself but I think from a RabbitMQ's POV it would be valuable to include nodes repair in the regular repair that is done.

[kjnilsson]

the obvious change in ra woudl be to change the add_member command to await_consensus rather than replying as soon as the entry as been written to the log.

[tvhong-amazon]

I also saw a similar issue in 3.10.10 while testing. Here are some observations:

mnesia shows 3 members:

$ rabbitmqctl eval "{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<\"/\">>, queue,
> <<\"test-queue-1\">>)), Q."
{amqqueue,{resource,<<"/">>,queue,<<"test-queue-1">>},
          true,false,none,
          [{<<"x-queue-type">>,longstr,<<"quorum">>}],
          {'%2F_test-queue-1','rabbit@ip-10-0-4-12.us-west-2.compute.internal'},
          [],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,
          #{user => <<"admin">>},
          rabbit_quorum_queue,
          #{nodes =>
                ['rabbit@ip-10-0-12-108.us-west-2.compute.internal',
                 'rabbit@ip-10-0-20-16.us-west-2.compute.internal',
                 'rabbit@ip-10-0-4-12.us-west-2.compute.internal']}}

But ra shows only 2 members

$ rabbitmqctl eval "QName = rabbit_misc:r(<<\"/\">>, queue, <<\"test-queue-1\">>), {ok, Q0} = rabbit_amqqueue:lookup(QName), Leader = amqqueue:get_pid(Q0), {ok, Members, _} =ra:members(Leader), Members."

[{'%2F_test-queue-1','rabbit@ip-10-0-20-16.us-west-2.compute.internal'},
 {'%2F_test-queue-1','rabbit@ip-10-0-4-12.us-west-2.compute.internal'}]

rabbitmq-queues quorum_status output:

┌──────────────────────────────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                                        │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├──────────────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@ip-10-0-12-108.us-west-2.compute.internal │ follower   │ 978432    │ 28917882     │ undefined      │ 6    │ 2               │
├──────────────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@ip-10-0-20-16.us-west-2.compute.internal  │ follower   │ 28917883  │ 28917883     │ undefined      │ 6    │ 2               │
├──────────────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@ip-10-0-4-12.us-west-2.compute.internal   │ leader     │ 28917883  │ 28917883     │ undefined      │ 6    │ 2               │
└──────────────────────────────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

rabbitmq-queues delete_queue output:

rabbitmq-queues delete_member test-queue-1 rabbit@ip-10-0-12-108.us-west-2.compute.internal
Removing a replica of queue test-queue-1 on node rabbit@ip-10-0-12-108.us-west-2.compute.internal...
Error:
{:no_more_servers_to_try, [error: :not_member, error: :not_member, error: :not_member]}

---

I'm still trying to get minimal reproduction steps, but IIRC, what I did was:

1. Load a queue with ~30M messages (then stop the clients)
2. Delete a quorum member (i.e. rabbit@ip-10-0-12-108.us-west-2.compute.internal)
3. Re-add the member
4. Delete the member again while it's catching up

I repeated steps 2-4 a few times.

[kjnilsson]

yes we need to do periodic repair of the members stored in mnesia.

[lukebakken]

@michaelklishin @kjnilsson I've had this discussion in my TODO list for a while. Should we promote it to a real issue in rabbitmq-server and/or ra?

[illotum]

FWIW couldn't reproduce this issue on the current main, 4ec0ca9.

[michaelklishin]

main has periodic re-evaluation of QQ replicas #8218, as suggested above :)