Crash because of erofs on read/write filesystem

[womblep]

I am using a modified docker container 3.13 managament-alpine, rabbitmq 3.13.3/erlang 26 running in a cluster

I was experimenting in Fargate with healthchecks and had a lot of containers start and stop. This left the cluster with lots of nodes that were shutdown.

I had a couple of instances where the new node would contact the cluster, join, sync and then get a erofs error, except the directory should have been read/writable as that is setup for the mnesia files and had been working correctly for that node up to this point.

The error is below. This caused the node to start clustering again and then crash due to already being part of the cluster.

I wouldn't have worried too much but I removed all the old nodes and didn't have the issue happen again. This suggests (maybe incorrectly) that it might be number of nodes that caused the issue.

I have attached a complete log file
log-events-viewer-result - crash rabbitmq.csv

{"time":"2024-05-25 05:45:35.863754+00:00","level":"error","msg":"Mnesia('rabbit@ip-10-10-43-99'): ** ERROR ** (could not write core file: erofs)
** FATAL ** Failed to merge schema: {aborted,
{combine_error,schema,
['rabbit@ip-10-10-26-45',
'rabbit@ip-10-10-21-112',
'rabbit@ip-10-10-43-99',
'rabbit@ip-10-10-43-137',
'rabbit@ip-10-10-38-176',
'rabbit@ip-10-10-35-18',
'rabbit@ip-10-10-34-195',
'rabbit@ip-10-10-33-19',
'rabbit@ip-10-10-32-173',
'rabbit@ip-10-10-31-181',
'rabbit@ip-10-10-30-116',
'rabbit@ip-10-10-26-45',
'rabbit@ip-10-10-23-49',
'rabbit@ip-10-10-23-250',
'rabbit@ip-10-10-21-68',
'rabbit@ip-10-10-21-112',
'rabbit@ip-10-10-18-254',
'rabbit@ip-10-10-16-36']}}
","pid":"<0.1531.0>"}