Filesystem operations sometimes fail with an ebusy return code when EBS volumes are used for storage #11047
-
Describe the bugAWS EC2 Disk setup: Partitioning: Mount options: Permissions: Observed Error: Error does not occur on 100% of servers . Reproduction steps
Expected behaviorExpected behavior is a good startup with rabbitmq being able to create directories and files under /var/lib/rabbitmq/mnesia//quorum Additional contextNo response |
Beta Was this translation helpful? Give feedback.
Replies: 8 comments 14 replies
-
|
What kind of EBS volumes are you using? |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
I think @kjnilsson is wondering if they are using HDD EBS volumes - https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html |
Beta Was this translation helpful? Give feedback.
-
|
@daveofthedogs you cannot claim that an |
Beta Was this translation helpful? Give feedback.
-
|
Our Production Checklist guide explicitly suggests against using network-attached storage where possible. It is certainly possible to use local storage on AWS, including SSDs (that quorum queues, Khepri and in particular streams will greatly benefit from). This question should be directed to AWS support, and these nodes should use VM-local storage, as recommended in the docs. |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin , The documentation states this: Note, the storage I provided is:
What are the actual latency and throughput requirements of the disks? I have seen articles by Jack Vanlightly regarding message throughput and comparing hdd vs sdd and mixed loads vs single-types of workloads. These have shown SDD/Quorum workloads reaching 15ms latencies. I cannot even start RMQ I am wondering how VMWare/Pivotal handles this situation since they have moved from traditional deployments to Kubernetes, which is also not locally-attached storage under most circumstances. Below this post, I am including fio write stats running against a working install and non-working. Both are exactly the same server builds with the exception of the availability zone of the servers and ebs volumes. I am opening a ticket with AWS, but I highly suspect that locally attached storage will be cost prohibitive. Tests io2 ebs @ 1000 iops Working server (us-east-1a)Not working (us-east-1b) |
Beta Was this translation helpful? Give feedback.
-
|
Disks returning Disk I/O errors preventing nodes from starting is not a commonly reported issue, and the recommendation would generally be "use volumes that do not exhibit this behavior". RabbitMQ nodes can and usually will produce a lot of I/O reads on boot, the write I/O should be fairly limited up to the point when client listeners are enabled. Maybe an ongoing routing of disk I/O chaos testing and improvements would help with these EBS devices, or maybe not, they are not the norm across the (very diverse) customer and user base our team interacts with. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, this discussion is literally the first time I have seen @daveofthedogs any time you include the output of a command, it is essential that you also include the command invocation itself. We have no idea how you are running
Do you see this issue when you do NOT separate out disks like this? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry. I did not see the last part of your query. I will modify my terraform to only build with a large, fast root volume. |
Beta Was this translation helpful? Give feedback.
-
|
@lukebakken , it looks like the simplified file system layout has helped. The concern I had/have was after reading @Vanlightly's articled about separating the WAL and segemt files i/o load. Perhaps my issue was specifically mounting a volume for the quorum? I was trying to isolate the log and data volumes. Current layout: This worked 100 %. Not sure if I should take this as a win and just publish these settings or try to figure out ther issue with the separate quorum volume. ¯_(ツ)_/¯ Thanks for the followup. |
Beta Was this translation helpful? Give feedback.
-
|
@daveofthedogs those blog posts are four years old and Jack is no longer on our team. Tagging him or other team members directly here is completely unnecessary. When RabbitMQ or its Raft implementation performs file I/O, it does not know what volume the target directory is on. It should not have to. Most people do not store QQ data directory on a separate volume, which may explain why we haven't see this behavior before. But when someone does use a separate volume, it is always to move QQ and stream data directories devices that offer higher write throughput. Using N EBS volumes over one won't necessarily make a lot of difference. Can this behavior be reproduced with a single EBS volume used for all of node's data? |
Beta Was this translation helpful? Give feedback.
-
|
This error has nothing to do with performance but rather indeed with volume mounting. It can be easily triggered by mounting a volume in the
The same will happen if trying to remove this folder from the operating system level (which is effectively the same thing that RabbitMQ does):
|
Beta Was this translation helpful? Give feedback.
-
|
@mkuratczyk Is it a bad practice to have a dedicated volume for quorum? I figured it would give a more optimal i/o profile for the WAL and other data. |
Beta Was this translation helpful? Give feedback.
-
|
Have I said anything that would suggest this is a bad practice? A vast majority of users doesn't do that, which would explain why we haven't seen this before. I'd expect this separation to have some performance benefit, although I'd be surprised if it was significant under typical workloads. Given you are already testing this, please share what difference you see between using and not using a separate volume. |
Beta Was this translation helpful? Give feedback.
-
|
cc @mkuratczyk - #11066 |
Beta Was this translation helpful? Give feedback.
-
|
Closing this discussion, as the answer was to not mount the quorum directory separately, Team reproduced the error and suggested a fix for it. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for following up. |
Beta Was this translation helpful? Give feedback.
-
|
According to So looks like we must retry up to N times with a delay in between (of, say, 5ms?) and it (in theory) should only affect file deletes, for which we usually can afford to retry N times. |
Beta Was this translation helpful? Give feedback.
-
|
In addition, I cannot find anything specific about EBS or multi-volume scenarios but I have a hypothesis that when there are multiple mounts and they overlap, the Linux kernel will use a different filesystem operation locking strategy (a more coarse-grained one), which may explain why the following sequence of operations may run into an
|
Beta Was this translation helpful? Give feedback.
-
|
@bakkenl has clarified what I'm missing in #11066: if you have a root node data directory on volume A, and one of its subdirectories on volume B, deleting all subdirectories from A, in particular B, won't be possible because a portion of the path is shared, and so are the associated kernel locks. So ignoring the errors might help the node start but this configuration is going to cause problems one way or another, eventually. |
Beta Was this translation helpful? Give feedback.
-
|
@bakkenl has clarified what I'm missing in #11066: if you have a root node data directory on volume A, and one of its subdirectories on volume B, deleting all subdirectories from A, in particular B, won't be possible because a portion of the path is shared, and so are the associated kernel locks. An alternative multi-volume solution might look like this:
But it should be at least mentioned that this kind of setup is rare and we don't test or document it enough. And that for two (slow compared to local SSDs) EBS volumes it may or may not yield any practical improvements in I/O throughput or latency. |
Beta Was this translation helpful? Give feedback.
-
|
Besides the suggestion to avoiding overlapping filesystem volumes, #11066 should at least avoid the exception under such conditions. |
Beta Was this translation helpful? Give feedback.
FWIW, this discussion is literally the first time I have seen
ebusyin a RabbitMQ environment. Clearly there is something odd in your specific case.@daveofthedogs any time you include the output of a command, it is essential that you also include the command invocation itself. We have no idea how you are running
fio.Do you see this issue when you do NOT separate out disks like this?