rabbitmq - When does it page out to disk?

[luarx]

- RabbitMQ version: 3.13.1
- Docker image: rabbitmq:3
- Calculation strategy: rss
- Memory high watermark setting: 0.4 of available memory, computed to: 6.5492 gb
- How runs: Docker image in Kubernetes
- Node host architecture: ARM

First of all, excuse me for opening this discussion as maybe it is trivial but I can not find in rabbitmq docs information about this (if there is , just point me out to the right place)🙏

Afaik, rabbitmq considers vm_memory_high_watermark and vm_memory_high_watermark_paging_ratio values (this lask one maybe obsolete? ) to know when it needs to page out to disk.

My use case is that from time to time:

• My rabbitmq container starts to page out to disk and it is not able to revert the situation unless I restart it and delete all memory/disk data.
  • When that happens, memory usage is lower than vm_memory_high_watermark and vm_memory_high_watermark_paging_ratio
• Because of the pagination (or maybe it is in the other way round) rabbitmq is slower

I would appreciate if you could point me out in the right direction 🙏 🚀

I share screenshots of a real situation that started at 9am:

• CPU/Memory resources were enough and lower than the vm_memory_high_watermark values (Memory usage: 300Mi and vm_memory_high_watermark: 6.5492 gb) but rabbitmq started to page out to disk when the memory usage increased a little bit
  
• When the previous point happened, rabbitmq server paged out to disk:
  
• And here you have some more metrics around that moment:
  
  
  
  
  
  
  
  
  
  
  
  

[michaelklishin]

vm_memory_high_watermark_paging_ratio is only relevant for classic queues v1, not classic queues v2 (CQv2).

Quorum queues move data to disk immediately but keep a certain number of WAL entries in memory.

Streams always move data to disk "straight from the socket" and keep very little in memory.

Classic queues v2 act as classic queues v1 in lazy mode if we oversimplify, that is, they generally move data to disk quickly.

[michaelklishin]

I don't know how to interpret these charts, "message pipeline" is not a metric that RabbitMQ exposes and not a part of our own Grafana dashboards. According to the chart, there is a massive backlog build-up that then very quickly clears, possibly because a consumer shows up.

There can be any number of reasons for that, and "slow RabbitMQ" very often turns out to be a slow group of consumers which publishers easily outpace. One way or another resource use (even if you use streams) will grow until there's enough consumers online or publishers are throttled heavily enough to make the inflow/outflow more balanced.

There are also storage layer recommendations and certain tuning recommendations for quorum queues that may benefit the QQs running at close to their peak throughput.

We have no details on the queue types used, their durability, on whether applications publish messages as transient (the latter two are important for CQv1s) to offer an informed hypothesis. We do no guess in this community.

When in doubt, use quorum queues, streams and CQv2s, and fast disks.

[luarx]

Thank you for your answer! Will take a look and see if I can debug further with your advises 🙏