Go application CPU usage is higher than expected as the number of queues (streams) grows

[LucasZhanye]

Describe the bug

I have three node as rabbitmq cluster, and I also have three golang applications as distributed clients.

These applications lisnten on rabbitmq classic queue on vhost "messagebus".

And then I use this rabbitmq cluster to test stream queue on other vhost "/"(one producer send message(about 1.4 million msg) to stream and one counsumer consume it), it test ok

After my test, I found that it affects classic queue, I use top command to look up the golang application's CPU, it is too high.

Because of using golang, I use pprof to check the code, it shows below:

My guess is that the CPU usage is high due to constantly entering this 'select'.
But this Rabbitmq message channel should be blocked and monitored, and I confirmed that no message is sent to this queue. Why do messages keep coming?

the code like below:

func LoadSubscription() error {
        exchangeName := DEFAULT_TOPIC_SYNC
	queueName := config.Config.ServiceName
        config := AmqpConfig
	config.Queue.Arguments = amqp.Table{
		"x-queue-mode": "lazy",
	}
	config.Exchange.Type = amqp.ExchangeFanout

	err := ListenMQ(exchangeName, queueName, "", config, func(messages <-chan amqp.Delivery, closed chan *amqp.Error) {
		for {
			select {
			case msg, ok := <-messages:
				if ok {
				    ........
				} else {
                                   .........
				}
			case closedErr, ok := <-closed:
				if ok {
					logger.Error("Channel Close error : %+v", closedErr)
				}
				break
			}

		}
	})

	return err
}

Reproduction steps

1. run a rabbitmq cluster
2. create classic queue and listen it
3. create stream and send it for many message
4. check the CPU usage of the application
   ...

Expected behavior

the CPU usage of the application should not too high.

Additional context

No response

[michaelklishin]

I'm not sure why you believe that our team should consider a "high CPU usage" in your app a problem, however small, in RabbitMQ server. Even less sure why you expect to receive any kind of help without sharing your code (not a single function, a public repository with an executable way to reproduce).

Should we guess what ListenMQ does? We do not guess in this community.

How Streams Can Affect Clients

Streams, just like classic queues and quorum queues, do consume some resources, even though streams is the most efficient data structure in certain terms (namely memory usage).

Streams emit metrics and can perform periodic operations, just like queues.

If a stream is not used, the only way it can affect an AMQP 0-9-1 client activity is the same way quorum queues or other classic queues would: by increasing context switching on the node. If the node only has one core available to it, this may have a visible effect on delivery rate, which means consumers that rely on polling may start spending more CPU time waiting for I/O.

RabbitMQ was not designed for single core systems and will use all the cores available to it, given there are enough queue and stream replicas and connections (channels).

There are dedicated documentation sections on Reducing CPU Usage for mostly idle RabbitMQ nodes (not clients) and Reducing Metrics Emission Frequency which is directly tied to the average number of context switches a node will perform.

As for how to debug your app, we won't guess.

[LucasZhanye]

Thanks, I think I did miss some code.
But what I mean is that after testing the stream, the classic queue I have been listening to will always receive messages, so my application will occupy a high CPU. It is not that Rabbitmq causes a high CPU usage, I just don’t understand why no message is sent to this classic queue, but it keeps triggering select.

there is the whole code by golang:
mbs.zip

[mkuratczyk]

For me the CPU usage is 0. If you have pprof report or something else to show your CPU usage - please do. I don't see any problems here.

By the way, if you are using RabbitMQ 3.12 or 3.13 (and you should), lazy mode option does nothing: https://www.rabbitmq.com/blog/2023/05/17/rabbitmq-3.12-performance-improvements#classic-queues-changes-to-the-lazy-mode

[LucasZhanye]

For me the CPU usage is 0. If you have pprof report or something else to show your CPU usage - please do. I don't see any problems here.

By the way, if you are using RabbitMQ 3.12 or 3.13 (and you should), lazy mode option does nothing: https://www.rabbitmq.com/blog/2023/05/17/rabbitmq-3.12-performance-improvements#classic-queues-changes-to-the-lazy-mode

@mkuratczyk Thanks, my rabbitmq version is 3.9.16, maybe it is too old, and I have pprof,it's strange to me that he settled into this line of work

[michaelklishin]

3.9.x has been out of support for ages.