RabbitMQ 3.12 - Unexpected Channel shutdown ERROR with NACKS RECEIVED message

[ramazangirgin]

Hello everyone,
We recently upgraded our RabbitMQ cluster to RabbitMQ 3.12. After setting up the new environment, we started getting the error log for "Shutdown Signal". Some examples of logged errors.

2024-02-08 07:41:26.339 [Catalina-utility-4] INFO org.springframework.amqp.rabbit.connection.CachingConnectionFactory - Created new connection: connectionFactory#3c1b8bd2:0/SimpleConnection@55bc2132 [delegate=amqp://webapp@XXX.80.32.XXX:5672/, localPort= 39050] 2024-02-08 07:49:12. 946 [api@thread-pool-rabbitmq-connection@thread-2] ERROR org.springframework.amqp.rabbit.connection.CachingConnectionFactory - shutdown signal: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=NACKS RECEIVED, class-id=0, method-id=0) 2024-02-08 07:49:13.048 [api@thread-pool-rabbitmq-connection@thread-8] ERROR org.springframework.amqp.rabbit.connection.CachingConnectionFactory - Shutdown Signal: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=NACKS RECEIVED, class-id=0, method-id=0)

Before updating the rabbitMQ version (old version RabbitMQ 3.8.27) there were no logged errors of this kind. We use the same spring boot application to access both cluster environments. We are using spring boot 2.7.18.

Do you have any suggestion regarding root cause and possible solutions in server side or client config ?
Unfortunately i am not expert on RabbitMQ topic and not sure if this is a client side or server side issue.

I want to run some performance test by using the https://perftest.rabbitmq.com/ but i don't know which parameters should i use to simulate / re-generated NACKS RECEIVED case.

Thanks in advance. Ramazan

[garyrussell]

I don't recall ever seeing a "NACKS RECEIVED" normal close.

We normally suppress normal close logs (but only with "OK" text).

https://github.com/spring-projects/spring-amqp/blob/61e7a5e46e7fb4a8ffa6ebb79a2cd869768c6677/spring-rabbit/src/main/java/org/springframework/amqp/rabbit/connection/RabbitUtils.java#L252-L257

BTW, I am no longer involved with the spring-amqp project; cc: @artembilan

[lukebakken]

Thank you @garyrussell for the fast response.

[lukebakken]

@ramazangirgin what would assist us GREATLY is for you to provide a simple, console-based project that I can clone, compile, and run to see this issue. I am not a Java expert and it would take me some time to get a project going.

Anything you can do to make it easy for us to reproduce this issue saves us time, and then we can focus on explaining it and fixing it.

[ramazangirgin]

@lukebakken thank you but it is not required for now since i think we found the root cause of the issue. If we face with it again and needs support, i will do it

[michaelklishin]

reply-text=NACKS RECEIVED

suggests that RabbitMQ responds with nacks.

A traffic capture will be able to confirm it. Node logs may be able to provide certain clues, although not with every queue overflow behavior.

[artembilan]

Just to trace everything together:
https://stackoverflow.com/questions/77978435/rabbitmq-3-12-unexpected-channel-shutdown-error-with-nacks-received-message
spring-projects/spring-amqp#2617

So, @michaelklishin , @lukebakken ,
do you confirm that we need to look into that NACKS RECEIVED reply text as well, not only OK as we do right now?

Thanks

[michaelklishin]

I'm afraid I don't understand what you mean by that. There is no single right approach to using publisher confirms and nacks can be returned for different reasons, including transient but common conditions that depend on how the queue is set up.

My goal was to point out that is clearly something that should be investigated by the author of the question, nothing beyond that. We don't have enough specifics to conclude much anyway.

[ramazangirgin]

@artembilan I don't know the details but i think in logs "NACKS RECEIVED" responses shouldn't be visible as OK. The reason is; it results with some incorrect approaches to the issue, like just ignoring it.
But from my understanding, it should be always checked since it is some signs of problems with underlying queues or rabbitMQ installation similar to the situation which we faced ( because of incorrect configuration for the queues ).

[ramazangirgin]

Hi all, thank you for all responses and comments.
After investigating it in details, we realized that some queues were not configured properly for all nodes and as a result some queues were without consumers.
So RabbitMQ server was returning "NACKS RECEIVED" response since it was not able to deliver the message to proper queue based on our setup.
We recreated all queues and will check it for some days in our test environment.
But interesting thing is our infrastructure team couldn't find any error log in RabbitMQ server side regarding non-configured consumer for the queue while trying to deliver the message to the queue. Personally i would hope to see some logs ( at least warnings ) there.
We will also check our client configuration to not lose messages in case it is not delivered successfully to server.
Thanks again
Ramazan

[garyrussell]

Note that Spring can be configured to return unroutable messages to the application - when using publisher confirms, it is particularly easy because the future in the CorrelationData is completed after any returned message is added to the CorrelationData.

You can also use a callback to get returned messages asynchronously.

See https://docs.spring.io/spring-amqp/reference/amqp/template.html#template-confirms

[lukebakken]

Thank you @garyrussell

@ramazangirgin -

But interesting thing is our infrastructure team couldn't find any error log in RabbitMQ server side regarding non-configured consumer for the queue while trying to deliver the message to the queue

No, RabbitMQ does not log this. Dropping messages is something that RabbitMQ can do in normal operation, so it's not logged.

It is up to application developers to correctly use publisher confirmations and (perhaps) mandatory when publishing messages.

[michaelklishin]

This doc guide should be a good starting point (that links to several other guides that are relevant)

[michaelklishin]

There are metrics that demonstrate the number of unroutable or dropped messages, too (that RabbitMQ tracks and reports).