Consumer can process only messages from one topic at a time

[ari-github]

When consuming messages from topic A, the consumer gets blocked and does not pull messages from topic B until it has finished processing the messages from topic A.

Reproduction Steps and Original Issue:

This behavior can be reproduced and has been originally reported in the NestJS repository. For more details, you can refer to the issue and related comments here.

[fabiomagagnin]

I think you can leverage partitionsConsumedConcurrently as this doc: https://kafka.js.org/docs/consuming#a-name-concurrent-processing-a-partition-aware-concurrency. I used it in a Nest JS application and it's working. But I'm facing a similar issue, if a message get stuck (can't be consumed because of an error, and will be retried) in topic A, then it partially blocks all partitions of that consumer. Partially, because on the next attempt to consume the bad/blocking message, it will consume other topics.

[ari-github]

Yes, I have tried using partitionsConsumedConcurrently, but it does not resolve the issue. It only works if there are already messages in both Topic A and Topic B before the consumer starts processing. However, if new messages arrive in Topic B while the consumer is processing messages from Topic A, the consumer will not attempt to consume them until it has finished processing all messages from Topic A.

It is understandable that new messages in Topic A should wait until the current batch is processed to maintain order, as required by Kafka. However, there is no reason why the consumer should not process messages from different topics concurrently. Each topic should be processed independently to ensure efficient message consumption.

[fabiomagagnin]

Oh, I missunderstood it, I was thinking about partitions in the same topic partitionsConsumedConcurrently is to solve that issue, but your problem is consuming different topics. I 100% agree that one topic should not block another.