[Stream] SAC Active and Waiting Comumers both must be able to listen event.

[laststem]

I'm not sure this repository is suitable to discuss this topic.

in the https://github.com/rabbitmq/rabbitmq-stream-java-client, there is ConsumerUpdateListener.

I expected to any update event(regardless of active, waiting) will invoke ConsumerUpdateListener.
because there is ConsumerUpdateListener.Context.isActive() that can inform this consumer is active(isActive()=true) or waiting(isActive()=false).

but, it invoked active(isActive()=true) consumer process only.

        val consumer = env.consumerBuilder()
            .name("test")
            .stream("playground")
            .singleActiveConsumer()
            .messageHandler { context, message ->
                println("offset: ${context.offset()}, message: ${String(message.bodyAsBinary)}")
            }
            .consumerUpdateListener { ctx ->
                println("UpdateListener: $ctx")
                OffsetSpecification.last()
            }
            .build()

The above code prints UpdateListener: $ctx only once even if there are multiple processes.

Is it Stream SAC specification that only call active consumer?

[michaelklishin]

It's a philosophical question, whether events should be broadcast to all consumers or just the active one if the SAC mode is used. I'd say only the active one. If you want N active consumers then don't use the SAC feature.

There may be situation where some metadata updates exposed to the user (your app) might be worth distributing to all consumers but you have to come up with a specific example. SAC tries to create an illusion that there's only one consumer online.

[laststem]

sorry for not enough explanation.

I didn't mean to broadcast events to all consumers. but events should be delivered when a consumer is registered.

When an event is delivered when each consumer registers with a stream, the consumer application can determine whether it is a 'leader (active) application' or a 'follower (standby) application'.

Currently consumer applications do not know whether they are 'follower applications'. This is because the broker does not send update events to the waiting consumer except the active application.

The specific scenario is as follows.

1. Consumers do not exist.
2. A consumer was registered.
3. Broker send update event (A is active)
4. B consumer was registered.
5. A got event(active=true), execute Leader logic.
6. Broker send update event (B is waiting)
7. B got event(active=false), execute Follower logic.

If the broker does not send an event to B, B cannot know whether it is not receiving the event because the connection is still in progress or whether it is a follower.

[michaelklishin]

I see your point that applications would need a hint of sorts. We will discuss what options do we have for connections/consumers within a single consumer group.

[michaelklishin]

From a team member who was one of the key contributors to the SAC feature:

The consumer update listener is triggered only when there's a change for the consumer.
Events could in theory be sent to the entire SAC member group but then consumers
would receive events that do not apply to them.

[acogoluegnes]

In a single active consumer group, a consumer is considered inactive by default, until it receives a notification that it is active. Then an active consumer can receive a notification that it becomes inactive.

A consumer gets to decide where to start consuming from when it becomes active: it responds to the notification with an offset specification and the broker will use it to initialize the message flow at the appropriate location in the stream. It can get the offset notification from any data source (RabbitMQ with server-side offset tracking or a custom database).

An active consumer has a chance to store the last offset it processed when it gets a notification that it becomes inactive. This way the new active consumer will be able to resume where the former active consumer left off.

The frequency of the consumer update notification is meant to make the system work, not to broadcast events to consumers. We do not plan on changing this unless there is a good reason to do it.

[acogoluegnes]

@laststem I pushed a change to notify a SAC that goes from active to inactive for cases the broker does not do so. Can try out the latest snapshot to check it behaves as expected? Thanks.

[laststem]

0.18.0-SNAPSHOT seems to works well. Thanks @acogoluegnes

[acogoluegnes]

OK, thanks for confirming.

[laststem]

When will this be released? @acogoluegnes thanks

[acogoluegnes]

We will release RabbitMQ Stream Java client 0.18.0 this week.