Streams & policy behaviour in 3.12.2

[zaeemarshad]

I am trying to understand how message retention with Streams with 3.12.2. I have a test stream with about a 100+ messages of varying lengths. The policy only seems

Scenario 1:

1. Create a stream and add a few messages
2. Apply policy with max-age and max-segment-size-bytes e.g. 40KB

{"vhost":"/","name":"stream-expiry-2","pattern":".*","apply-to":"streams","definition":"{\"max-age\":\"10m\",\"stream-max-segment-size-bytes\": 40960}","priority":0}

4. Publish messages - policy has no effect
5. Removing policy and applying another one has no effect. The stream sticks to a single index segment

Scenario2:

1. Create a stream but define max-age and max-segment-size-bytes (40KB) at creation time
2. Add messages to the stream - new segments are created in accordance with the policy i.e. 42KB segments
3. Remove the stream policy and add messages - new segments are still created in accordance with the now removed policy - new segments are still created after reaching the 42KB limit
4. Apply a new policy e.g. 50KB limit and add messages - new policy has no effect. Segments are still created after reaching the 42KB size

It seems that policies are not a runtime thing but can only be defined at the creation of the stream.

[michaelklishin]

@zaeemarshad policies absolutely are "a runtime thing" but not everything that can be applied via a policy can be applied retroactively, or sometimes it can be but it won't have an effect on the existing messages. This is the case of several TTL-related features in queues.

Moreover, some things in RabbitMQ cannot be changed once set. You cannot change the maximum number of priorities of a queue after defining it. You cannot change a queue type after declaring it, and so on. There is no migration path from one queue type to another at runtime, and there won't be.

Moreover, you cannot set every setting to a completely arbitrary value. Streams were not designed to be limited to a single offset of 40 or 50 kiB, that's just not a practical workload for a data structure that can keep its data around for hours, days, and maybe even weeks (if you are certain you won't run out of disk over this time period). See what kind of values the example in the docs uses.

I do not understand where in Scenario 2 you remove the policy; you claim to "remove the stream and add messages". Add messages where, since the stream is gone?

In any case, just like with queue TTL, this is not a feature you are expected to flip back and forth after the stream has been created and is in operation, in particular changing the the segment size part at runtime does not make much sense.

[zaeemarshad]

Thanks for the quick reply @michaelklishin. There's a typo in scenario 2 - it should have been policy instead of stream. I have updated the original description.

You cannot change the maximum number of priorities of a queue after defining it. You cannot change a queue type after declaring it, and so on. There is no migration path from one queue type to another at runtime, and there won't be.

This is understandable and very well documented/observed - no issues there. I don't think the same is true for Streams which are very new and hence the documentation on the official website as well as the Internet has to catch-up.

Streams were not designed to be limited to a single offset of 40 or 50 kiB, that's just not a practical workload for a data structure that can keep its data around for hours, days, and maybe even weeks (if you are certain you won't run out of disk over this time period).

This is just a test setup to understand the behaviour. In the interest of saving time, it is easier to publish a few large messages and see if the policy behaviour is as intended. This is not for production use. However, does this mean that there's a minimum segment size under which max-segment-size-bytes policy has no effect?

In any case, just like with queue TTL, this is not a feature you are expected to flip back and forth after the stream has been created and is in operation, in particular changing the the segment size part at runtime does not make much sense.

The expectation here is that we will learn a few things over time such as retention which varies depending upon the use case. In scenario-1, the stream was created without any policy and subsequently adding the policy had no effect. This was unexpected and is not clear from the docs. Perhaps, the documentation or Best Practice guide should be updated to reflect these constraints and current limitations.

[michaelklishin]

A mandatory mention of the fact that 3.12.2 is from July 2023 and is 10 patch releases behind. Not one, not three, ten.

[zaeemarshad]

Thanks - this is a test up but I will update the docker image to pull 3.12.latest

[acogoluegnes]

@zaeemarshad I did some testing that confirms your findings: the policy does not apply automatically after the stream creation. I managed to get the expected behavior (max segment size in my tests) after restarting the stream (with rabbitmq-streams restart_stream).

Can you try again with a stream restart? You can use Stream PerfTest to publish messages.

[zaeemarshad]

Yup - that worked. Thanks for the confirming that.

[kjnilsson]

@zaeemarshad the max-segment-size-bytes configuration is a configuration that will only be picked up at stream declaration time. The fact that it kind of works if you restart the stream after the policy is a bug and this behaviour will be removed so don't rely on this.

Updating the retention at runtime is supported however, doesnt require a stream restart and should work fine right now.

[zaeemarshad]

Thanks for looking into this. I am curious why is this considered a bug? Does changing segment size cause performance issues?

[michaelklishin]

@zaeemarshad it's not a safe thing to do. If you have on disk data, you must be extra careful about changing any code assumptions about it. With settings like these, by far the best option is to not allow these values to be modified at all. No one really needs to change them at runtime.

[zaeemarshad]

@michaelklishin Thanks for the explanation. It is worth adding these to the documentation and best practices on what not to do with the Streams and potential gotchas.

[michaelklishin]

rabbitmq/rabbitmq-website#1796

[kjnilsson]

#10498