why thanos compactor do compacting before apply retention policy

[RayHuangCN]

I found that thanos compactor do compacting and meta syncing (many times) before applying retention policy. This may cause old blocks never be removed if something wrong during compacting ,and then, old blocks become more and more, and lead to compacting or meta syncing error again.

I think old blocks will never be used for compacting and can be removed before compacting. 🤔

[ahurtaud]

I think this is done on pupose in case retention times are different across resolution.

If the compaction is running late, and the raw resolution is expired before having the 5m block, then it would remove the raw block and data will be lost before having the 5m downsampled data.

[yeya24]

This makes sense as old blocks might be still able for compaction ( < level 4). The retention check is based on block max time. So once an old block is compacted with other new blocks and a new block is generated, the new block might still be within the retention period and the data from the old block is kept.

For some cases, even if olds blocks are compacted, the time range is still beyond the retention time, this is something we can optimize, maybe in the planner side specifically. Ideally, this is not hard to do. But as @ahurtaud mentioned, downsampling will affect it as well because level 4 blocks might be able for downsampling so we cannot simply delete the blocks.

For most of cases, if there are not too much data in object storage, the current logic is fine as the retention will be applied after the compacting and downsampling phase is done.

If you have too much data in obj store, try scaling your compactor more or use tools bucket retention tool to clean up old data first.

[yeya24]

Move this issue to discussion.