Compaction is taking longer time to complete(~2minutes)

[rajanarayanaa]

On my ETCD setup in a k8s cluster, I'm observing that the compaction is taking longer time to complete. Adding the following log.
The RTT b/w nodes is around 100ms, disk latency is around 50ms, which is an induced latency to simulate certain environments. This happens when the node is a leader node. When it changes to follower, the compaction time drops to 1-2ms

2024-02-15 01:34:27.483165 I | mvcc: finished scheduled compaction at 17374725 (took 2m1.458949288s)

Can someone please guide me on how to debug the setup to identify reasons for such a high delay? Normally I see the compaction operation gets completed within 1-2ms.

[jmhbnz]

Hi @rajanarayanaa - Thanks for your question. Can you please confirm which version of etcd you are running?

Can you also please provide the configuration settings you are using for etcd compaction along with some context on the scale of your etcd cluster data?

[rajanarayanaa]

ETCD version is 3.4.17. The compaction is triggered by API server every 5 minutes.
ETCD is used for 3 node kubernetes cluster with around 40 pods running. The workload isn't high to be specific. The total number of keys are around 4000 and the DB size currently is ~70MB.

[rajanarayanaa]

I could not understand what could be the potential cause for high compaction time on leader node only. The heartbeat time is 500ms and timeout is 5000ms. As mentioned earlier there is a induced latency of 100ms b/w etcd nodes.
The compaction time on follower nodes completes in 1-2ms where as on leader node it varies between 1 minutes to 2 minutes.
Looking at the metrics we have compaction duration ranging from 12.8s to 102.4s

etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="12800"} 7434
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="25600"} 7521
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="51200"} 7624
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="102400"} 7834

[jmhbnz]

Personally I've not seen this issue with compaction times that long before and I certainly wouldn't expect it with a relatively small cluster.

The only advice I can give 3.4.17 is nearly three years old and a lot of bug and security fixes have gone in since then so I would suggest upgrading to the latest 3.4.x release as soon as possible. Another project member may have some more insight.

[rajanarayanaa]

Thanks for the information @jmhbnz. Is there any specific areas to debug to figure out the bottleneck?