high read latencies

[ltagliamonte-dd]

Hello folks,
we are recording high read latencies, when we write to kvrocks, and I'm looking for suggestions on what to explore/try next.

Background:
We have a job that nightly prepares an updated kvrocks backup, compacts it, and then we deploy this new update to prod where clients reads from it. We don't use replication and the clients only reads from kvrocks.
We wanted to improve this, by constantly updating production, but when we turn the sync on (just a sidecar process that downloads and applies diffs) our P999 latency goes up significantly from ~30ms to 70ms:

All the reads are HGET and all the set are HSET.

We are using latest kvrocks version 2.10.1 and our config looks like:

--bind 0.0.0.0
--port 6379
--rocksdb.write_options.disable_wal yes
--rocksdb.wal_ttl_seconds 0
--rocksdb.wal_size_limit_mb 0
--rocksdb.compaction_readahead_size 10485760
--rocksdb.target_file_size_base 1024
--max-backup-to-keep 0
--tcp-backlog 4096
--log-dir stdout
--log-level error
--dir /data/kvrocks
--workers 4
--maxclients 20000
--timeout 120
--slowlog-log-slower-than 1000
--profiling-sample-commands hmget
--profiling-sample-record-threshold-ms 100
--rocksdb.block_cache_size 6144
--rocksdb.block_cache_type hcc
--rocksdb.row_cache_size 1024

if we turn off the sync and start from the nightly compacted backup latencies go back to prev values, so we know that writes are afflicting our system and well compacted data also play a big role in read latencies.

• Is there something we are missing or wrong with our config?
• Is there a way to compact more?

[git-hulk]

@ltagliamonte-dd I cannot tell if it's affected by the compact operation. However, the compact might bring extra disk IO and CPU usage since it needs to read old SSTs and write back when generating the new SST. To confirm this, you can check if the latency is back to normal after the compaction is finished. Also have a check at the disk io/cpu metrics if too high while compaction.

[ltagliamonte]

Thanks for the answer @git-hulk.
What are the compactions tunables?