Intermittent recording rule failures during WAL operation #10398
rishabhkumar92
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi Team,
We are seeing intermittent recording rule failures (Ruler -> Ruler QFE -> Ruler Querier -> Ingesters) during WAL operations.
We adjusted head compaction interval to 15 minutes to spread out the WAL operations and thus spreading out the disk write/read bandwidth. Are there any side effect of spreading it to 30 minutes, what are other recommendations to solve for this backpressure issues?
Current Mimir version: 2.14.1
Infrastructure: K8s
Beta Was this translation helpful? Give feedback.
All reactions