Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamically increase/decrease the etcd quota size and update the doc #984

Open
ishan16696 opened this issue Jan 22, 2025 · 0 comments
Open
Labels
area/disaster-recovery Disaster recovery related kind/enhancement Enhancement, improvement, extension

Comments

@ishan16696
Copy link
Member

ishan16696 commented Jan 22, 2025

How to categorize this issue?

/area disaster-recovery
/kind enhancement

What would you like to be added:
It has been observed that when etcd exceeds its quota limit of 8GB, the typical solution is to perform compaction and defragmentation to decrease the etcd db size. However, these operations may not always be effective, especially if the etcd database contains numerous unique writes (unique keys), potentially due to a bug in the user's deployed workload.
To address this, consider dynamically adjusting the etcd quota size and document the steps required to recover from this scenario in our documentation.

Why is this needed:
In situations where etcd exceeds its quota size, operator need to usually have to baby sit that cluster The typical solution involves manually deleting some user resources directly from etcd, followed by running compaction and defragmentation command on etcd.
In this proposal, I suggest dynamically increasing the etcd quota size to stabilize the etcd, allowing users to deploy necessary fixes. Once the issue is resolved, the etcd quota size can be dynamically reduced back to its original limit.

@gardener-robot gardener-robot added area/disaster-recovery Disaster recovery related kind/enhancement Enhancement, improvement, extension labels Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/disaster-recovery Disaster recovery related kind/enhancement Enhancement, improvement, extension
Projects
None yet
Development

No branches or pull requests

2 participants