Introduce a nutanix prism client cache #425
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Skipping CI for Draft Pull Request. |
thunderboltsid
force-pushed
the
jira/krbn-8098-1.2
branch
from
3b7d552 to
3b4e896
Compare
thunderboltsid
force-pushed
the
jira/krbn-8098-1.2
branch
from
1a67a10 to
5bdcd7c
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release-v1.2 #425 +/- ##
===============================================
- Coverage 9.62% 8.57% -1.05%
===============================================
Files 10 22 +12
Lines 1340 2017 +677
===============================================
+ Hits 129 173 +44
- Misses 1211 1844 +633 ☔ View full report in Codecov by Sentry. |
deepakm-ntnx
approved these changes
May 2, 2024
thunderboltsid
force-pushed
the
jira/krbn-8098-1.2
branch
from
479da10 to
2044556
Compare
thunderboltsid
force-pushed
the
jira/krbn-8098-1.2
branch
5 times, most recently
from
62f4c24 to
519b8f4
Compare
|
/retest |
The cache stores a prismgoclient.V3 client instance for each NutanixCluster instance. The cache is shared between nutanixcluster and nutanixmachine controllers.
thunderboltsid
force-pushed
the
jira/krbn-8098-1.2
branch
3 times, most recently
from
7be47a4 to
f3adaa1
Compare
|
/retest |
thunderboltsid
force-pushed
the
jira/krbn-8098-1.2
branch
from
f3adaa1 to
0663761
Compare
|
/retest |
This is to ensure newer versions of interfaces from SharedIndexInformers don't cause compile failures. Update go version to v1.22 becasue cmp.Or is only available in go v1.22 update prism-go-client
thunderboltsid
force-pushed
the
jira/krbn-8098-1.2
branch
from
0663761 to
88bb02f
Compare
✅ None of your dependencies violate policy! |
|
/retest |
|
Closed in favor of #430 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During a recent incident, it was observed that creating a new Nutanix client for each request implies basic authentication for every request. This places unnecessary stress on IAM services. This stress was particularly problematic when the IAM services were already in a degraded state, thereby prolonging recovery efforts. Each basic auth request gets processed through the entire IAM stack, significantly increasing the load and impacting performance.
It's recommend that the client use session-auth cookies instead of basic auth for requests to Prism Central where possible. Given how the CAPX controller works currently, a new client is created per reconcile cycle. In #418 we switched to using Session-Auth instead of Basic-Auth. However, switching from Basic-Auth to Session-Auth alone wouldn’t solve the problem of consistent Basic-Auth calls. This is because each time a client is created, which is every reconcile cycle, it will still result in one Basic-Auth call to
/users/meto fetch the session cookie. To alleviate this, we are adding a cache of clients and reusing the client from the cache across reconciliation cycles of the same cluster for both the NutanixCluster and NutanixMachine reconciliation.In a large-scale setup of 40+ clusters w/ 4 nodes each, we were able to see a noticeable drop in QPS to the IAM stack for the
oidc/tokencalls. Before the client caching, a controller restart led to 10+ QPS onoidc/tokenendpoint with a steady state at around 0.5 QPS. After deploying the client cache changes, we saw a peak of ~3 QPS as caches warmed up and dropped to 0 QPS afterwards with sporadic requests only when session token refresh was needed. As we can see, with the changes proposed in this document, we were able to reduce the number of high-impact calls to IAM significantly.