-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add VTGate drain flag #15953
Add VTGate drain flag #15953
Conversation
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Tests
Documentation
New flags
If a workflow is added or modified:
Backward compatibility
|
log.Infof("Starting drain loop, waiting for all clients to disconnect") | ||
reported := time.Now() | ||
for srv.vtgateHandle.numConnections() > 0 { | ||
if time.Since(reported) > 5*time.Second { | ||
log.Infof("Still waiting for client connections to drain (%d connected)...", srv.vtgateHandle.numConnections()) | ||
reported = time.Now() | ||
} | ||
time.Sleep(1000 * time.Millisecond) | ||
} | ||
return | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With Application side connection pooling this will most likely not change and VTGate can remain in the same state before being shut down.
I'm trying to understand how this differs from / interacts with the connection shutdown functionality we added some while ago? When For connections that are inside of a transactions, we allow them to continue on until the end of the grace period is reached (at which point we explicitly terminate their connection). Same for idle connections - they get terminated explicitly if the grace period is reached. So what is this PR attempting to fix / change exactly? 🤔 |
Also, I think the existing logic to wait for connections to become idle is not working correctly / doesn't make sense. |
I am closing this PR as some discussion is needed, and potentially more changes than the one proposed in this PR. I updated the RFC taking your comments into account. cc @harshit-gangal @arthurschreiber RFC: #15971 |
Description
This PR adds a new flag to VTGate to allow for all the MySQL connections to be drained when terminating VTGate. The drain wait until all the idle connections have disconnected (or the timeout expires, whichever happens first). When the server goes into OnTerm state, new connections will stop being accepted, and the load balancer will connect new application sessions to other vtgates. Once all the idle app connections have drained (because of the max lifetime of 1 hour of a session on the client side), the vtgate will exit.
Checklist
Deployment Notes