PXC-4500: When innodb_thread_concurrency is set, the cluster can get stuck during SST #1963
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
venkatesh-prasad-v
requested changes
Oct 10, 2024
mysql-test/suite/galera/t/galera_innodb_thread_concurrency.test
Outdated
Show resolved
Hide resolved
…stuck during SST https://perconadev.atlassian.net/browse/PXC-4500 Problem 1: When innodb_thread_concurrency is set and the node has a heavy user workload, it can become stuck when it receives a CC event from Galera. Cause: Let's say innodb_thread_concurrency = 2. 1. We have two user threads that have been granted access to InnoDB. They will release the InnoDB access lock after Galera certification, but for certification, they need to acquire LocalMonitor. 2. At the same time CC happens. The Applier thread acquires LocalMonitor and notifies the application about the new view. The application tries to store the view in wsrep schema. Before doing that it turns off wsrep_on thread's variable (the thread is not marked as wsrep-enabled thread). Then, it tries to enter InnoDB. As we already have two user threads in InnoDB and our thread is not wsrep thread, we have to wait. The above results in a deadlock: 1. The user thread is holding InnoDB lock, waiting for LocalMonitor 2. The Applier thread is holding LocalMonitor, waiting for InnoDB lock Solution: The wrong condition was used to detect the wsrep applier thread in innobase_srv_conc_enter_innodb(). The applier thread should always be granted access. Fixed. Problem 2: Even after fixing Problem 1, the cluster was stuck during SST. Cause: The SST thread creates an SST user. For this, it needs to enter InnoDB. We end up in a similar situation as in Problem 1. The applier thread holds LocalMonitor and waits for SST to finish. The SST thread waits for InnoDB. User threads hold the InnoDB lock and wait for LocalMonitor. Solution: Allow SST thread to enter InnoDB always.
kamil-holubicki
force-pushed
the
PXC-4500-8.0
branch
from
f64278f
to
f8b2f56
Compare
venkatesh-prasad-v
approved these changes
Oct 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://perconadev.atlassian.net/browse/PXC-4500
Problem 1:
When innodb_thread_concurrency is set, and the node has a heavy user workload, it can become stuck when it receives a CC event from Galera.
Cause:
Let's say innodb_thread_concurrency = 2.
The above results in a deadlock:
Solution:
The wrong condition was used to detect the wsrep applier thread in innobase_srv_conc_enter_innodb(). The applier thread should always be granted access. Fixed.
Problem 2:
Even after fixing Problem 1, the cluster was stuck during SST.
Cause:
The SST thread creates an SST user. For this, it needs to enter InnoDB. We end up in a similar situation as in Problem 1. The applier thread holds LocalMonitor and waits for SST to finish. The SST thread waits for InnoDB. User threads hold the InnoDB lock and wait for LocalMonitor.
Solution:
Allow SST thread to enter InnoDB always.