feat(spooler): Implement backpressure mechanism based on atomics #4040
Conversation
| @@ -169,6 +173,11 @@ impl EnvelopeBufferService { | |||
| tokio::time::sleep(self.sleep).await; | |||
| } | |||
|
|
|||
| // In case the project cache is not ready, we don't want to attempt popping. | |||
| if !self.project_cache_ready.load(Ordering::Relaxed) { | |||
| return future::pending().await; | |||
There was a problem hiding this comment.
can we do a sleep for DEFAULT_SLEEP here? If Relay is in a mode where it does not receive any messages, this might block forever.
There was a problem hiding this comment.
Am I understanding this correctly: The EnvelopeBufferService checks the flag and if it's set (i.e. ready), it sends an envelope to the ProjectCacheService and flips the flag to false. The ProjectCacheService flips the flag back to true once it has processed what it was sent to signal that it's ready for more work?
Yes, correct! |
This PR adds a new, simpler backpressure mechanism that relies on a shared
AtomicBoolflag. The rationale behind this implementation is that we observedtokiooneshotchannels can experience spurious failures, causing theReceiverto returnPoll::Pending, which skews the buffer towards more writes instead of reads, resulting in disk spooling and a consequent system slowdown. WithAtomicBool, we aim to achieve better performance since the flag update should be relatively fast.#skip-changelog