-
Notifications
You must be signed in to change notification settings - Fork 91
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(spooler): Add EnvelopeStack based on SQLite #3855
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, I think we can merge this into master soon, and then decide on final interfaces (sync vs async etc.) once I merge it with my work.
} | ||
|
||
if let Some(batch) = self.batches_buffer.front_mut() { | ||
batch.push(envelope); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a bit confused about pushing batches to the front but individual elements to the back. Can we sort both the batches_buffer
and the individual batches left-to-right, i.e. push to the back in both cases?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As discussed, push front vs push back.
This PR adds the
SQLite
implementation of theEnvelopeStack
trait. The implementation works in the following way:The idea of the implementation is to allow an in-memory data structure to always serve the most recent envelope and only go to disk when really necessary. So, during low load, it could easily be that the memory buffer is only used and never spilled to disk (assuming thresholds are defined correctly).
It's important to note that we are trading off durability for speed with this design.
Closes: #3855