Added batching using DB. #731
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Dzejkop
reviewed
May 24, 2024
Dzejkop
reviewed
May 24, 2024
Dzejkop
approved these changes
May 24, 2024
0xForerunner
requested changes
May 24, 2024
piohei
force-pushed
the
piohei/batches_in_db
branch
from
32e0b61 to
2c6b9f9
Compare
0xForerunner
approved these changes
May 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a first step to introduce batches and transactions table. This is not HA solution yet - just the first step towards it.
Motivation
Make small changes instead of one big.
Solution
Now processing batches is changed into 2 steps. First one is to create a batch (select identities, leaf indexes, etc.). Then the batch is saved in database. Second step is reading data from database (batch) and executes it. Proper batch execution includes storing transaction id in database.
As an alternative solution we could save in database a batch with a proves (no need to calculate it later). It will allow second step (transaction creation/execution) to not use any tree at all. It will require more memory used in database (it can be cleared from time to time as we don't need all the batches forever) but will make HA solution much easier.
This version is also removing batches/transactions when identities are again marked as pending.
PR Checklist