Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi val db access pr #69

Closed
wants to merge 2 commits into from

Conversation

dan-da
Copy link
Collaborator

@dan-da dan-da commented Nov 18, 2023

In general, this PR makes locking around DB accesses (both read and write) less granular.

There is a tradeoff here that became more apparent the further I dug into things.

This may make functions that perform a lot of DB accesses a bit more performant. It might lessen occurrence of deadlocks.

For certain, it:

  1. forces all other threads to block until these functions complete.
  2. requires allocating memory for all values retrieved from get_many(), get_all().

I have commented the locking issue in the code at each point of usage.

Our previous approach processing each value at a time might be a bit slower and acquire many more locks, but it is much more concurrent and works with arbitrary number of values without exploding memory for large datasets, eg wallets with thousands of monitored utxo. For this reason, I am making this a draft PR, and not recommending it be merged until further testing/benchmarking/analysis is conducted.

HOWEVER: the elephant in the room is the nature of the DB accesses. The rusty-leveldb crate we are using does not support concurrent accesses:

"As opposed to the original, this implementation is not concurrent (yet):

Whereas the leveldb crate (wrapping C++ leveldb) does support concurrency. Also the DB::get() fn in rusty-leveldb requires (&self mut), so we cannot use an RwLock, but must instead use a Mutex. Readers block eachother as well as writers. I believe that if we were to move to the leveldb crate we could get rid of the locking around DB calls and the code in this PR would work with good concurrency. Note also that get() method takes &self. We should not need any locks around these db calls (greatly simplifying our code), but if we wanted to make our own RwLock, we could.

Also, there is a draft PR for twenty-first that provides iterators for the DB types. This would enable usage of the get many/all set-based semantics in this PR without blowing up memory for large data sets. So that is a possibility as well and would work best with the concurrent version of leveldb.

Finally, bolded on the rusty-leveldb repo index page:

NOTE: I do not endorse using this library for any data that you care about.

one last clarification: I find this code a bit cleaner and more readable than what came before, so I hope we can merge it, or something close if/when the above issues are resolved.

commit messages follow:


Author: danda dan-da@users.noreply.github.com
Date: Fri Nov 17 21:01:57 2023 -0800

feat: use get/set_many, get/set_all() for DB

Use multi-item DB operations instead of locking DB per item

Cargo.toml is modified to override twenty-first dep with
master from github.

Author: sword-smith thor@neptune.cash
Date: Thu Nov 9 21:57:36 2023 +0100

Improve lock-handling in method to prune abandoned MUTXOs

Co-authored-by: Dan <danda@neptune.cash>

Sword-Smith and others added 2 commits November 9, 2023 21:57
Co-authored-by: Dan <danda@neptune.cash>
Use multi-item DB operations instead of locking DB per item

Cargo.toml is modified to override twenty-first dep with
master from github.
@dan-da
Copy link
Collaborator Author

dan-da commented Jan 20, 2024

closing in favor of #89, already merged.

@dan-da dan-da closed this Jan 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants