ethdb/pebble: cherry pick pebble backend from go-ethereum #385
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
minh-bq
force-pushed
the
feat/pebble-db
branch
3 times, most recently
from
0843910 to
9e0177a
Compare
…(#24392) This PR adds an addtional API called `NewBatchWithSize` for db batcher. It turns out that leveldb batch memory allocation is super inefficient. The main reason is the allocation step of leveldb Batch is too small when the batch size is large. It can take a few second to build a leveldb batch with 100MB size. Luckily, leveldb also offers another API called MakeBatch which can pre-allocate the memory area. So if the approximate size of batch is known in advance, this API can be used in this case. It's needed in new state scheme PR which needs to commit a batch of trie nodes in a single batch. Implement the feature in a seperate PR.
* ethdb: use pebble Co-authored-by: Gary Rong <garyrong0905@gmail.com> foo update * apply suggested changes * flags: go format node: fix ddir lookup mistake accounts/abi/bind: fix go.mod replacement for generated binding deps: update pebble + with fix 32-bit build * ethdb/pebble: respect max memtable size * core/rawdb, ethdb: enable pebble on non-32bit platforms only * core/rawdb: fix build tags, fix some review concerns * core/rawdb: refactor methods for database opening * core/rawdb: remove erroneous build tag * cmd/geth: fix the flag default handling + testcase * cmd/geth: improve testing regarding custom backends * ethdb/pebble, deps: update pebble dependency * core/rawdb: replace method with Open * ethdb/pebble: several updates for pebble (axieinfinity#49) * ethdb/pebble: fix size count in batch * ethdb/pebble: disable seek compaction * ethdb/pebble: more fixes * ethdb, core, cmd: polish and fixes (axieinfinity#50) * cmd/utils, core/rawdb, ethdb/pebble: address some review concerns * Update flags.go * ethdb/pebble: minor refactors * ethdb/pebble: avoid copy on batch replay * ethdb: fix compilation flaw * cmd: fix test fail due to mismatching error message * cmd/geth, node: rename backingdb to db.engine --------- Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com>
MemTableStopWritesThreshold was set to the max size of all memtables before blocking writing but should be set to the max number of memtables. This is documented [here](https://github.com/cockroachdb/pebble/blob/master/options.go#L738-L742).
* ethdb/pebble: fix range compaction * ethdb/pebble: add comment
…27136) * cmd/utils, node: switch to Pebble as the default db if none exists * node: fall back to LevelDB on platforms not supporting Pebble * core/rawdb, node: default to Pebble at the node level * cmd/geth: fix some tests explicitly using leveldb * ethdb/pebble: allow double closes, makes tests simpler
One difference between pebble and leveldb is that the latter returns error when performing Get on a closed database, the former does a panic. This may be triggered during shutdown (see #27237) This PR changes the pebble driver so we check that the db is not closed already, for several operations. It also adds tests to the db test-suite, so the previously implicit assumption of "not panic:ing at ops on closed database" is covered by tests.
This is likely the culprit behind several data corruption issues, e.g. where data has been written to the freezer, but the deletion from pebble does not go through due to process crash.
Adds an option to disable fsync for database operations. This is to make tests faster.
…elDB (#28224) ethdb, internal/ethapi: support exposing Pebble stats too, besinde LevelDB
* ethdb/pebble: upgrade pebble * ethdb/pebble, go.mod: update pebble to master (aa077af62593) --------- Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too * ethdb/pebble: use Pebble's internal constant calculation
cockroachdb/pebble@422dce9 added Errorf to the Logger interface, this change makes it possible to compile geth with that version of pebble by adding the corresponding method to panicLogger.
Adds 'released' flag to pebbleIterator to avoid double closing cockroachdb/pebble.Iterator as it is an invalid operation. Fixes #28565
The dependency was not really used anyway, so we can get rid of it. Co-authored-by: Felix Lange <fjl@twurst.com>
This commit adds --db.engine to database related commands in Ronin to specify the engine to use between leveldb and pebble.
minh-bq
force-pushed
the
feat/pebble-db
branch
from
9e0177a to
dee6db8
Compare
huyngopt1994
approved these changes
Jan 16, 2024
Francesco4203
pushed a commit
to Francesco4203/ronin
that referenced
this pull request
Jun 18, 2024
…ty#385) * core, ethdb, tests, trie: implement NewBatchWithSize API for batcher (#24392) This PR adds an addtional API called `NewBatchWithSize` for db batcher. It turns out that leveldb batch memory allocation is super inefficient. The main reason is the allocation step of leveldb Batch is too small when the batch size is large. It can take a few second to build a leveldb batch with 100MB size. Luckily, leveldb also offers another API called MakeBatch which can pre-allocate the memory area. So if the approximate size of batch is known in advance, this API can be used in this case. It's needed in new state scheme PR which needs to commit a batch of trie nodes in a single batch. Implement the feature in a seperate PR. * core, ethdb, tests, trie: introduce database snapshot (#24486) * ethdb: pebble backend (64bit platforms only) (#26517) * ethdb: use pebble Co-authored-by: Gary Rong <garyrong0905@gmail.com> foo update * apply suggested changes * flags: go format node: fix ddir lookup mistake accounts/abi/bind: fix go.mod replacement for generated binding deps: update pebble + with fix 32-bit build * ethdb/pebble: respect max memtable size * core/rawdb, ethdb: enable pebble on non-32bit platforms only * core/rawdb: fix build tags, fix some review concerns * core/rawdb: refactor methods for database opening * core/rawdb: remove erroneous build tag * cmd/geth: fix the flag default handling + testcase * cmd/geth: improve testing regarding custom backends * ethdb/pebble, deps: update pebble dependency * core/rawdb: replace method with Open * ethdb/pebble: several updates for pebble (axieinfinity#49) * ethdb/pebble: fix size count in batch * ethdb/pebble: disable seek compaction * ethdb/pebble: more fixes * ethdb, core, cmd: polish and fixes (axieinfinity#50) * cmd/utils, core/rawdb, ethdb/pebble: address some review concerns * Update flags.go * ethdb/pebble: minor refactors * ethdb/pebble: avoid copy on batch replay * ethdb: fix compilation flaw * cmd: fix test fail due to mismatching error message * cmd/geth, node: rename backingdb to db.engine --------- Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com> * ethdb/pebble: fix nil callbacks (#26650) * ethdb: add benchmark test suite (#26659) * ethdb/pebble: Fix `MemTableStopWritesThreshold` (#26692) MemTableStopWritesThreshold was set to the max size of all memtables before blocking writing but should be set to the max number of memtables. This is documented [here](https://github.com/cockroachdb/pebble/blob/master/options.go#L738-L742). * ethdb/pebble: fix range compaction (#26771) * ethdb/pebble: fix range compaction * ethdb/pebble: add comment * ethdb/pebble: fix max memorytable size (#26776) * core/rawdb, ethdb/pebble: disable pebble on openbsd (#26801) * ethdb/pebble: use atomic type (#27014) * cmd/utils, node: switch to Pebble as the default db if none exists (#27136) * cmd/utils, node: switch to Pebble as the default db if none exists * node: fall back to LevelDB on platforms not supporting Pebble * core/rawdb, node: default to Pebble at the node level * cmd/geth: fix some tests explicitly using leveldb * ethdb/pebble: allow double closes, makes tests simpler * ethdb/pebble: prevent shutdown-panic (#27238) One difference between pebble and leveldb is that the latter returns error when performing Get on a closed database, the former does a panic. This may be triggered during shutdown (see #27237) This PR changes the pebble driver so we check that the db is not closed already, for several operations. It also adds tests to the db test-suite, so the previously implicit assumption of "not panic:ing at ops on closed database" is covered by tests. * ethdb/pebble: fix NewBatchWithSize to set db (#27350) * ethdb/pebble: fsync for batch writes (#27522) This is likely the culprit behind several data corruption issues, e.g. where data has been written to the freezer, but the deletion from pebble does not go through due to process crash. * ethdb/pebble: use sync mode for pebble writes (#27615) * core/rawdb, ethdb/pebble: avoid fsync db in tests (#27836) Adds an option to disable fsync for database operations. This is to make tests faster. * ethdb, internal/ethapi: support exposing Pebble stats too, beside LevelDB (#28224) ethdb, internal/ethapi: support exposing Pebble stats too, besinde LevelDB * ethdb/pebble: upgrade pebble to master (aa077af62593) (#28070) * ethdb/pebble: upgrade pebble * ethdb/pebble, go.mod: update pebble to master (aa077af62593) --------- Co-authored-by: Péter Szilágyi <peterke@gmail.com> * ethdb/pebble: luv you linter * ethdb/pebble: add level file metrics (#28271) * cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too (#28335) * cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too * ethdb/pebble: use Pebble's internal constant calculation * ethdb/pebble: cap memory table size as maxMemTableSize-1 (#28444) * ethdb/pebble: add `Errorf` function to panicLogger (#28491) cockroachdb/pebble@422dce9 added Errorf to the Logger interface, this change makes it possible to compile geth with that version of pebble by adding the corresponding method to panicLogger. * ethdb/pebble: don't double-close iterator inside pebbleIterator (#28566) Adds 'released' flag to pebbleIterator to avoid double closing cockroachdb/pebble.Iterator as it is an invalid operation. Fixes #28565 * ethdb/pebble: remove a dependency (#28627) The dependency was not really used anyway, so we can get rid of it. Co-authored-by: Felix Lange <fjl@twurst.com> * cmd: add db.engine flags to Ronin commands This commit adds --db.engine to database related commands in Ronin to specify the engine to use between leveldb and pebble. * entrypoint: add DB_ENGINE environment variable --------- Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com> Co-authored-by: Patrick O'Grady <prohb125@gmail.com> Co-authored-by: s7v7nislands <s7v7nislands@gmail.com> Co-authored-by: Jim McDonald <Jim@mcdee.net> Co-authored-by: Maciej Kulawik <10907694+magicxyyz@users.noreply.github.com> Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de> Co-authored-by: Felix Lange <fjl@twurst.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.