Skip to content

Commit

Permalink
ethdb/pebble: cherry pick pebble backend from go-ethereum (axieinfini…
Browse files Browse the repository at this point in the history 
…ty#385)

* core, ethdb, tests, trie: implement NewBatchWithSize API for batcher (#24392)

This PR adds an addtional API called `NewBatchWithSize` for db
batcher. It turns out that leveldb batch memory allocation is
super inefficient. The main reason is the allocation step of
leveldb Batch is too small when the batch size is large. It can
take a few second to build a leveldb batch with 100MB size.

Luckily, leveldb also offers another API called MakeBatch which can
pre-allocate the memory area. So if the approximate size of batch is
known in advance, this API can be used in this case.

It's needed in new state scheme PR which needs to commit a batch of
trie nodes in a single batch. Implement the feature in a seperate PR.

* core, ethdb, tests, trie: introduce database snapshot (#24486)

* ethdb: pebble backend (64bit platforms only) (#26517)

* ethdb: use pebble

Co-authored-by: Gary Rong <garyrong0905@gmail.com>

foo

update

* apply suggested changes

* flags: go format

node: fix ddir lookup mistake

accounts/abi/bind: fix go.mod replacement for generated binding

deps: update pebble + with fix 32-bit build

* ethdb/pebble: respect max memtable size

* core/rawdb, ethdb: enable pebble on non-32bit platforms only

* core/rawdb: fix build tags, fix some review concerns

* core/rawdb: refactor methods for database opening

* core/rawdb: remove erroneous build tag

* cmd/geth: fix the flag default handling + testcase

* cmd/geth: improve testing regarding custom backends

* ethdb/pebble, deps: update pebble dependency

* core/rawdb: replace method with Open

* ethdb/pebble: several updates for pebble (axieinfinity#49)

* ethdb/pebble: fix size count in batch

* ethdb/pebble: disable seek compaction

* ethdb/pebble: more fixes

* ethdb, core, cmd: polish and fixes (axieinfinity#50)

* cmd/utils, core/rawdb, ethdb/pebble: address some review concerns

* Update flags.go

* ethdb/pebble: minor refactors

* ethdb/pebble: avoid copy on batch replay

* ethdb: fix compilation flaw

* cmd: fix test fail due to mismatching error message

* cmd/geth, node: rename backingdb to db.engine

---------

Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>

* ethdb/pebble: fix nil callbacks (#26650)

* ethdb: add benchmark test suite (#26659)

* ethdb/pebble: Fix `MemTableStopWritesThreshold` (#26692)

MemTableStopWritesThreshold was set to the max size of all memtables before blocking writing but should be set to the max number of memtables. This is documented [here](https://github.com/cockroachdb/pebble/blob/master/options.go#L738-L742).

* ethdb/pebble: fix range compaction (#26771)

* ethdb/pebble: fix range compaction

* ethdb/pebble: add comment

* ethdb/pebble: fix max memorytable size (#26776)

* core/rawdb, ethdb/pebble: disable pebble on openbsd (#26801)

* ethdb/pebble: use atomic type (#27014)

* cmd/utils, node: switch to Pebble as the default db if none exists (#27136)

* cmd/utils, node: switch to Pebble as the default db if none exists

* node: fall back to LevelDB on platforms not supporting Pebble

* core/rawdb, node: default to Pebble at the node level

* cmd/geth: fix some tests explicitly using leveldb

* ethdb/pebble: allow double closes, makes tests simpler

* ethdb/pebble: prevent shutdown-panic (#27238)

One difference between pebble and leveldb is that the latter returns error when performing Get on a closed database, the former does a panic. This may be triggered during shutdown (see #27237)

This PR changes the pebble driver so we check that the db is not closed already, for several operations. It also adds tests to the db test-suite, so the previously implicit assumption of "not panic:ing at ops on closed database" is covered by tests.

* ethdb/pebble: fix NewBatchWithSize to set db (#27350)

* ethdb/pebble: fsync for batch writes (#27522)

This is likely the culprit behind several data corruption issues, e.g. where data has been
written to the freezer, but the deletion from pebble does not go through due to process
crash.

* ethdb/pebble: use sync mode for pebble writes (#27615)

* core/rawdb, ethdb/pebble: avoid fsync db in tests (#27836)

Adds an option to disable fsync for database operations.
This is to make tests faster.

* ethdb, internal/ethapi: support exposing Pebble stats too, beside LevelDB (#28224)

ethdb, internal/ethapi: support exposing Pebble stats too, besinde LevelDB

* ethdb/pebble: upgrade pebble to master (aa077af62593) (#28070)

* ethdb/pebble: upgrade pebble

* ethdb/pebble, go.mod: update pebble to master (aa077af62593)

---------

Co-authored-by: Péter Szilágyi <peterke@gmail.com>

* ethdb/pebble: luv you linter

* ethdb/pebble: add level file metrics (#28271)

* cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too (#28335)

* cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too

* ethdb/pebble: use Pebble's internal constant calculation

* ethdb/pebble: cap memory table size as maxMemTableSize-1 (#28444)

* ethdb/pebble: add `Errorf` function to panicLogger (#28491)

cockroachdb/pebble@422dce9 added Errorf to the Logger interface, this change makes it possible to compile geth with that version of pebble by adding the corresponding method to panicLogger.

* ethdb/pebble: don't double-close iterator inside pebbleIterator (#28566)

Adds 'released' flag to pebbleIterator to avoid double closing cockroachdb/pebble.Iterator as it is an invalid operation.

Fixes #28565

* ethdb/pebble: remove a dependency (#28627)

The dependency was not really used anyway, so we can get rid of it.

Co-authored-by: Felix Lange <fjl@twurst.com>

* cmd: add db.engine flags to Ronin commands

This commit adds --db.engine to database related commands in Ronin to specify
the engine to use between leveldb and pebble.

* entrypoint: add DB_ENGINE environment variable

---------

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Patrick O'Grady <prohb125@gmail.com>
Co-authored-by: s7v7nislands <s7v7nislands@gmail.com>
Co-authored-by: Jim McDonald <Jim@mcdee.net>
Co-authored-by: Maciej Kulawik <10907694+magicxyyz@users.noreply.github.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
  • Loading branch information
@minh-bq @rjl493456442
@holiman @jwasinger @karalabe @patrick-ogrady @s7v7nislands @mcdee @magicxyyz @MariusVanDerWijden @fjl
11 people committed Feb 16, 2024
1 parent b7474db commit c331512
Show file tree
Hide file tree
Showing 33 changed files with 1,796 additions and 51 deletions.
6 changes: 6 additions & 0 deletions cmd/ronin/chaincmd.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ var (
ArgsUsage: "<genesisPath>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.ForceOverrideChainConfigFlag,
},
Category: "BLOCKCHAIN COMMANDS",
Expand Down Expand Up @@ -82,6 +83,7 @@ The dumpgenesis command dumps the genesis block configuration in JSON format to
ArgsUsage: "<filename> (<filename 2> ... <filename N>) ",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.CacheFlag,
utils.SyncModeFlag,
utils.GCModeFlag,
Expand Down Expand Up @@ -119,6 +121,7 @@ processing will proceed even if an individual RLP-file import failure occurs.`,
ArgsUsage: "<filename> [<blockNumFirst> <blockNumLast>]",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.CacheFlag,
utils.SyncModeFlag,
},
Expand All @@ -137,6 +140,7 @@ be gzipped.`,
ArgsUsage: "<datafile>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.CacheFlag,
utils.SyncModeFlag,
},
Expand All @@ -153,6 +157,7 @@ It's deprecated, please use "geth db import" instead.
ArgsUsage: "<dumpfile>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.CacheFlag,
utils.SyncModeFlag,
},
Expand All @@ -169,6 +174,7 @@ It's deprecated, please use "geth db export" instead.
ArgsUsage: "[? <blockHash> | <blockNum>]",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.CacheFlag,
utils.IterativeOutputFlag,
utils.ExcludeCodeFlag,
Expand Down
2 changes: 1 addition & 1 deletion cmd/ronin/dao_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ func testDAOForkBlockNewChain(t *testing.T, test int, genesis string, expectBloc
}
// Retrieve the DAO config flag from the database
path := filepath.Join(datadir, "ronin", "chaindata")
db, err := rawdb.NewLevelDBDatabase(path, 0, 0, "", false)
db, err := rawdb.NewPebbleDBDatabase(path, 0, 0, "", false, true)
if err != nil {
t.Fatalf("test %d: failed to open test database: %v", test, err)
}
Expand Down
10 changes: 10 additions & 0 deletions cmd/ronin/dbcmd.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ Remove blockchain and state databases`,
ArgsUsage: "<prefix> <start>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.AncientFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
Expand All @@ -94,6 +95,7 @@ Remove blockchain and state databases`,
Usage: "Print leveldb statistics",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand All @@ -108,6 +110,7 @@ Remove blockchain and state databases`,
Usage: "Compact leveldb database. WARNING: May take a very long time",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand All @@ -128,6 +131,7 @@ corruption if it is aborted during execution'!`,
ArgsUsage: "<hex-encoded key>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand All @@ -144,6 +148,7 @@ corruption if it is aborted during execution'!`,
ArgsUsage: "<hex-encoded key>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand All @@ -161,6 +166,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<hex-encoded key> <hex-encoded value>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand All @@ -178,6 +184,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<hex-encoded storage trie root> <hex-encoded start (optional)> <int max elements (optional)>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand All @@ -194,6 +201,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<type> <start (int)> <end (int)>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand All @@ -210,6 +218,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<dumpfile> <start (optional)",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand All @@ -225,6 +234,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<type> <dumpfile>",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.SyncModeFlag,
utils.MainnetFlag,
utils.RopstenFlag,
Expand Down
101 changes: 101 additions & 0 deletions cmd/ronin/genesis_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,11 @@
package main

import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strconv"
"testing"
)

Expand Down Expand Up @@ -71,6 +73,7 @@ var customGenesisTests = []struct {
// Tests that initializing Geth with a custom genesis block and chain definitions
// work properly.
func TestCustomGenesis(t *testing.T) {
t.Parallel()
for i, tt := range customGenesisTests {
// Create a temporary data directory to use and inspect later
datadir := tmpdir(t)
Expand All @@ -92,3 +95,101 @@ func TestCustomGenesis(t *testing.T) {
geth.ExpectExit()
}
}

// TestCustomBackend that the backend selection and detection (leveldb vs pebble) works properly.
func TestCustomBackend(t *testing.T) {
t.Parallel()
// Test pebble, but only on 64-bit platforms
if strconv.IntSize != 64 {
t.Skip("Custom backends are only available on 64-bit platform")
}
genesis := `{
"alloc" : {},
"coinbase" : "0x0000000000000000000000000000000000000000",
"difficulty" : "0x20000",
"extraData" : "",
"gasLimit" : "0x2fefd8",
"nonce" : "0x0000000000001338",
"mixhash" : "0x0000000000000000000000000000000000000000000000000000000000000000",
"parentHash" : "0x0000000000000000000000000000000000000000000000000000000000000000",
"timestamp" : "0x00",
"config" : {}
}`
type backendTest struct {
initArgs []string
initExpect string
execArgs []string
execExpect string
}
testfunc := func(t *testing.T, tt backendTest) error {
// Create a temporary data directory to use and inspect later
datadir := t.TempDir()

// Initialize the data directory with the custom genesis block
json := filepath.Join(datadir, "genesis.json")
if err := os.WriteFile(json, []byte(genesis), 0600); err != nil {
return fmt.Errorf("failed to write genesis file: %v", err)
}
{ // Init
args := append(tt.initArgs, "--datadir", datadir, "init", json)
geth := runGeth(t, args...)
geth.ExpectRegexp(tt.initExpect)
geth.ExpectExit()
}
{ // Exec + query
args := append(tt.execArgs, "--networkid", "1337", "--syncmode=full", "--cache", "16",
"--datadir", datadir, "--maxpeers", "0", "--port", "0",
"--nodiscover", "--nat", "none", "--ipcdisable",
"--exec", "eth.getBlock(0).nonce", "console")
geth := runGeth(t, args...)
geth.ExpectRegexp(tt.execExpect)
geth.ExpectExit()
}
return nil
}
for i, tt := range []backendTest{
{ // When not specified, it should default to pebble
execArgs: []string{"--db.engine", "pebble"},
execExpect: "0x0000000000001338",
},
{ // Explicit leveldb
initArgs: []string{"--db.engine", "leveldb"},
execArgs: []string{"--db.engine", "leveldb"},
execExpect: "0x0000000000001338",
},
{ // Explicit leveldb first, then autodiscover
initArgs: []string{"--db.engine", "leveldb"},
execExpect: "0x0000000000001338",
},
{ // Explicit pebble
initArgs: []string{"--db.engine", "pebble"},
execArgs: []string{"--db.engine", "pebble"},
execExpect: "0x0000000000001338",
},
{ // Explicit pebble, then auto-discover
initArgs: []string{"--db.engine", "pebble"},
execExpect: "0x0000000000001338",
},
{ // Can't start pebble on top of leveldb
initArgs: []string{"--db.engine", "leveldb"},
execArgs: []string{"--db.engine", "pebble"},
execExpect: `Fatal: Failed to register the Ethereum service: db.engine choice was pebble but found pre-existing leveldb database in specified data directory`,
},
{ // Can't start leveldb on top of pebble
initArgs: []string{"--db.engine", "pebble"},
execArgs: []string{"--db.engine", "leveldb"},
execExpect: `Fatal: Failed to register the Ethereum service: db.engine choice was leveldb but found pre-existing pebble database in specified data directory`,
},
{ // Reject invalid backend choice
initArgs: []string{"--db.engine", "mssql"},
initExpect: `Fatal: Invalid choice for db.engine 'mssql', allowed 'leveldb' or 'pebble'`,
// Since the init fails, this will return the (default) mainnet genesis
// block nonce
execExpect: `0x0000000000000042`,
},
} {
if err := testfunc(t, tt); err != nil {
t.Fatalf("test %d-leveldb: %v", i, err)
}
}
}
1 change: 1 addition & 0 deletions cmd/ronin/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ var (
utils.BlsWalletPath,
utils.DisableRoninProtocol,
utils.AdditionalChainEventFlag,
utils.DBEngineFlag,
}

rpcFlags = []cli.Flag{
Expand Down
5 changes: 5 additions & 0 deletions cmd/ronin/snapshot.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ var (
Category: "MISCELLANEOUS COMMANDS",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.AncientFlag,
utils.RopstenFlag,
utils.SepoliaFlag,
Expand Down Expand Up @@ -91,6 +92,7 @@ the trie clean cache with default directory will be deleted.
Category: "MISCELLANEOUS COMMANDS",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.AncientFlag,
utils.RopstenFlag,
utils.SepoliaFlag,
Expand All @@ -112,6 +114,7 @@ In other words, this command does the snapshot to trie conversion.
Category: "MISCELLANEOUS COMMANDS",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.AncientFlag,
utils.RopstenFlag,
utils.SepoliaFlag,
Expand All @@ -135,6 +138,7 @@ It's also usable without snapshot enabled.
Category: "MISCELLANEOUS COMMANDS",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.AncientFlag,
utils.RopstenFlag,
utils.SepoliaFlag,
Expand All @@ -159,6 +163,7 @@ It's also usable without snapshot enabled.
Category: "MISCELLANEOUS COMMANDS",
Flags: []cli.Flag{
utils.DataDirFlag,
utils.DBEngineFlag,
utils.AncientFlag,
utils.RopstenFlag,
utils.SepoliaFlag,
Expand Down
1 change: 1 addition & 0 deletions cmd/ronin/usage.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ var AppHelpFlagGroups = []flags.FlagGroup{
utils.StoreInternalTransactions,
utils.DisableRoninProtocol,
utils.AdditionalChainEventFlag,
utils.DBEngineFlag,
},
},
{
Expand Down
17 changes: 15 additions & 2 deletions cmd/utils/flags.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,11 @@ var (
Name: "datadir.minfreedisk",
Usage: "Minimum free disk space in MB, once reached triggers auto shut down (default = --cache.gc converted to MB, 0 = disabled)",
}
DBEngineFlag = cli.StringFlag{
Name: "db.engine",
Usage: "Backing database implementation to use ('leveldb' or 'pebble')",
Value: node.DefaultConfig.DBEngine,
}
KeyStoreDirFlag = DirectoryFlag{
Name: "keystore",
Usage: "Directory for the keystore (default = inside the datadir)",
Expand Down Expand Up @@ -924,12 +929,12 @@ var (
}

MockValidatorsFlag = cli.StringFlag{
Name: "mock.validators",
Name: "mock.validators",
Usage: "List of mock validators",
}

MockBlsPublicKeysFlag = cli.StringFlag{
Name: "mock.blspublickeys",
Name: "mock.blspublickeys",
Usage: "List of mock bls public keys which are reflect 1:1 with mock.validators",
}
)
Expand Down Expand Up @@ -1403,6 +1408,14 @@ func SetNodeConfig(ctx *cli.Context, cfg *node.Config) {
if ctx.GlobalIsSet(EnableSigningMethodsFlag.Name) {
cfg.EnableSigningMethods = ctx.GlobalBool(EnableSigningMethodsFlag.Name)
}
if ctx.GlobalIsSet(DBEngineFlag.Name) {
dbEngine := ctx.GlobalString(DBEngineFlag.Name)
if dbEngine != "leveldb" && dbEngine != "pebble" {
Fatalf("Invalid choice for db.engine '%s', allowed 'leveldb' or 'pebble'", dbEngine)
}
log.Info(fmt.Sprintf("Using %s as db engine", dbEngine))
cfg.DBEngine = dbEngine
}
}

func setFastFinality(ctx *cli.Context, cfg *node.Config) {
Expand Down
Loading

0 comments on commit c331512

Please sign in to comment.