Skip to content

Commit

Permalink
README: add information about v2 and format upgrade
Browse files Browse the repository at this point in the history
  • Loading branch information
RaduBerinde committed Jan 6, 2025
1 parent e768f25 commit 8d697d1
Show file tree
Hide file tree
Showing 2 changed files with 38 additions and 25 deletions.
61 changes: 38 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,16 +86,16 @@ differences.

## RocksDB Compatibility

Pebble strives for forward compatibility with RocksDB 6.2.1 (the latest version
of RocksDB used by CockroachDB). Forward compatibility means that a DB generated
by RocksDB 6.2.1 can be upgraded for use by Pebble. Pebble versions in the `v1`
series may open DBs generated by RocksDB 6.2.1. Since its introduction, Pebble
has adopted various backwards-incompatible format changes that are gated behind
new 'format major versions'. The Pebble `master` branch does not support opening
DBs generated by RocksDB. DBs generated by RocksDB may only be used with recent
versions of Pebble after migrating them through format major version upgrades
using previous versions of Pebble. See the below section of format major
versions.
Pebble `v1` strives for forward compatibility with RocksDB 6.2.1 (the latest
version of RocksDB used by CockroachDB). Forward compatibility means that a DB
generated by RocksDB 6.2.1 can be upgraded for use by Pebble. Pebble versions in
the `v1` series may open DBs generated by RocksDB 6.2.1. Since its introduction,
Pebble has adopted various backwards-incompatible format changes that are gated
behind new 'format major versions'. Pebble `v2` and newer does not support
opening DBs generated by RocksDB. DBs generated by RocksDB may only be used with
recent versions of Pebble after migrating them through format major version
upgrades using previous versions of Pebble. See the below section of format
major versions.

Even the RocksDB-compatible versions of Pebble only provide compatibility with
the subset of functionality and configuration used by CockroachDB. The scope of
Expand Down Expand Up @@ -157,10 +157,13 @@ range of Pebble versions support that format.
| FormatMinTableFormatPebblev1 | 9 | No | v1 |
| FormatPrePebblev1Marked | 10 | Background | v1 |
| FormatSSTableValueBlocks | 12 | No | v1 |
| FormatFlushableIngest | 13 | No | v1, master |
| FormatPrePebblev1MarkedCompacted | 14 | Blocking | v1, master |
| FormatDeleteSizedAndObsolete | 15 | No | v1, master |
| FormatVirtualSSTables | 16 | No | v1, master |
| FormatFlushableIngest | 13 | No | v1, v2, master |
| FormatPrePebblev1MarkedCompacted | 14 | Blocking | v1, v2, master |
| FormatDeleteSizedAndObsolete | 15 | No | v1, v2, master |
| FormatVirtualSSTables | 16 | No | v1, v2, master |
| FormatSyntheticPrefixSuffix | 17 | No | v2, master |
| FormatFlushableIngestExcises | 18 | No | v2, master |
| FormatColumnarBlocks | 19 | No | v2, master |

Upgrading to a format major version with 'Background' in the migration
column may trigger background activity to rewrite physical file
Expand All @@ -171,18 +174,30 @@ writes if upgrading a live database through
`RatchetFormatMajorVersion`, but the method call will not return until
the migration is complete.

Upgrading existing stores can be performed via the `RatchetFormatMajorVersion`
method. If the database does not use a custom comparer, merger, or block
property collectors, the `pebble` tool can also be used, at the latest version
that supports the format. For example:
```
# WARNING: only use if no custom comparer/merger/property collector are necessary.
go run github.com/cockroachdb/pebble/cmd/pebble@v1.1.3 db upgrade <db-dir>
```

For reference, the table below lists the range of supported Pebble format major
versions for CockroachDB releases.

| CockroachDB release | Earliest supported | Latest supported |
|---------------------|------------------------------------|---------------------------|
| 20.1 through 21.1 | FormatMostCompatible | FormatMostCompatible |
| 21.2 | FormatMostCompatible | FormatSetWithDelete |
| 21.2 | FormatMostCompatible | FormatSetWithDelete |
| 22.1 | FormatMostCompatible | FormatSplitUserKeysMarked |
| 22.2 | FormatMostCompatible | FormatPrePebblev1Marked |
| 23.1 | FormatSplitUserKeysMarkedCompacted | FormatFlushableIngest |
| 23.2 | FormatSplitUserKeysMarkedCompacted | FormatVirtualSSTables |
| CockroachDB release | Earliest supported | Latest supported |
|---------------------|------------------------------------|-----------------------------|
| 20.1 through 21.1 | FormatMostCompatible | FormatMostCompatible |
| 21.2 | FormatMostCompatible | FormatSetWithDelete |
| 21.2 | FormatMostCompatible | FormatSetWithDelete |
| 22.1 | FormatMostCompatible | FormatSplitUserKeysMarked |
| 22.2 | FormatMostCompatible | FormatPrePebblev1Marked |
| 23.1 | FormatSplitUserKeysMarkedCompacted | FormatFlushableIngest |
| 23.2 | FormatPrePebblev1Marked | FormatVirtualSSTables |
| 24.1 | FormatFlushableIngest | FormatSyntheticPrefixSuffix |
| 24.2 | FormatVirtualSSTables | FormatSyntheticPrefixSuffix |
| 24.3 | FormatSyntheticPrefixSuffix | FormatColumnarBlocks |

## Pedigree

Expand Down
2 changes: 0 additions & 2 deletions format_major_version.go
Original file line number Diff line number Diff line change
Expand Up @@ -197,8 +197,6 @@ const (
// block.
FormatColumnarBlocks

// TODO(msbutler): add major version for synthetic suffixes

// -- Add new versions here --

// FormatNewest is the most recent format major version.
Expand Down

0 comments on commit 8d697d1

Please sign in to comment.