Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sstable: use github.com/klauspost/compress for snappy #3693

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

sstable: use github.com/klauspost/compress for snappy #3693

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6773026
go.mod: update github.com/klauspost/compress to v1.17.9
jbowens Jun 24, 2024
f50c379
sstable: use github.com/klauspost/compress for snappy
jbowens Jun 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ require (
github.com/ghemawat/stream v0.0.0-20171120220530-696b145b53b9
github.com/golang/snappy v0.0.4
github.com/guptarohit/asciigraph v0.5.5
github.com/klauspost/compress v1.16.7
github.com/klauspost/compress v1.17.9
github.com/kr/pretty v0.3.1
github.com/pkg/errors v0.9.1
github.com/pmezard/go-difflib v1.0.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,8 +229,8 @@ github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5/go.mod h1:7Id9E
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.16.7 h1:2mk3MPGNzKyxErAw8YaohYh69+pa4sIQSC0fPGCFR9I=
github.com/klauspost/compress v1.16.7/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
Expand Down
2 changes: 1 addition & 1 deletion replay/testdata/corpus/simple
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,4 +82,4 @@ simple/MANIFEST-000001:
simple/MANIFEST-000008:
size: 122
simple/000007.sst:
size: 614
size: 625
4 changes: 2 additions & 2 deletions replay/testdata/replay
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ tree
89 000004.log
586 000005.sst
49 000006.log
614 000007.sst
625 000007.sst
0 LOCK
98 MANIFEST-000001
122 MANIFEST-000008
1240 OPTIONS-000003
0 marker.format-version.000001.013
0 marker.manifest.000002.MANIFEST-000008
simple/
614 000007.sst
625 000007.sst
98 MANIFEST-000001
122 MANIFEST-000008
checkpoint/
Expand Down
8 changes: 4 additions & 4 deletions replay/testdata/replay_paced
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,23 +5,23 @@ tree
----
/
build/
864 000005.sst
911 000005.sst
560 000007.sst
89 000009.log
560 000010.sst
200 000012.log
614 000013.sst
625 000013.sst
0 LOCK
122 MANIFEST-000008
205 MANIFEST-000011
1240 OPTIONS-000003
0 marker.format-version.000001.013
0 marker.manifest.000003.MANIFEST-000011
high_read_amp/
614 000013.sst
625 000013.sst
205 MANIFEST-000011
checkpoint/
864 000005.sst
911 000005.sst
560 000007.sst
39 000009.log
560 000010.sst
Expand Down
2 changes: 1 addition & 1 deletion sstable/compression.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"github.com/cockroachdb/errors"
"github.com/cockroachdb/pebble/internal/base"
"github.com/cockroachdb/pebble/internal/cache"
"github.com/golang/snappy"
"github.com/klauspost/compress/snappy"
)

func decompressedLen(blockType blockType, b []byte) (int, int, error) {
Expand Down
2 changes: 1 addition & 1 deletion sstable/properties_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ func TestPropertiesLoad(t *testing.T) {
CompressionOptions: "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ",
},
ComparerName: "leveldb.BytewiseComparator",
DataSize: 13913,
DataSize: 13712,
IndexSize: 325,
MergerName: "nullptr",
NumDataBlocks: 14,
Expand Down
Binary file modified sstable/testdata/h.sst
View file
Open in desktop
Binary file not shown.
Binary file modified sstable/testdata/h.table-bloom.sst
View file
Open in desktop
Binary file not shown.
20 changes: 10 additions & 10 deletions sstable/testdata/readerstats_Pebblev3/iter
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ e@27.SET.37:eAT27
e@26.SET.36:eAT26
----
index entries:
f: size 228
f: size 226

# Iterating across older versions and fetching the older version values.
iter
Expand All @@ -33,13 +33,13 @@ next
stats
----
<c@10:10>
{BlockBytes:251 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:0 ValueBytes:0 ValueBytesFetched:0}}
{BlockBytes:249 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:0 ValueBytes:0 ValueBytesFetched:0}}
<c@9:9>
{BlockBytes:328 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:1 ValueBytes:4 ValueBytesFetched:4}}
{BlockBytes:316 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:1 ValueBytes:4 ValueBytesFetched:4}}
<c@8:8>
{BlockBytes:328 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:2 ValueBytes:8 ValueBytesFetched:8}}
{BlockBytes:316 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:2 ValueBytes:8 ValueBytesFetched:8}}
<d@7:9>
{BlockBytes:328 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:2 ValueBytes:8 ValueBytesFetched:8}}
{BlockBytes:316 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:2 ValueBytes:8 ValueBytesFetched:8}}

# seek-ge e@37 starts at the restart point at the beginning of the block and
# iterates over 3 irrelevant separated versions before getting to e@37
Expand All @@ -55,12 +55,12 @@ next
stats
----
<e@37:47>
{BlockBytes:328 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:4 ValueBytes:18 ValueBytesFetched:5}}
{BlockBytes:316 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:4 ValueBytes:18 ValueBytesFetched:5}}
<e@36:46>
<e@35:45>
<e@34:44>
<e@33:43>
{BlockBytes:328 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:8 ValueBytes:38 ValueBytesFetched:25}}
{BlockBytes:316 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:8 ValueBytes:38 ValueBytesFetched:25}}

# seek-ge e@26 lands at the restart point e@26.
iter
Expand All @@ -72,8 +72,8 @@ prev
stats
----
<e@26:36>
{BlockBytes:328 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:1 ValueBytes:5 ValueBytesFetched:5}}
{BlockBytes:316 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:1 ValueBytes:5 ValueBytesFetched:5}}
<e@27:37>
{BlockBytes:328 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:2 ValueBytes:10 ValueBytesFetched:10}}
{BlockBytes:316 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:2 ValueBytes:10 ValueBytesFetched:10}}
<e@28:38>
{BlockBytes:328 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:3 ValueBytes:15 ValueBytesFetched:15}}
{BlockBytes:316 BlockBytesInCache:0 BlockReadDuration:0s KeyBytes:0 ValueBytes:0 PointCount:0 PointsCoveredByRangeTombstones:0 SeparatedPointValue:{Count:3 ValueBytes:15 ValueBytesFetched:15}}
10 changes: 5 additions & 5 deletions sstable/testdata/writer_v3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,11 +288,11 @@ layout
81 index (22)
108 index (22)
135 index (22)
162 top-index (51)
218 properties (453)
676 meta-index (33)
714 footer (53)
767 EOF
162 top-index (50)
217 properties (453)
675 meta-index (33)
713 footer (53)
766 EOF

scan
----
Expand Down
80 changes: 40 additions & 40 deletions sstable/testdata/writer_value_blocks
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ layout
350 [restart 288]
354 [restart 308]
358 [restart 331]
352 [trailer compression=snappy checksum=0x8bd0d63a]
352 [trailer compression=snappy checksum=0xc5f9e812]
357 value-block (11)
373 value-block (15)
393 value-index (8)
Expand Down Expand Up @@ -296,7 +296,7 @@ c@5#6,DEL:

layout
----
0 data (66)
0 data (56)
0 record (17 = 3 [0] + 11 + 3) [restart]
b@5#7,SET:b5
17 record (14 = 3 [1] + 10 + 1)
Expand All @@ -306,41 +306,41 @@ layout
45 record (13 = 3 [1] + 10 + 0)
c@5#6,DEL:
58 [restart 0]
66 [trailer compression=none checksum=0x4e91250f]
71 index (22)
71 block:0/66 [restart]
85 [restart 71]
93 [trailer compression=none checksum=0xf80f5bcf]
98 properties (479)
98 obsolete-key (16) [restart]
114 pebble.raw.point-tombstone.key.size (39)
153 rocksdb.block.based.table.index.type (43)
196 rocksdb.comparator (37)
233 rocksdb.compression (16)
249 rocksdb.compression_options (106)
355 rocksdb.data.size (13)
368 rocksdb.deleted.keys (15)
383 rocksdb.filter.size (15)
398 rocksdb.index.size (14)
412 rocksdb.merge.operands (18)
430 rocksdb.merge.operator (24)
454 rocksdb.num.data.blocks (19)
473 rocksdb.num.entries (11)
484 rocksdb.num.range-deletions (19)
503 rocksdb.property.collectors (36)
539 rocksdb.raw.key.size (16)
555 rocksdb.raw.value.size (14)
569 [restart 98]
577 [trailer compression=none checksum=0xdd2d145f]
582 meta-index (32)
582 rocksdb.properties block:98/479 [restart]
606 [restart 582]
614 [trailer compression=none checksum=0xbfc95718]
619 footer (53)
619 checksum type: crc32c
620 meta: offset=582, length=32
623 index: offset=71, length=22
625 [padding]
660 version: 4
664 magic number: 0xf09faab3f09faab3
672 EOF
56 [trailer compression=snappy checksum=0xb816bb83]
61 index (22)
61 block:0/56 [restart]
75 [restart 61]
83 [trailer compression=none checksum=0x94eb7290]
88 properties (479)
88 obsolete-key (16) [restart]
104 pebble.raw.point-tombstone.key.size (39)
143 rocksdb.block.based.table.index.type (43)
186 rocksdb.comparator (37)
223 rocksdb.compression (16)
239 rocksdb.compression_options (106)
345 rocksdb.data.size (13)
358 rocksdb.deleted.keys (15)
373 rocksdb.filter.size (15)
388 rocksdb.index.size (14)
402 rocksdb.merge.operands (18)
420 rocksdb.merge.operator (24)
444 rocksdb.num.data.blocks (19)
463 rocksdb.num.entries (11)
474 rocksdb.num.range-deletions (19)
493 rocksdb.property.collectors (36)
529 rocksdb.raw.key.size (16)
545 rocksdb.raw.value.size (14)
559 [restart 88]
567 [trailer compression=none checksum=0x19159187]
572 meta-index (32)
572 rocksdb.properties block:88/479 [restart]
596 [restart 572]
604 [trailer compression=none checksum=0xd592ce8e]
609 footer (53)
609 checksum type: crc32c
610 meta: offset=572, length=32
613 index: offset=61, length=22
615 [padding]
650 version: 4
654 magic number: 0xf09faab3f09faab3
662 EOF
90 changes: 45 additions & 45 deletions testdata/checkpoint
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,28 +211,28 @@ close: db/000009.sst
sync: db
sync: db/MANIFEST-000001
open: db/000005.sst (options: *vfs.randomReadsOption)
read-at(558, 53): db/000005.sst
read-at(521, 37): db/000005.sst
read-at(74, 447): db/000005.sst
read-at(557, 53): db/000005.sst
read-at(520, 37): db/000005.sst
read-at(73, 447): db/000005.sst
open: db/000009.sst (options: *vfs.randomReadsOption)
read-at(553, 53): db/000009.sst
read-at(516, 37): db/000009.sst
read-at(69, 447): db/000009.sst
read-at(548, 53): db/000009.sst
read-at(511, 37): db/000009.sst
read-at(64, 447): db/000009.sst
open: db/000007.sst (options: *vfs.randomReadsOption)
read-at(558, 53): db/000007.sst
read-at(521, 37): db/000007.sst
read-at(74, 447): db/000007.sst
read-at(47, 27): db/000005.sst
read-at(557, 53): db/000007.sst
read-at(520, 37): db/000007.sst
read-at(73, 447): db/000007.sst
read-at(46, 27): db/000005.sst
open: db/000005.sst (options: *vfs.sequentialReadsOption)
read-at(0, 47): db/000005.sst
read-at(47, 27): db/000007.sst
read-at(0, 46): db/000005.sst
read-at(46, 27): db/000007.sst
open: db/000007.sst (options: *vfs.sequentialReadsOption)
read-at(0, 47): db/000007.sst
read-at(0, 46): db/000007.sst
create: db/000010.sst
close: db/000005.sst
read-at(42, 27): db/000009.sst
read-at(37, 27): db/000009.sst
open: db/000009.sst (options: *vfs.sequentialReadsOption)
read-at(0, 42): db/000009.sst
read-at(0, 37): db/000009.sst
close: db/000007.sst
close: db/000009.sst
sync-data: db/000010.sst
Expand Down Expand Up @@ -291,17 +291,17 @@ close: checkpoints/checkpoint1/000006.log
scan checkpoints/checkpoint1
----
open: checkpoints/checkpoint1/000007.sst (options: *vfs.randomReadsOption)
read-at(558, 53): checkpoints/checkpoint1/000007.sst
read-at(521, 37): checkpoints/checkpoint1/000007.sst
read-at(74, 447): checkpoints/checkpoint1/000007.sst
read-at(47, 27): checkpoints/checkpoint1/000007.sst
read-at(0, 47): checkpoints/checkpoint1/000007.sst
read-at(557, 53): checkpoints/checkpoint1/000007.sst
read-at(520, 37): checkpoints/checkpoint1/000007.sst
read-at(73, 447): checkpoints/checkpoint1/000007.sst
read-at(46, 27): checkpoints/checkpoint1/000007.sst
read-at(0, 46): checkpoints/checkpoint1/000007.sst
open: checkpoints/checkpoint1/000005.sst (options: *vfs.randomReadsOption)
read-at(558, 53): checkpoints/checkpoint1/000005.sst
read-at(521, 37): checkpoints/checkpoint1/000005.sst
read-at(74, 447): checkpoints/checkpoint1/000005.sst
read-at(47, 27): checkpoints/checkpoint1/000005.sst
read-at(0, 47): checkpoints/checkpoint1/000005.sst
read-at(557, 53): checkpoints/checkpoint1/000005.sst
read-at(520, 37): checkpoints/checkpoint1/000005.sst
read-at(73, 447): checkpoints/checkpoint1/000005.sst
read-at(46, 27): checkpoints/checkpoint1/000005.sst
read-at(0, 46): checkpoints/checkpoint1/000005.sst
a 1
b 5
c 3
Expand Down Expand Up @@ -358,11 +358,11 @@ close: checkpoints/checkpoint2/000006.log
scan checkpoints/checkpoint2
----
open: checkpoints/checkpoint2/000007.sst (options: *vfs.randomReadsOption)
read-at(558, 53): checkpoints/checkpoint2/000007.sst
read-at(521, 37): checkpoints/checkpoint2/000007.sst
read-at(74, 447): checkpoints/checkpoint2/000007.sst
read-at(47, 27): checkpoints/checkpoint2/000007.sst
read-at(0, 47): checkpoints/checkpoint2/000007.sst
read-at(557, 53): checkpoints/checkpoint2/000007.sst
read-at(520, 37): checkpoints/checkpoint2/000007.sst
read-at(73, 447): checkpoints/checkpoint2/000007.sst
read-at(46, 27): checkpoints/checkpoint2/000007.sst
read-at(0, 46): checkpoints/checkpoint2/000007.sst
b 5
d 7
e 8
Expand Down Expand Up @@ -400,17 +400,17 @@ close: checkpoints/checkpoint3/000006.log
scan checkpoints/checkpoint3
----
open: checkpoints/checkpoint3/000007.sst (options: *vfs.randomReadsOption)
read-at(558, 53): checkpoints/checkpoint3/000007.sst
read-at(521, 37): checkpoints/checkpoint3/000007.sst
read-at(74, 447): checkpoints/checkpoint3/000007.sst
read-at(47, 27): checkpoints/checkpoint3/000007.sst
read-at(0, 47): checkpoints/checkpoint3/000007.sst
read-at(557, 53): checkpoints/checkpoint3/000007.sst
read-at(520, 37): checkpoints/checkpoint3/000007.sst
read-at(73, 447): checkpoints/checkpoint3/000007.sst
read-at(46, 27): checkpoints/checkpoint3/000007.sst
read-at(0, 46): checkpoints/checkpoint3/000007.sst
open: checkpoints/checkpoint3/000005.sst (options: *vfs.randomReadsOption)
read-at(558, 53): checkpoints/checkpoint3/000005.sst
read-at(521, 37): checkpoints/checkpoint3/000005.sst
read-at(74, 447): checkpoints/checkpoint3/000005.sst
read-at(47, 27): checkpoints/checkpoint3/000005.sst
read-at(0, 47): checkpoints/checkpoint3/000005.sst
read-at(557, 53): checkpoints/checkpoint3/000005.sst
read-at(520, 37): checkpoints/checkpoint3/000005.sst
read-at(73, 447): checkpoints/checkpoint3/000005.sst
read-at(46, 27): checkpoints/checkpoint3/000005.sst
read-at(0, 46): checkpoints/checkpoint3/000005.sst
a 1
b 5
c 3
Expand Down Expand Up @@ -550,11 +550,11 @@ e 8
f 9
g 10
open: checkpoints/checkpoint4/000011.sst (options: *vfs.randomReadsOption)
read-at(526, 53): checkpoints/checkpoint4/000011.sst
read-at(489, 37): checkpoints/checkpoint4/000011.sst
read-at(70, 419): checkpoints/checkpoint4/000011.sst
read-at(43, 27): checkpoints/checkpoint4/000011.sst
read-at(0, 43): checkpoints/checkpoint4/000011.sst
read-at(524, 53): checkpoints/checkpoint4/000011.sst
read-at(487, 37): checkpoints/checkpoint4/000011.sst
read-at(68, 419): checkpoints/checkpoint4/000011.sst
read-at(41, 27): checkpoints/checkpoint4/000011.sst
read-at(0, 41): checkpoints/checkpoint4/000011.sst
h 11
i i
k k
Expand Down
Loading
Loading