How to identify corrupted chunk?

[jimethn]

Thanos Compact is failing with a checksum mismatch when it tries to load a block:

level=error ts=2022-07-14T10:58:57.504159689Z caller=compact.go:495 msg="critical error detected; halting" err="compaction: group 0@12040338854421827242: compact blocks
[/var/thanos/compact/compact/0@12040338854421827242/01FA9RETDX9QJWADB3YS92NMAD
/var/thanos/compact/compact/0@12040338854421827242/01FAFHZ0P0A8WKWQ5TQR2BWXWP
/var/thanos/compact/compact/0@12040338854421827242/01FANBC0GJAFTEP9DNJERHCRV6
/var/thanos/compact/compact/0@12040338854421827242/01FAV4Q3VHNZ1NYMKF55T5JQ9Y
/var/thanos/compact/compact/0@12040338854421827242/01FB0Y3ZFZCESQ5VHEGBXNHD35]:
populate block: chunk iter: cannot populate chunk 385: checksum mismatch expected:6b128655, actual:1c69f469"

Other discussions have suggested deleting the corrupted chunk. Okay, so one of these blocks contains a corrupted chunk, but which one? Fortunately, the in-progress blocks are in the local data dir.

/var/thanos/compact/compact/0@12040338854421827242 $ ls
01FA9RETDX9QJWADB3YS92NMAD  01FANBC0GJAFTEP9DNJERHCRV6  01FB0Y3ZFZCESQ5VHEGBXNHD35
01FAFHZ0P0A8WKWQ5TQR2BWXWP  01FAV4Q3VHNZ1NYMKF55T5JQ9Y

Unfortunately, there is no chunk 385. In fact no chunk is numbered beyond double digits.

/var/thanos/compact/compact/0@12040338854421827242 $ find . -name '*385*'
/var/thanos/compact/compact/0@12040338854421827242 $ 
/var/thanos/compact/compact/0@12040338854421827242 $ find . -wholename '*chunks*'
./01FB0Y3ZFZCESQ5VHEGBXNHD35/chunks
./01FB0Y3ZFZCESQ5VHEGBXNHD35/chunks/000023
./01FB0Y3ZFZCESQ5VHEGBXNHD35/chunks/000016
./01FB0Y3ZFZCESQ5VHEGBXNHD35/chunks/000014
./01FB0Y3ZFZCESQ5VHEGBXNHD35/chunks/000031
<snip>

Well, I don't know where the number 385 comes from, but maybe I can find the one with the erroneous CRC, 1c69f469.

/var/thanos/compact/compact/0@12040338854421827242 $ find . -type f -wholename \*chunks\* -exec crc32 {} \;
d8d1f433 ./01FB0Y3ZFZCESQ5VHEGBXNHD35/chunks/000023
168bc176 ./01FB0Y3ZFZCESQ5VHEGBXNHD35/chunks/000016
aa038baf ./01FB0Y3ZFZCESQ5VHEGBXNHD35/chunks/000014
<snip>

None of the files matched. Now I am at a loss. I guess I could delete the whole block, as has been suggested, but I don't even know how to narrow down to which block contains the corruption. I guess I could delete all 5?

Does anyone know a good way to identify the problematic chunk, or to otherwise allow Compact to get past this error?