How to migrate from invalid chunker parameters?

[ethus3h]

Hi, when I use the latest version of Borg packaged in Guix (updating from 1.1.11 to 1.2.3), I get invalid ChunkerParams value because the script I was using used parameters 10,14,16,4095 that do not pass the validation (parseformat.py#L164).

For now I've rolled back to the old version that allowed those parameters, but I'm wondering what parameters should I put to allow deduplication to continue working against the chunks made with those invalid values?

(I'd like to avoid having to recompress the whole repository since I've already stored several terabytes/many millions of files so re-compressing would be really inconvenient, or I'd just do that.)

Thank you!

[ThomasWaldmann]

The values are chunk_min,chunk_max,chunk_mask,window_size and min <= mask <= max should be true.

The idea is that the target chunk size is about 2 ^ mask, but due to the way the buzhash chunker works, that is only a statistical target and there is quite some random variation in the actually generated chunk sizes. Also, we usually do not want very small or very large chunks. Because of that, one also needs to give min and max and it will always generate chunks between 2 ^ min and 2 ^ max.

So, in your case, the max value is lower (14) than expected (>= 16).

https://github.com/borgbackup/borg/blob/1.2.4/src/borg/_chunker.c#L122 looks like the max chunk size is not directly enforced, but rather indirectly over using a buffer of that size.

I just tried how the code behaves:

Here are some results for 10,16,14 (cutting 10MB random into chunks) - that is how it should look like:

count: 570 min: 1036 max: 65536 min count: 0 max count: 16
count: 560 min: 1041 max: 65536 min count: 0 max count: 18
count: 580 min: 1056 max: 65536 min count: 0 max count: 15
count: 566 min: 1058 max: 65536 min count: 0 max count: 11

The chunks were always between min and max allowed (as expected) and only rarely were cut at max size. So 97% of chunks were cut as determined by buzhash value and only 3% were cut due to exceeding the max size.

Here are some results for 10,14,16 (cutting 10MB random into chunks) - for your chunker params:

count: 683 min: 1077 max: 16384 min count: 0 max count: 568
count: 671 min: 1071 max: 16384 min count: 0 max count: 574
count: 675 min: 1031 max: 16384 min count: 0 max count: 569
count: 680 min: 1033 max: 16384 min count: 0 max count: 563

So, as you see here, the cut chunks were very frequently at the maximum allowed size (due to max chunk size limiting them) and only a few chunks (about 20%) were at some size as determined by the buzhash value.

[ThomasWaldmann]

It's a bit hard to give good advice here, because likely there is no quick and good way to resolve this.

I maybe could convert the validation failure you see in borg 1.2.3 (and also 1.2.4) into a warning, so you can continue with the bad chunker params for now without having to use an old borg version.

You could borg recreate all your archives with valid chunker params - this is likely going to take a long time, depending on your archives count. But this would have the advantage that then the buzhash chunking actually would work like expected. What you have now is somehow a buzhash chunker 80% degenerated into a fixed chunker. You could speed it up by first reducing the archive count to the minimum you need.

You could also just switch to valid chunker params and do backups and pruning for some months (will need more space in the repo due to bad dedup) and then only recreate the fewer leftover archives that still have the old chunker params.

Side note: borg2 transfer does not help with this as re-chunking is not planned there. You only could choose to start from 0 with borg2 and maybe run 1.2.x and 2 for a while in parallel in different repos.

[ThomasWaldmann]

Ticket: #7590, "fix" will be in borg 1.2.5.

[ethus3h]

Thank you so much for all the details and information and the change you linked! This was very informative and helps me understand the issue and the ways I can move forward. I really appreciate it!