Borg check question: for SSHFS repo, how much data is transferred for each option?

[AndrewDDavis]

I'm backing up to an older NAS (MyCloud EX4) with glibc that is too old for the borg binaries, so I'm using SSHFS. Also there is a 100 Mbit router in the network, so data transfers are limited to about 10.5 MB/s. The repo size is ~ 1 TB, so transfer of all the data should take ~ 28 hours.

Recently I got integrity errors when backing up, so I'm running borg check commands. I was surprised that borg check --repository-only took more than a day to complete, which likely means all of the repo data was transferred over the network. I was expecting that --repository-only would need somewhat less than the full data to do its checks. My question is: how much data will borg transfer over SSHFS to perform its checks, with --repository-only, --archives-only, or --verify-data?

[ThomasWaldmann]

The repository part of the check will read all chunks in the repo (all contents of the segment files) to verify the crc32 checksums of the stored chunks and also to build a known-good fresh repo index.

The archives part will then iterate over all archives and do misc. checks, e.g. whether the referenced chunks are present in the repo index - also it will build a new known-good chunks index (that has refcount and size infos for each chunk).

--verify-data will also do a cryptographic check, whether the chunk contents are authentic. this will read a lot of data.

More details are in the borg check docs.

About integrity errors:

• if the repo was ever used with an unpatched borg < 1.1.11 make sure to read and follow the advisory at top of the borg changelog. borg 1.1.9 in debian/ubuntu has been patched with the fix for that issue.

• check your hardware, hw issues are a common cause for integrity errors. RAM, disk, ...

[ThomasWaldmann]

Time for archive check grows linearly with the amount of archives (assuming they are all of similar content, amount of files, volume of data chunks). But if you do not use --verify-data it is mostly a metadata check.

There is no local caching, so the repo check already having read all the repo data does not help the archives check, it will just read and check again (in a different way). Maybe also showing linear growth with archive count.

If there is a chunk missing from your most recent archive, you have a good chance that your source data still reproduces this chunk if you do a backup. So do a check --repair, another backup, and another check --repair - maybe it will heal the missing chunk then.

Interrupted backups are unlikely to corrupt the repo due to its log-like structure (always append) and borg's transactional behaviour. Either stuff is completed / valid and committed. Or it is not (e.g. due to interruption) and will be just rolled back to the last valid (committed) state next time you use the repo.

[AndrewDDavis]

Thanks, that's helpful -- I think the part I was missing was that --verify-data would apply to the archive check, as well as the repo check. And I will try your advice on doing another check --repair after the next backup. The missing chunk was on a file that hasn't changed in years, so I'm hopeful it can be healed.

[ThomasWaldmann]

--verify-data does not change how the repository part of the check works. for the archives part of the check, it adds a cryptographic verification of the chunk contents (without that option, it only checks for the chunks presence).

[AndrewDDavis]

Thanks for all your help with this -- since the last, I have done a lot of hard-drive maintenance on both the source and NAS side using smartctl and other tools. Borg helped me realize that there were problems on the drives (on both ends!) that were not becoming apparent until all the data was checked. I'm now getting clean reports from the full repository checks (previous problems were healed), and closely monitoring the health of these drives.

IMO, the man page for borg-check makes it clear that --verify-data applies only to the archives part under the option's description:

--verify-data: perform cryptographic archive data integrity verification (conflicts with --repository-only)

However, this could be made more clear in the main "description" section of the man page, and the online documentation for borg, by making the paragraph about --verify-data a bullet point of the archives check section, and including the part about "conflicts with --repository-only".

[ThomasWaldmann]

@AndrewDDavis agreed, filed there: #5808