Release 2.0.0b10 - very beta, for testing only

[ThomasWaldmann]

borgbackup 2.0.0b10 beta release - do not use this on production backup repositories.

Long changelog:
https://borgbackup.readthedocs.io/en/2.0.0b10/changes.html#version-2-0-0b10-2024-09-09

If you use pip to install this, use: pip install "borgbackup==2.0.0b10"

Docs: https://borgbackup.readthedocs.io/en/2.0.0b10/

Discussion:

Note: No OpenIndiana and *BSD platform testing, no FreeBSD binary this time, see issue tracker.

---

This discussion was created from the release Release 2.0.0b10 - very beta, for testing only.

[ThomasWaldmann]

Ideas for testing:

• remote sftp: repos (with some latency), speed?
• file:, ssh:, socket: repos with a somewhat bigger source data set, speed?
• transferring 1.2.x / 1.4.x repos to borg2
• set it up as addtl. backup in parallel to your prod backup
• backup multiple machines to same repo at the same time (see changelog), try to beat the repo as hard as you can.
• kill operations in the middle, check what happens
• can you corrupt a repo somehow? orphan chunks don't count, these are expected and actually a feature. ;-)
• note: stale remote locks should go away in 30mins, stale local locks should get removed immediately.

[knutov]

repository stores objects separately now, not using segment files. this has more fs overhead, but needs much less I/O because no segment files compaction is required anymore. also, no repository index is needed anymore because we can directly find the objects by their ID.

does that means there will be as much files in repo as in source? There are cases with literally millions small files (less than 1kb)...

[ThomasWaldmann]

If you would use a rather access- and space-inefficient mail storage at the src, why would you object doing the same thing at the dst?

Processing "billions of small files" will take ages just for accessing and reading them (and you need to do that once in ANY case, no matter how the repo is implemented). Even stat-ing all these files takes a long time and happens for each backup.

Many fs offer 4kB block size and also many block devices have a hw block size of 4kB.

Emails are larger nowadays than they used to be in the last millennium.

2014: "The [average email is 75 kilobytes (except spam, which is about 5k)."

Storage space is cheap as it never was: we have 30TB on a single disk now.

So, optimizing for the least space overhead of very small files is maybe not the most important thing anymore IF that also would imply wasting a lot of time trying to manage that space better than the fs does.

E.g. if you store your maildir with borg1 and 5% of these mails get deleted, the only way to free that space is to copy 500MiB segment files to free 25MiB per segment file. Worst case is that you need to copy your whole repository just to free a little space and your repo will be locked all the time. borg2 b10 would just directly delete the chunks not needed anymore and everything else stays in place.

Also, the log-like, transaction-based segment files of borg1 made it impossible to do stuff in parallel (like multiple clients backing up to same repo at the same time). With borg2, this now "just works" and we get inter-client dedup.

And I don't think you can compare this to zfs snapshots / zfs send or rsnapshot.

[knutov]

last 10 emails from my inbox

does not look like average 75k

anyway, being compressed many emails will took just small amount of 4k sectors/records, one-by-one each will took at least the same 4kb(+inode)

another case I'm very familiar is shared hosting with millions of small php files (most of them are smaller than 512 bytes).

I understand the cases where new behavior is good - backup of something like media files. But in all other cases old behavior should be better I think.

Is it planned to have some configurable option for this, may be file size limit with option to combine smaller files to something like old segments?

[ThomasWaldmann]

The old repo format and API is like a roadblock to many long-planned features (less locking, parallelism / multithreading, cloud storage, ...), so the plan is to get rid of it.

There are 2 ideas though that maybe could optimize the new repo storage for some cases:

• client-side assembling of "packs" of multiple chunks - that would bring back the compaction issue though (which is even worse for cloud btw. as it needs copy all compaction data from the client). also it would bring back some sort of repo index eating a lot of memory.
• creating a stream of src file data (of multiple / many src files) and chunking that stream instead of individual files. that also has issues, though.

[knutov]

E.g. if you store your maildir with borg1 and 5% of these mails get deleted, the only way to free that space is to copy 500MiB segment files to free 25MiB per segment file. Worst case is that you need to copy your whole repository just to free a little space and your repo will be locked all the time. borg2 b10 would just directly delete the chunks not needed anymore and everything else stays in place.

This does not look like a problem to me, may be except of time being locked. Actually it's more about how segments are technically organized. May be there is some room here to improve.

Also, the log-like, transaction-based segment files of borg1 made it impossible to do stuff in parallel (like multiple clients backing up to same repo at the same time). With borg2, this now "just works" and we get inter-client dedup.

I have some experience with kopia on windows laptop - it uses chunks AND it allows to use repo for multiple clients at the same time.

"inter-client dedup" is a killer feature of course, but I hope there is another way to do it.

Except file size used:

• If backup server is on zfs there will be the problem in case of deleting a lot of files.
• If backup server is on ext4 there will be the problem during creating files (huge problem in case with rsnapshot for example)

[Deric-W]

Maybe do a "pack-and-split" approach like done by Duplicacy where all files are conceptually packed into a big tar archive which is chunked?
This would allow many small files to be packed into a single chunk but requires some reordering (like sorting the files by name to prevent them from having a different order every time we do a backup on some file systems) to make deduplication work.

(I just saw that you had the same idea)

[bjo81]

Transferring 1.4 repos to borg2:

• repo-create also needs --from-borg1
• --from-borg1 and --upgrader=From12To20 looks like some option duplication

Testing with borg2 with borg2 on a remote server

querying the chunk IDs list from the repo...
queried 455185 chunk IDs in 427.14834810979664 s (6 requests), ~36.23 kB/s

So querying the chunk IDs takes some time while borg2 on the remote server creates some high IO.

[ThomasWaldmann]

1. correct
2. --from-borg1 implies --upgrader=From12To20, so one can leave the latter away. the first was needed because borg internally needs to use the legacy repository code for that case.

The high activity you saw on the server is borgstore listing all files in the repository to generate the list of chunk IDs from the filenames.
As all chunks are stored separately, listing them all is a rather expensive operation and there is no optimization for that yet. borg does that to ad-hoc build the chunks index in memory.