Why does Borg not automatically exclude /proc, /sys etc.?

[pjoc]

I've spent a week trying to diagnose a problem with my nightly backup script, which uses Borgmatic. This has been working for over two years without problems, and with no recent changes, yet it was suddenly failing to terminate, i.e. I would look at the logs in the morning and find that Borg was still running, though its normal running time should be in the order of 5-10 minutes. On closer examination, and using the maximum logging verbosity, I found that Borg was constantly examining files under /proc and /sys, even though my Borgmatic configuration doesn't list / as a filesystem to be backed up.

I decided to explicitly exclude /proc, /sys, /tmp and /dev in the --patterns-from file, and now the script works.

Is this a recent change to Borg that I should have been aware of? I'm using Borg version 1.2.2 on Fedora 36.

Do people think it's reasonable for Borg to look at those system directories unless told otherwise? I'm finding it hard to come up with a use case for that which would make sense.

[ThomasWaldmann]

Borg tries to avoid including behaviour addressing:

• operating system dependent stuff (like what you mentioned)
• filesystem, storage device, ... dependent stuff
• stuff that is somehow a matter of taste / opinion
• "magic" behaviour in general. because it usually needs a lot of explanation and sometimes even introducing the corresponding anti-magic options in case the magic fails. usually it is easier not to include the magic stuff in the first place.

[pjoc]

I understand the motivation, however I still think that this behaviour for a Linux implementation is simply bizarre. I can't think of any situation in which it would make sense. Presumably there are similar scenarios on other platforms. It took me a long time to figure out what was going on, and I still have no idea what changed between my script working and it not working, as none of my config files were altered.

I appreciate the effort that goes into Borg and in general it has worked very well for me, but it can be quite difficult to diagnose issues when they occur. At least, please consider including a warning in the documentation. The docs do include specific warnings about (e.g.) BTRFS and issues related to mounted filesystems, so this shouldn't be a problem.

[ThomasWaldmann]

Well, borg usually does what you tell it to do. If you tell it to do unreasonable things (like backing up /proc), the outcome might be bad. The simplest solution is not to tell it to do unreasonable things.

When backing up the root filesystem, you'll need some excludes. You can also use the --one-filesystem but then you have to be careful to give the roots of all valuable mounted file systems.

Alternatively, you can also specifically give some non-root directories to borg, like /etc /home /root /var.

[pjoc]

My patterns file is modelled on the one in the docs, which starts by specifying "R /" and then excludes a bunch of stuff. I presume that's why it was trying to back up /proc etc. That much I understand.

What I don't understand is why this wasn't failing before. I've gone back and verified that my patterns file had exactly the same content when it was working as when it wasn't working, so clearly something else changed.

It appears to be caused by an update to Borgmatic, which has a Changelog mentioning a workaround for #6994. However I'd need to delve deeper into this to be sure.

I still think that some mention of this in the docs would not hurt.

[ThomasWaldmann]

There was a small change from borg 1.1 to 1.2 for the patterns, see our changelog.

[pjoc]

Thanks. Because of the way the changelog is packaged in the Fedora version, I hadn't noticed the comments.

Once again, I think it would be helpful to have some platform-specific caveats in some obvious place in the docs. In fact there already are, as I already mentioned, just not this particular one.