Trying to understand how include & R patterns work

[cyb3rz3us]

Elsewhere in the Q&A, @ThomasWaldmann mentioned that an "R" pattern is essentially like providing the path at the CLI. However, in my testing, it doesn't seem to do this at all. Here is a very simple test I crafted to show this:

TO START I have the ~/borgtest3 directory which contains 3 subDirs (3a, 3b, 3c) each with some content. The goal is to to backup only the files in the 3a directory and without borg scanning the entire FS below borgtest3.

First, I tried without any patterns or a pattern file and this worked fine. Here is what I did:

borg create --list --show-rc --debug --dry-run ::'TEST-ONLY_{now:%Y%m%d-%H%M%S}' ~/borgtest/borgtest3a

OR

borg create --list --show-rc --debug --dry-run ::'TEST-ONLY_{now:%Y%m%d-%H%M%S}' /Volumes/borgLab/cyb3rzeus/borgtest/borgtest3a

RESULT: Only the files in borgtest3a were backed-up and no other files were scanned.

However, when trying to achieve he same result using a pattern-file, the R pattern does not work as if I had merely given the path at the CLI as shown above. No matter what I tried, the only way to get the same result was to always specify the full path of the directory containing the files to be backed up.

Here are the many pattern iterations I tried along with the result:

iteration-1

R /Volumes/borgLab/cyb3rzeus/borgtest3
+ borgtest3a

RESULT: Scans & backs up the entire FS below borgtest3 directory

iteration-2

R /Volumes/borgLab/cyb3rzeus/borgtest3
+ borgtest3a
- *

RESULT: Scans all files but excludes everything

iteration-3

R /Volumes/borgLab/cyb3rzeus/borgtest3
+ borgtest3a
! /Volumes/borgLab/cyb3rzeus/borgtest3

RESULT: Scans nothing and excludes the directory in the R pattern

iteration-4

+ /Volumes/borgLab/cyb3rzeus/borgtest3/borgtest3a

RESULT: Error saying I need at least one path argument

iteration-5

R /Volumes/borgLab/cyb3rzeus/borgtest3/borgtest3a

RESULT: Works exactly the same as the no-pattern test shown above

iteration-6

R /Volumes/borgLab/cyb3rzeus/borgtest3/borgtest3a
+ /Volumes/borgLab/cyb3rzeus/borgtest3/borgtest3a

RESULT: Works exactly the same as the no-pattern test shown above

So it seems all that is really needed is to specify the entire path as an R pattern which then leads to not understanding how an R pattern works with an include pattern when they are the same thing.

Clearly I'm missing something here so some clarification would be greatly appreciated.

Thanks.

[ThomasWaldmann]

Elsewhere in the Q&A, @ThomasWaldmann <https://github.com/
ThomasWaldmann> mentioned that an "R" pattern is essentially like
performing a change dir at the CLI.

Ehrm, did I? Do you have an URL?

If so, I am sorry, because that would be wrong. It is rather like giving these paths on the commandline to borg to use as root paths (starting paths) for the fs recursion, but it does not do a "change dir".

The goal is to to backup only the files in the 3a directory and> without borg scanning the entire FS below borgtest3.

Then just give the 3a path as root path, that is the simplest way.

Giving the full path is best, because it will work no matter from where you invoke the backup.

R /Volumes/borgLab/cyb3rzeus/borgtest3 +borgtest3a
RESULT: Scans & backs up the entire FS below borgtest3 directory

R /Volumes/borgLab/cyb3rzeus/borgtest3 +borgtest3a -*
RESULT: Scans all files but excludes everything

R means to start the recursion there (which will discover all the file paths below there by default).

"+" will try to match the given pattern against the current path from the recursion and add the file to the archive if the path matches the pattern.

"-" will also try to match, but won't add the file if the path matches the pattern.

"!" is a special thing: it behaves similar to "-", but does not recurse any further if the pattern matches. On Linux, one usually uses this to avoid pointless recursion into /proc or /sys.

The first match counts, so order of the patterns is important.

Please note that there is no relation between the root paths and the patterns:

• you can't specify the patterns "relative to the roots / omitting the path prefix given by the root". So it is best to pattern-match the full path as generated by the recursion.
• the patterns do NOT "belong to" the roots

This is because borg lets you specify paths and patterns on the cli as well as in the pattern file and it collects paths and patterns in 2 separate lists.

As is seen in these last 2 tests, all that is really needed is to
specify the entire path as an R pattern which then leads to me to having
no idea how or why an R pattern with an include pattern is used\needed.

If you give a root path to borg, the default behaviour is to back up all files in there (which is somehow the natural expectation from a backup program).

The primary use of "+" is to include some files that would be otherwise excluded by a following (usually more general) "-" pattern.

The borg docs and borg help patterns tries to explain this all. In case you read it and something was missing or misunderstandable, file an issue or a PR with an improved text.

[cyb3rz3us]

Hi @ThomasWaldmann --- first a correction on my part; I misspoke when I said "...performing a change dir...". I really intended to say "...providing the path..." and have corrected my initial post accordingly.

Next, thank you for this incredibly helpful answer. I now see where my misunderstanding crept in; I thought I could use a combination of an R pattern & include\exclude rules to specify which items to scan and backup but I now see this isn't possible. Because part of my goal is to prevent scanning items I know won't be backed up, I just need to actually specify the entire path as the R pattern.

Thank you again --- very much appreciated.