Backed up files are not "registered" and keep being displayed again and again during backups

[wsw70]

I've been using Borg for years and have a mounted external disk on which I backed up my system. Worked great, restored great.

I changed the OS of the server (from Arch to Debian) and wanted to keep on backing up on the same repository (Borg was 1.4.0 on both systems). To do this, I mounted the disk again after switching the OSes, and backed up as usual via

borg \
        create \
        --list \
        --exclude '**/__pycache__' \
        --exclude '**/*.pyc' \
        --exclude '**/.*.swp' \
        --exclude '**/.*.swo' \
        --exclude '**/node_modules' \
        --exclude-caches \
        --exclude-if-present .nobackup \
        '/backup/borg/srv::{now}' \
        /etc \
        /data/externalFilesToBackup \
        /var/lib/docker/volumes

The backup went on, with lots and lots of files scrolling. OK I thought, there may have been a change of the timestamp, it will back up again. Not a problem thanks to de-duplication.

And then right after a backup was finished, I tried to back up again and the same long list of files scrolled. And then tried again, and it scrolled again.

It is too fast to be a complete backup, though the scrolling stops at some points to back up a slightly larger file. So it does something.

An example of the list being outputted again and again (this is the tail of the output). None of these files changed between backups.

d /var/lib/docker/volumes/unifi_unifi
U /var/lib/docker/volumes/03b7077d005aa1c4f1b74d2aa164333e23d1e093494ed7baf2137886fda5a93c/_data/dump.rdb
d /var/lib/docker/volumes/03b7077d005aa1c4f1b74d2aa164333e23d1e093494ed7baf2137886fda5a93c/_data
d /var/lib/docker/volumes/03b7077d005aa1c4f1b74d2aa164333e23d1e093494ed7baf2137886fda5a93c
U /var/lib/docker/volumes/proxy_authelia/_data/configuration.yml
U /var/lib/docker/volumes/proxy_authelia/_data/db.sqlite3
U /var/lib/docker/volumes/proxy_authelia/_data/new config.yml
U /var/lib/docker/volumes/proxy_authelia/_data/notification.txt
U /var/lib/docker/volumes/proxy_authelia/_data/users_database.yml
d /var/lib/docker/volumes/proxy_authelia/_data
d /var/lib/docker/volumes/proxy_authelia
U /var/lib/docker/volumes/f06157de8c7933b347add458b5e2c581384733cda67e6f9a7d97eea787daea13/_data/dump.rdb
d /var/lib/docker/volumes/f06157de8c7933b347add458b5e2c581384733cda67e6f9a7d97eea787daea13/_data
d /var/lib/docker/volumes/f06157de8c7933b347add458b5e2c581384733cda67e6f9a7d97eea787daea13
U /var/lib/docker/volumes/9db0e58922a09f79aebf8c40da873b1e4a213ff60a2b6053ae900da32b3260eb/_data/database.sqlite
d /var/lib/docker/volumes/9db0e58922a09f79aebf8c40da873b1e4a213ff60a2b6053ae900da32b3260eb/_data
d /var/lib/docker/volumes/9db0e58922a09f79aebf8c40da873b1e4a213ff60a2b6053ae900da32b3260eb
d /var/lib/docker/volumes

The backups seem to be there:

root@srv ~# borg list /backup/borg/srv
2024-06-30T23:00:28+02:00            Sun, 2024-06-30 23:00:34 [d5007aa4965e73d22d27d91a8a1111f4552e308efff860bfaca536d61153e3ff]
2024-07-07T23:00:19+02:00            Sun, 2024-07-07 23:00:26 [7e272fb201c78d4886af785c2a877e15c087addd744afc24ec3650466afc3c30]
2024-07-14T23:00:06+02:00            Sun, 2024-07-14 23:00:12 [2d4353a25423e80e7172ef3c7dcde4632aad43479684043ffa69b71da051d958]
2024-07-21T23:00:19+02:00            Sun, 2024-07-21 23:00:25 [9ed6b570f23cbda30b99f32a63a495c531cbb06f0a0e763d7742718f0f4c4646]
2024-07-28T23:00:22+02:00            Sun, 2024-07-28 23:00:28 [0d52d03d3fe593df0e836d5320ca083e90e03035ca4b19b86a19ba107861d4ee]
2024-07-31T23:00:22+02:00            Wed, 2024-07-31 23:00:28 [35fd6cf748a11f11cb3227805b4e3ee5fe455ca2b3839ca32d9b46af33120df9]
2024-08-01T23:00:20+02:00            Thu, 2024-08-01 23:00:27 [03a858748940dff19e0c47531e67e65ec8f305167143b6e9fb1248ffd4e78e58]
2024-08-02T23:00:22+02:00            Fri, 2024-08-02 23:00:28 [cb468a8da4733cd47791404aec8f2544597959de8a04eb7720c1c72168aee35a]
2024-08-03T23:00:18+02:00            Sat, 2024-08-03 23:00:24 [eddb99dccb6767cb97b7f65271260b88e236828207ed08850b23f6eb38766d08]
2024-08-04T23:00:16+02:00            Sun, 2024-08-04 23:00:22 [f883cb3e057a5fe30a1dd27ff6266b6546fcf67a9a43cefe3664cf6285ab25c6]
2024-08-05T23:00:26+02:00            Mon, 2024-08-05 23:00:32 [cadf16527a5d222bf22020faf089ac735993135a6b6dee0e00a23ffa92e593a1]
2024-08-06T23:00:04+02:00            Tue, 2024-08-06 23:00:11 [1aa4af60631096852a19398d59ea6e864d3f524bf5de3f70ac8ac34af5169ab6]
2024-08-07T11:00:01+02:00            Wed, 2024-08-07 11:00:02 [8e0653408ec9b439eff98a1bc988ac0c6b2d389bdbea6679eb59a2fcf7e5896a]
2024-08-08T17:43:28                  Thu, 2024-08-08 17:43:28 [6fed40defbaba6af5b8649f5e0fdfa20cc2b8eebe8bb7175c61a1de0761bbec8]
2024-08-09T09:26:39                  Fri, 2024-08-09 09:26:39 [0efb77f1be43c2b24e2c70f2866bb5ebe24af9b98f3c9d80a1d268391012ed0c]
2024-08-09T10:19:13                  Fri, 2024-08-09 10:19:13 [1be7667c594c52c34508f3d489a0dca33aae1ecf9e451184b2a2d588e5b72705]
2024-08-09T10:23:31                  Fri, 2024-08-09 10:23:31 [4c4c134b4a2271c35f4390305e16237ff81f1584cd4555f48a18e85ff63d5c62]
2024-08-09T10:24:27                  Fri, 2024-08-09 10:24:27 [3f0ce003b111e9f1d1b74bf810cfb8c09fe5ffcd09f9adafe7ebdd2b05edea05]
2024-08-09T10:28:22                  Fri, 2024-08-09 10:28:23 [3b1cd75297c2d3cffaa0f803b340908e6527c838206f32f15c8ea3d035cbc73a]

General info about the repo

root@srv ~ [2]# borg info /backup/borg/srv
Repository ID: c6853ce826e2c5c62c58df2ac0eabca4cf087980944b57995703bd44cceb981e
Location: /backup/borg/srv
Encrypted: No
Cache: /root/.cache/borg/c6853ce826e2c5c62c58df2ac0eabca4cf087980944b57995703bd44cceb981e
Security dir: /root/.config/borg/security/c6853ce826e2c5c62c58df2ac0eabca4cf087980944b57995703bd44cceb981e
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:              814.13 GB            671.88 GB             41.50 GB

                       Unique chunks         Total chunks
Chunk index:                  197166              5123088

Looking at the last two backups (4 minutes apart):

root@srv ~# borg info /backup/borg/srv::2024-08-09T10:28:22
Archive name: 2024-08-09T10:28:22
Archive fingerprint: 3b1cd75297c2d3cffaa0f803b340908e6527c838206f32f15c8ea3d035cbc73a
Comment:
Hostname: srv
Username: root
Time (start): Fri, 2024-08-09 10:28:23
Time (end): Fri, 2024-08-09 10:29:04
Duration: 41.49 seconds
Number of files: 153551
Command line: borg create --list --exclude '**/__pycache__' --exclude '**/*.pyc' --exclude '**/.*.swp' --exclude '**/.*.swo' --exclude '**/node_modules' --exclude-caches --exclude-if-present .nobackup '/backup/borg/srv::{now}' /etc /data/externalFilesToBackup /var/lib/docker/volumes
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               30.10 GB             24.68 GB             74.16 MB
All archives:              814.13 GB            671.88 GB             41.50 GB

                       Unique chunks         Total chunks
Chunk index:                  197166              5123088
root@srv ~# borg info /backup/borg/srv::2024-08-09T10:24:27
Archive name: 2024-08-09T10:24:27
Archive fingerprint: 3f0ce003b111e9f1d1b74bf810cfb8c09fe5ffcd09f9adafe7ebdd2b05edea05
Comment:
Hostname: srv
Username: root
Time (start): Fri, 2024-08-09 10:24:27
Time (end): Fri, 2024-08-09 10:25:11
Duration: 43.81 seconds
Number of files: 153547
Command line: borg create --list --exclude '**/__pycache__' --exclude '**/*.pyc' --exclude '**/.*.swp' --exclude '**/.*.swo' --exclude '**/node_modules' --exclude-caches --exclude-if-present .nobackup '/backup/borg/srv::{now}' /etc /data/externalFilesToBackup /var/lib/docker/volumes
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               30.10 GB             24.68 GB             15.99 MB
All archives:              814.13 GB            671.88 GB             41.50 GB

                       Unique chunks         Total chunks
Chunk index:                  197166              5123088

A borg diff between the last two backups shows just a few volatile files - as expected.

[wsw70]

OK, I misunderstood what U means. I thought it was "Updated" but it means "Unchanged". Everything is fine then.