git-remote-rclone

This is a small git remote helper, which allows hosting a git repo in cloud storage (s3, etc.), in encrypted format.

Technologies used:

• rclone for encrypted sync to the cloud
• kopia for encrypted backups to the cloud

How it works

This helper introduces a custom rclone:: protocol for git remotes. i.e.

git remote add origin "rclone::${rclone_remote_path}"

where rclone_remote_path refers to any valid rclone remote path.

This helper supports two basic operations:

• git fetch (read from remote)
• git push (write to remote)

(For more details, read the source code!!)

git fetch

graph LR
    cloud["Encrypted bare git repo"]
    local_cache["Decrypted bare git repo"]
    local_real["Hydrated git repo"]

    cloud -->|"rclone (decrypt)"| local_cache --> local_real
    
    subgraph "Cloud Storage"
        cloud
    end
    subgraph "Local Machine"
        subgraph "$XDG_CACHE_HOME"
            local_cache
        end
        subgraph "Local repo"
            local_real
        end
    end

Loading

git push

The git push operation is a bit more complex. FIRST, ensure that the local cache is up-to-date:

graph LR
    cloud["Encrypted bare git repo"]
    local_cache["Decrypted bare git repo"]
    
    cloud -->|"rclone (decrypt)"| local_cache    
    
    subgraph "Cloud Storage"
        cloud
    end
    subgraph "Local Machine"
        subgraph "$XDG_CACHE_HOME"
            local_cache
        end
    end

Loading

Now, we upload to the cloud.

Also, since this is a "write" operation, store a backup of the bare git repo, just in case:

graph RL
    cloud_main["Encrypted bare git repo"]
    cloud_backups["Encrypted backups"]
    local_cache["Decrypted bare git repo"]
    local_real["Hydrated git repo"]

    local_real --> local_cache 
    local_cache -->|"rclone (encrypt)"| cloud_main
    local_cache -->|"kopia (encrypt)"| cloud_backups
    
    subgraph "Cloud Backups"
        cloud_backups
    end
    subgraph "Cloud Storage"
        cloud_main
    end
    subgraph "Local Machine"
        subgraph "$XDG_CACHE_HOME"
            local_cache
        end
        subgraph "Local repo"
            local_real
        end
    end

Loading

Note

The push from the local repo to local cache is important - it helps detect and prevent bad scenarios, i.e. when the push might overwrite a commit in the remote.

Installation

Download the file git-remote-rclone, and make it available in PATH.

Other required dependencies:

• bash
• rclone
• kopia

Setup

Configure rclone on your machine

• Use the rclone docs
• Set up an encrypted remote

FIRST TIME ONLY: Create a bare git repo locally, then sync to rclone

BARE_DIR=/tmp/repo-bare
REPO_DIR=/tmp/repo
mkdir -p $BARE_DIR && cd $BARE_DIR
git init --bare
rm ./hooks/* # hooks not needed

git clone $BARE_DIR $REPO_DIR
cd $REPO_DIR
git commit --allow-empty -m "first commit"
git branch -m master main
git push -u origin main

cd $BARE_DIR
git symbolic-ref HEAD refs/heads/main # change HEAD to the main branch
rclone sync $BARE_DIR rclone-remote:path/to/use

Clone the repo to a new machine

NOTE: git clone does NOT work! Use this instead:

mkdir ./cloned && cd ./cloned
git init
git remote add origin "rclone::rclone-remote:path/to/use"
git fetch
git checkout main

Set up kopia for backups

• If setting up for the first time, use kopia repository create
• Otherwise, use kopia repository connect
• Set up the kopia policy. Here are my default values:

Retention:
  Annual snapshots:                     0   (defined for this target)
  Monthly snapshots:                    0   (defined for this target)
  Weekly snapshots:                     0   (defined for this target)
  Daily snapshots:                     90   (defined for this target)
  Hourly snapshots:                     0   (defined for this target)
  Latest snapshots:                   200   (defined for this target)
  Ignore identical snapshots:        true   (defined for this target)

Usage

Use git fetch and git push as normal.

KNOWN LIMITATIONS:

• git fetch sometimes shows an error, even though the fetch was successful
• The first git pull won't work
  • Instead, you can do git fetch and then git pull (or git pull twice)

Tests

We use the bats test framework.

Install dependencies:

• Install the bats binary to PATH.
• Install the test helper submodules, git submodule update --init --recursive

Run tests:

bats ./test

Alternatives

Here's a list of alternatives I considered. These aren't inherently bad, but none of them fits my personal use case.

Also the source code for these projects seem more complex than my small ~50-line bash script 😛

Project Name	Why not?
spwhitton/git-remote-gcrypt	Requires messing with gpg. Also, seems unmaintained.
Keybase encrypted git	Requires trusting a separate company to handle the encryption. I'd rather roll my own and have a bit more fine-grained control over my data.
CGamesPlay/git-remote-restic	Requires a custom fork of restic.
nathants/git-remote-aws	Stores some metadata unencrypted.
GenerousLabs/git-remote-encrypted	Seems unmaintained. Also only 7 GitHub stars (as of 2024-04-26)