How am I supposed to do application-aware-backups and restores?

[ccremer]

We have a AKS cluster running with Velero.
My use case is to backup an application running inside a single namespace. Disaster recovery for the whole cluster is not the goal.
However, we want to be able to restore a single application (e.g. a user accidentally deleted a file in PVC and needs a restore).

Our idea is to use Velero for application-aware backups. Our app consists of a Python app that creates some PDFs in a PVC with Azure-backed CSI disks, and a Database running on its own CSI disk.

Since filesystem snapshots are insufficient for reliable database backups, we create a SQL dump file before backing up a volume, and store it in the same volume as the PDFs, so we have everything in one place.

Therefore, we have the following requirements:

• Create a dump before backing up
• Backup the PDF volume (PVC)
• Scope the backup and restore to a single namespace
• Exclude cluster-scoped resources
• Exclude disk from DB (since we have a dump this would only incur unnecessary costs) resp. only include the PDF volume.

As long as we have a dump in the PDF volume, we don't care if the volume is backed with FSB or CSI, whatever works better and is only billed once.
It seems only FSB supports hooks though, so we'll use FSB.

What I have right now:

• Pod with hooks annotated:

      annotations:
        backup.velero.io/backup-volumes: api-media-volume
        pre.hook.backup.velero.io/command: >-
          ["/bin/bash", "-c", "cd api && python manage.py dumpsql -o
          /opt/api/var/media/dump.sql"]
        pre.hook.backup.velero.io/container-name: api
        pre.hook.backup.velero.io/timeout: 5m

• PVC api-media-volume with storage class managed-csi
• Storage class managed-csi with WaitForFirstConsumer binding mode and Delete reclaim policy
• Backup created with velero backup create test-backup-5 --include-namespace default --exclude-cluster-scoped-resources "*"

The backup seems to work:

$ velero backup describe test-backup-5 --details
Name:         test-backup-5
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.27.7
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=27

Phase:  Completed


Namespaces:
  Included:  default
  Excluded:  <none>

Resources:
  Included cluster-scoped:    <none>
  Excluded cluster-scoped:    *
  Included namespace-scoped:  *
  Excluded namespace-scoped:  <none>

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-02-12 09:16:38 +0100 CET
Completed:  2024-02-12 09:16:53 +0100 CET

Expiration:  2024-03-13 09:16:38 +0100 CET

Total items to be backed up:  130
Items backed up:              130

Resource List:
  ...(some resources omitted for brevity)
  apps/v1/Deployment:
    - default/nxt-api
    - default/nxt-mariadb
  apps/v1/ReplicaSet:
    - default/nxt-api-646d495b4
  v1/PersistentVolumeClaim:
    - default/nxt-api-media
    - default/nxt-mariadb-storage
  v1/Pod:
    - default/nxt-api-646d495b4-7dd94

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included>

  Pod Volume Backups - kopia:
    Completed:
      default/nxt-api-646d495b4-7dd94: api-media-volume

HooksAttempted:  1
HooksFailed:     0

I'm having trouble with the restore.

1. k delete pvc nxt-api-media (I'm deleting the PVC because if not deleted, the restore would skip it)
2. velero restore create --from-backup test-backup-5

$ velero restore describe test-backup-5-20240212103002
Name:         test-backup-5-20240212103002
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                                 InProgress
Estimated total items to be restored:  58
Items restored so far:                 34

Started:    2024-02-12 10:30:02 +0100 CET
Completed:  <n/a>

Backup:  test-backup-5

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Or label selector:  <none>

Restore PVs:  auto

Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s

Preserve Service NodePorts:  auto

It's stuck. Specifically, it's stuck in PVC Pending. And it is in Pending since it tries to bind the PVC to a volume that doesn't exist anymore. The PVC has the volume name volumeName: pvc-6e5ad802-f141-46a0-a33d-4061e4cacea2, but I can't clear this field since a PVC spec is mostly immutable.

How can I restore the filesystem in the volume to the last backup so that the next pod restart sees the recovered files?
What is the correct way to achieve this scenario?

Edit:
I've managed to un-stuck the PVC using this resource modifier policy:

apiVersion: v1
kind: ConfigMap
metadata:
  name: restore-pvc-patch
  namespace: velero
data:
  resource-patch.yaml: |
    version: v1
    resourceModifierRules:
    - conditions:
         groupResource: persistentvolumeclaims
      patches:
      - operation: remove
        path: "/spec/volumeName"

and started restore using velero restore create --from-backup test-backup-5 --resource-modifier-configmap restore-pvc-patch.
This worked, however, now the PodVolulmeRestore is stuck. In its spec, there's the pod field, which points to a pod that doesn't exist anymore. The logs also don't reveal why the pod volume restore is stuck...

Why is it so hard to do a simple file restore? This doesn't seem to be a niche use case.

[ccremer]

I have found a solution.

I went back to CSI Snapshots. However, this time I'm using the resource filter approach.
I created another Storage Class that the DB uses. This storage class is excluded by this policy:

apiVersion: v1
kind: ConfigMap
metadata:
  name: volume-policy
  namespace: velero
data:
  resource-policy.yaml: |
    version: v1
    volumePolicies:
    - conditions:
        storageClass:
          - managed-csi-no-backup
      action:
        type: skip

Further, the Pod annotation backup.velero.io/backup-volumes is removed, otherwise it's still using FSB.
The restore policy to remove the volume name is also not required.
The hook to create a SQL dump is still working.

The restore is a bit more complicated:

1. First, scale down the deployment to remove running pods
2. Delete the PVC (which releases the PV as well). Without this step, the restore skips the PVC.
3. Issue velero restore create --from-backup ...
4. Scale up again.