Backing up resources that are not managed by Flux

[rombert]

Hi,

I am wondering, what are others doing for backup? The functionality that I am looking for is to back up resources not managed by Flux, mainly persistent volumes. Cert-manager certificates are a nice-to-have.

I think the most used tool is velero, but it looks like the most prevalent use case is a full backup-restore of the full cluster. Since I am using flux, I am more concerned with restoring persistent volume data, either on a case-by-case basis, or all of them altogether.

I have not yet managed to convince velero (with restic) to back up just the persistent volumes, I seem to need to include the pods and then the persistent volumes are discovered...

At any rate, I am curious about what others are doing for backing up non-flux-managed objects in their Kubernetes deployments.

Thanks!

[boedy]

@rombert A couple of months have passed since your post. Have you managed to find a workable solution?

I'm also trying to figure out how to best combine Flux with Velero. Specifically how to restore a cluster running stateful applications in case of disaster recovery.

[rombert]

@boedy - I did not make progress yet unfortunately. I've recently moved all my SQL workloads ( well, a couple of MariaDB instance ) to a MariaDB instance hosted on a VM, which already has a backup process in place. I don't have a good solution for persistent volumes though.

[Nils98Ar]

@rombert Any progress?

[rombert]

Nope, I did not look to much into it. Right now I am manually taking snapshots of the persistent volumes manually, by executing commands directly on the storage server, while the workloads are scaled down.

[juliafmorgado]

@rombert - If you're still exploring options you might find Veeam Kasten a useful alternative (not open source). Kasten is purpose-built for Kubernetes data management, offering seamless backup, restore, and mobility capabilities, including for persistent volumes and stateful applications. Additionally, Kasten integrates with open-source projects like Kanister, which provides a framework for data operations, enabling more advanced workflows such as custom data transformations during backup and restore. This combination could give you more control over selective data protection and recovery in Kubernetes environments, tailored to your needs. It might be worth exploring!

[svengreb]

One popular OSS project (also used in commercial products like OpenShift) is VolSync. It is based on the concept of "movers" that are baked by large and stable projects like restic, rclone (used by restic), syncthing, and good ol' rsync.
It is based on CRDs where a backup is created with a ReplicationSource resource which can be restored with into a new volume with ReplicationDestination resource. The backup target is simply the name of the volume where, based on the used mover, some special configurations are possible, e.g. selecting only specific paths (sadly not yet possible with restic but other movers like rclone).

I used VolSync for all of my private clusters and the plan is to introduce it to clusters at my job too. Flux is used for all clusters already and the combo of both works like a charm 😄

Note that only the restic mover is a pure backup solution while all other movers can be used for backups, but are mainly used to continuously synchronize data between volumes, within the same cluster or across clusters.

[svengreb]

Another project is k8up which I also consider to test and maybe it even fits better for backup-only use cases while the basic way how the CRDs works are similar, but k8up comes with some additional features like "Pre-Backup Pods" that allow to run custom logic before a backups starts like creating database dump to ensure data consistency.