Moco Backups of big dbs

[pavel1337]

Hi!

I was checking the code in hope to understand how the backups are done and encountered this bit:
backup/backup.go 282-291

func (bm *BackupManager) backupFull(ctx context.Context, op bkop.Operator) error {
	dumpDir := filepath.Join(bm.workDir, "dump")
	if err := os.MkdirAll(dumpDir, 0755); err != nil {
		return fmt.Errorf("failed to make dump directory: %w", err)
	}
	defer os.RemoveAll(dumpDir)

	if err := op.DumpFull(ctx, dumpDir); err != nil {
		return fmt.Errorf("failed to take a full dump: %w", err)
	}

Am I understanding this right: moco-backup is going to dump DBs to a folder and then pipe-compress it into S3? I am having a database with around 600GB, does it mean I need to have ~600GB free space for the moco-backup container? Can I set up a storage-class for it?

Thanks

[ymmt2005]

moco-backup is going to dump DBs to a folder and then pipe-compress it into S3?

correct, but compression takes place beforehand.

does it mean I need to have ~600GB free space for the moco-backup container?

not correct. moco uses mysqlsh for backup that compresses data with zstd on the fly.
The result will be much smaller.
The exact directory usage can be checked through MySQLCluster's status.backup.workDirUsage field after the backup.

Can I set up a storage-class for it?

Yes. We are taking backup of database of ~ 2 TB using TopoLVM for generic ephemeral volume.
Specify the volume spec in BackupPolicy at spec.jobConfig.workVolume.
https://github.com/cybozu-go/moco/blob/main/docs/usage.md#backuppolicy

[pavel1337]

mysqlsh for backup that compresses data with zstd on the fly

Is mysqlsh doing it by default?

[pavel1337]

mysqlsh for backup that compresses data with zstd on the fly

Is mysqlsh doing it by default?

Yeah, found the confirmation here: https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-utilities-dump-instance-schema.html

Thanks!

[pavel1337]

Hi, if it's not secret, how long does it take for you to restore ~2 TB backup?
Have you applied any tweaks to make it faster?

[ymmt2005]

There is only one parameter to make it faster, that is, spec.jobConfig.threads.
https://cybozu-go.github.io/moco/crd_backuppolicy_v1beta2.html#jobconfig

@shunki-fujita do you have any number in our environment?

Hi, if it's not secret, how long does it take for you to restore ~2 TB backup?