Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TMP: test scylla backup api #4111

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

TMP: test scylla backup api #4111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b1ba828
chore(go.mod): bump SM submodules versions
Michal-Leszczynski Nov 13, 2024
2ee9dca
feat(testing): configure Scylla object storage (minio)
Michal-Leszczynski Nov 13, 2024
4690440
tmp: test backup with scylla api
Michal-Leszczynski Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ require (
github.com/scylladb/go-reflectx v1.0.1
github.com/scylladb/go-set v1.0.2
github.com/scylladb/gocqlx/v2 v2.8.0
github.com/scylladb/scylla-manager/v3/pkg/managerclient v0.0.0-20241104134613-aba35605c28b
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241104134613-aba35605c28b
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241104134613-aba35605c28b
github.com/scylladb/scylla-manager/v3/pkg/managerclient v0.0.0-20241113064359-18103e06f366
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241113064359-18103e06f366
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241113064359-18103e06f366
github.com/spf13/cobra v1.8.0
github.com/spf13/pflag v1.0.5
github.com/stoewer/go-strcase v1.3.0
Expand Down
12 changes: 6 additions & 6 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1053,12 +1053,12 @@ github.com/scylladb/google-api-go-client v0.34.1-patched h1:DW+T0HA+74o6FDr3TFzV
github.com/scylladb/google-api-go-client v0.34.1-patched/go.mod h1:RriRmS2wJXH+2yd9PRTEcR380U9AXmurWwznqVhzsSc=
github.com/scylladb/rclone v1.54.1-0.20240312172628-afe1fd2aa65e h1:lJRphCtu+nKd+mfo8whOTeFkgjMWvk8iCSlqgibKSa8=
github.com/scylladb/rclone v1.54.1-0.20240312172628-afe1fd2aa65e/go.mod h1:JGZp4EvCUK+6AM1Fe1dye5xvihTc/Bk0WnHHSCJOePM=
github.com/scylladb/scylla-manager/v3/pkg/managerclient v0.0.0-20241104134613-aba35605c28b h1:JRDV1d1FIiH0TIyHVmTAILAjQ2f8O4t7ZtZ/S+fT2sY=
github.com/scylladb/scylla-manager/v3/pkg/managerclient v0.0.0-20241104134613-aba35605c28b/go.mod h1:Tss7a99vrgds+B70w8ZFG3Skxfr9Br3kAzrKP2b3CmQ=
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241104134613-aba35605c28b h1:7CHNmPrQqSdApaEh5nkRL+D52KFHaOHVBBVDvytHEOY=
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241104134613-aba35605c28b/go.mod h1:+sPCx2oaOXmMpy/ODNNEDGJ7vCghBeKP4S7xEfMI+eA=
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241104134613-aba35605c28b h1:kKMaQhh+ThnPRmCbcatvvfBtglYEFrs5bKaBRIKL/bM=
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241104134613-aba35605c28b/go.mod h1:Oxfuz1XcXi9iV4ggSGfQdn+p6gPz6djPOegRMMe/6/s=
github.com/scylladb/scylla-manager/v3/pkg/managerclient v0.0.0-20241113064359-18103e06f366 h1:Vg4xD0sfXT6OoTHZZ10vQQ20Kyjpxna9H1JJJ1Er2lg=
github.com/scylladb/scylla-manager/v3/pkg/managerclient v0.0.0-20241113064359-18103e06f366/go.mod h1:7+yHDmds+qO/8b0w4xTptCLh/Gr/hXyAjIIZkMj4KHk=
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241113064359-18103e06f366 h1:4r8mRYxPDDFQYuMIAniCJLSA+ZgdSJfCWb6dflr56CY=
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241113064359-18103e06f366/go.mod h1:VKqHSrDj9zfgCKilpbdpmqV1TjmSglt/df79eIg6wuI=
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241113064359-18103e06f366 h1:jHTOgp7WDhvNwmrqZk2J6PNRPDU52L3lEHUH4MAAceA=
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241113064359-18103e06f366/go.mod h1:nCN5P0jiWL0W7jbcZ9p0ndtZAPoyEWXefddx/nbyFes=
github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4 h1:8qmTC5ByIXO3GP/IzBkxcZ/99VITvnIETDhdFz/om7A=
github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4/go.mod h1:C1a7PQSMz9NShzorzCiG2fk9+xuCgLkPeCvMHYR2OWg=
github.com/sirupsen/logrus v1.7.0 h1:ShrD1U9pZB12TX0cVy0DtePoCH97K8EtX+mg7ZARUtM=
Expand Down
27 changes: 27 additions & 0 deletions pkg/scyllaclient/client_scylla.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1125,6 +1125,33 @@
return err
}

func (c *Client) Backup(ctx context.Context, host, endpoint, bucket, prefix, keyspace, table, snapshotTag string) (string, error) {

Check failure on line 1128 in pkg/scyllaclient/client_scylla.go

View workflow job for this annotation

GitHub Actions / Various checks 

exported: exported method Client.Backup should have comment or be unexported (revive)
resp, err := c.scyllaOps.StorageServiceBackupPost(&operations.StorageServiceBackupPostParams{
Context: forceHost(ctx, host),
Endpoint: endpoint,
Bucket: bucket,
Prefix: prefix,
Keyspace: keyspace,
Table: table,
Snapshot: &snapshotTag,
})
if err != nil {
return "", err
}
return resp.GetPayload(), nil
}

func (c *Client) WaitTask(ctx context.Context, host, id string) (*models.TaskStatus, error) {

Check failure on line 1144 in pkg/scyllaclient/client_scylla.go

View workflow job for this annotation

GitHub Actions / Various checks 

exported: exported method Client.WaitTask should have comment or be unexported (revive)
resp, err := c.scyllaOps.TaskManagerWaitTaskTaskIDGet(&operations.TaskManagerWaitTaskTaskIDGetParams{
Context: forceHost(ctx, host),
TaskID: id,
})
if err != nil {
return nil, err
}
return resp.GetPayload(), nil
}

// ToCanonicalIP replaces ":0:0" in IPv6 addresses with "::"
// ToCanonicalIP("192.168.0.1") -> "192.168.0.1"
// ToCanonicalIP("100:200:0:0:0:0:0:1") -> "100:200::1".
Expand Down
39 changes: 28 additions & 11 deletions pkg/service/backup/worker_upload.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,20 +184,37 @@
return nil
}

func (w *worker) uploadDataDir(ctx context.Context, hi hostInfo, dst, src string, d snapshotDir) error {

Check failure on line 187 in pkg/service/backup/worker_upload.go

View workflow job for this annotation

GitHub Actions / Various checks 

unused-parameter: parameter 'dst' seems to be unused, consider removing or renaming it as _ (revive)
// Ensure file versioning during upload
id, err := w.Client.RcloneMoveDir(ctx, d.Host, hi.Transfers, hi.RateLimit.Limit, dst, src, VersionedFileExt(w.SnapshotTag))
//// Ensure file versioning during upload
//id, err := w.Client.RcloneMoveDir(ctx, d.Host, hi.Transfers, hi.RateLimit.Limit, dst, src, VersionedFileExt(w.SnapshotTag))

Check failure on line 189 in pkg/service/backup/worker_upload.go

View workflow job for this annotation

GitHub Actions / Various checks 

commentFormatting: put a space between `//` and comment text (gocritic)
//if err != nil {
// return err
//}
//
//w.Logger.Info(ctx, "Uploading dir", "host", d.Host, "from", src, "to", dst, "job_id", id)
//d.Progress.AgentJobID = id
//w.onRunProgress(ctx, d.Progress)
//
//if err := w.waitJob(ctx, id, d); err != nil {
// w.Logger.Error(ctx, "Upload dir failed", "host", d.Host, "from", src, "to", dst, "error", err)
// return err
//}
//return nil

dstt := w.remoteSSTableDir(hi, d)
// TODO: resolve endpoint either by:
// - making agent return endpoint information to SM
// - allowing for specifying endpoint instead of backed in --location flag
id, err := w.Client.Backup(ctx, hi.IP, "192.168.200.99", hi.Location.Path, dstt, d.Keyspace, d.Table, w.SnapshotTag)
if err != nil {
return err
return errors.Wrap(err, "backup")
}

w.Logger.Info(ctx, "Uploading dir", "host", d.Host, "from", src, "to", dst, "job_id", id)
d.Progress.AgentJobID = id
w.onRunProgress(ctx, d.Progress)

if err := w.waitJob(ctx, id, d); err != nil {
w.Logger.Error(ctx, "Upload dir failed", "host", d.Host, "from", src, "to", dst, "error", err)
return err
status, err := w.Client.WaitTask(ctx, hi.IP, id)
if err != nil {
return errors.Wrap(err, "wait task")
}
if status.State != "done" {
return errors.New(status.Error)
}
return nil
}
Expand Down
1 change: 1 addition & 0 deletions testing/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ up:
@cd scylla/certs && ./generate.sh
@echo "==> Generating Scylla configuration"
@cp scylla/config/scylla.yaml scylla/scylla.yaml
@cp scylla/config/object_storage.yaml scylla/object_storage.yaml
@cp scylla/config/cqlshrc scylla/cqlshrc
ifeq ($(RAFT_SCHEMA),enabled)
@$(YQ) write -i scylla/scylla.yaml 'consistent_cluster_management' true
Expand Down
24 changes: 24 additions & 0 deletions testing/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ services:
- type: bind
source: ./scylla/scylla.yaml
target: /etc/scylla/scylla.yaml
- type: bind
source: ./scylla/object_storage.yaml
target: /etc/scylla/object_storage.yaml
- type: bind
source: ./scylla/certs/
target: /etc/scylla/certs
Expand All @@ -29,6 +32,9 @@ services:
- type: bind
source: ./scylla/scylla.yaml
target: /etc/scylla/scylla.yaml
- type: bind
source: ./scylla/object_storage.yaml
target: /etc/scylla/object_storage.yaml
- type: bind
source: ./scylla/certs/
target: /etc/scylla/certs
Expand All @@ -49,6 +55,9 @@ services:
- type: bind
source: ./scylla/scylla.yaml
target: /etc/scylla/scylla.yaml
- type: bind
source: ./scylla/object_storage.yaml
target: /etc/scylla/object_storage.yaml
- type: bind
source: ./scylla/certs/
target: /etc/scylla/certs/
Expand All @@ -69,6 +78,9 @@ services:
- type: bind
source: ./scylla/scylla.yaml
target: /etc/scylla/scylla.yaml
- type: bind
source: ./scylla/object_storage.yaml
target: /etc/scylla/object_storage.yaml
- type: bind
source: ./scylla/certs/
target: /etc/scylla/certs
Expand All @@ -89,6 +101,9 @@ services:
- type: bind
source: ./scylla/scylla.yaml
target: /etc/scylla/scylla.yaml
- type: bind
source: ./scylla/object_storage.yaml
target: /etc/scylla/object_storage.yaml
- type: bind
source: ./scylla/certs/
target: /etc/scylla/certs
Expand All @@ -109,6 +124,9 @@ services:
- type: bind
source: ./scylla/scylla.yaml
target: /etc/scylla/scylla.yaml
- type: bind
source: ./scylla/object_storage.yaml
target: /etc/scylla/object_storage.yaml
- type: bind
source: ./scylla/certs/
target: /etc/scylla/certs
Expand All @@ -129,6 +147,9 @@ services:
- type: bind
source: ./scylla/scylla-second-cluster.yaml
target: /etc/scylla/scylla.yaml
- type: bind
source: ./scylla/object_storage.yaml
target: /etc/scylla/object_storage.yaml
- type: bind
source: ./scylla/certs/
target: /etc/scylla/certs
Expand All @@ -149,6 +170,9 @@ services:
- type: bind
source: ./scylla/scylla-second-cluster.yaml
target: /etc/scylla/scylla.yaml
- type: bind
source: ./scylla/object_storage.yaml
target: /etc/scylla/object_storage.yaml
- type: bind
source: ./scylla/certs/
target: /etc/scylla/certs
Expand Down
3 changes: 2 additions & 1 deletion testing/scylla/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
/cqlshrc
/scylla.yaml
/scylla-second-cluster.yaml
/scylla-second-cluster.yaml
/object_storage.yaml
8 changes: 8 additions & 0 deletions testing/scylla/config/object_storage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Scylla dev docs: https://github.com/scylladb/scylladb/blob/92db2eca0b8ab0a4fa2571666a7fe2d2b07c697b/docs/dev/object_storage.md?plain=1#L29-L39
endpoints:
- name: 192.168.200.99
port: 9000
https: true
aws_region: us-east-1
aws_access_key_id: miniouser
aws_secret_access_key: minio1234
4 changes: 2 additions & 2 deletions v3/pkg/managerclient/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ require (
github.com/lnquy/cron v1.1.1
github.com/pkg/errors v0.9.1
github.com/scylladb/go-set v1.0.2
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241030073626-e409ae491c83
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241030073626-e409ae491c83
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241113064359-18103e06f366
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241113064359-18103e06f366
github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4
)

Expand Down
8 changes: 4 additions & 4 deletions v3/pkg/managerclient/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,10 +84,10 @@ github.com/rogpeppe/go-internal v1.11.0 h1:cWPaGQEPrBb5/AsnsZesgZZ9yb1OQ+GOISoDN
github.com/rogpeppe/go-internal v1.11.0/go.mod h1:ddIwULY96R17DhadqLgMfk9H9tvdUzkipdSkR5nkCZA=
github.com/scylladb/go-set v1.0.2 h1:SkvlMCKhP0wyyct6j+0IHJkBkSZL+TDzZ4E7f7BCcRE=
github.com/scylladb/go-set v1.0.2/go.mod h1:DkpGd78rljTxKAnTDPFqXSGxvETQnJyuSOQwsHycqfs=
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241030073626-e409ae491c83 h1:6kyuL5XXRTPbBSUessmfz6TxhYgOiIHCwC63wvDExgQ=
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241030073626-e409ae491c83/go.mod h1:+sPCx2oaOXmMpy/ODNNEDGJ7vCghBeKP4S7xEfMI+eA=
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241030073626-e409ae491c83 h1:+2wvERSjYAhGushwWpqw8EemmntHOIcCjHCWzEdpXcM=
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241030073626-e409ae491c83/go.mod h1:Oxfuz1XcXi9iV4ggSGfQdn+p6gPz6djPOegRMMe/6/s=
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241113064359-18103e06f366 h1:4r8mRYxPDDFQYuMIAniCJLSA+ZgdSJfCWb6dflr56CY=
github.com/scylladb/scylla-manager/v3/pkg/util v0.0.0-20241113064359-18103e06f366/go.mod h1:VKqHSrDj9zfgCKilpbdpmqV1TjmSglt/df79eIg6wuI=
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241113064359-18103e06f366 h1:jHTOgp7WDhvNwmrqZk2J6PNRPDU52L3lEHUH4MAAceA=
github.com/scylladb/scylla-manager/v3/swagger v0.0.0-20241113064359-18103e06f366/go.mod h1:nCN5P0jiWL0W7jbcZ9p0ndtZAPoyEWXefddx/nbyFes=
github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4 h1:8qmTC5ByIXO3GP/IzBkxcZ/99VITvnIETDhdFz/om7A=
github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4/go.mod h1:C1a7PQSMz9NShzorzCiG2fk9+xuCgLkPeCvMHYR2OWg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
Expand Down
Loading
Loading