Bug Report: Not able to list backups using kubectl get vitessbackups

[frouioui]

Moving vitessio/vitess#14766 here as it is related to the vitess-operator.

cc @GenLN as the original author.

---

Overview of the Issue

This is not working for 2 vitess cluster (only for one in separate namespace) on same K8S cluster.

Reproduction Steps

1. Deploy operator and CRD into namespace A
2. Deploy vitess cluster1 on namespace A
3. Run full backups
4. Run kubectl get vitessbackups into namespace A(it works)
5. Update operator deployment and assign also to watch namespace B
6. Deploy vitess cluster2 on namespace B
7. Run full backups
8. Run kubectl get vitessbackups into namespace B (empty for cluster2, even the backups are done and uploaded to gcs,s3 etc...)
9. Run kubectl get vitessbackups into namespace A(list for cluster1 vitessbackups are shown)

Second test different K8S cluster:

1. Deploy operator and CRD into namespace A1
2. Deploy cluster11 into namespace A1
3. Run full backups
4. Run kubectl get vitessbackups into namespace A1 for cluster11(backup list shown)
5. Deploy Operator deployment kind only into namespace B1
6. Deploy cluster21 into namespace B1
7. Run full backups
8. Run kubectl get vitessbackups into namespace B1 for cluster 21 (empty list no backups shown)
9. Run kubectl get vitessbackups into namespace A1 for cluster 11 (vitessbackups list shown)

Binary Version

Vitess v18.0.1
latest tag vitess operator

Operating System and Environment details

K8S cluster

Log Fragments

No response

[frouioui]

@GenLN, what is the reason behind using different namespaces? Also could you provide a more in depth reproduction, maybe a script with Kind.

[GenLN]

What I tried:

• Set up two Vitess clusters on the same Kubernetes cluster, each in a different namespace, with a single Vitess operator watching both namespaces.

• Set up two Vitess clusters on the same Kubernetes cluster, each in a different namespace, with a dedicated Vitess operator for each, watching only their respective cluster namespace.

I encountered some issues with kubectl commands, like retrieving Vitess backups, etc.

I'm wondering if it's possible to run two Vitess clusters on the same Kubernetes cluster without them interfering with each other or causing issues with Kubernetes itself, such as with kubectl commands, etc.

[frouioui]

Gotcha, and what is the reason behind using two different namespace? Because running having a single vitess-operator manage multiple VitessCluster under the same namespace would work as expected.

$ kubectl get pods
NAME                                                           READY   STATUS             RESTARTS       AGE
cluster-2-commerce-x-x-zone1-vtorc-92ce45e3-7d8976d6dd-47ppx   1/1     Running            0              6m50s
cluster-2-etcd-6f56f71c-1                                      1/1     Running            0              6m51s
cluster-2-etcd-6f56f71c-2                                      1/1     Running            0              6m51s
cluster-2-etcd-6f56f71c-3                                      1/1     Running            0              6m51s
cluster-2-vttablet-zone1-0790125915-34ec1b9e                   3/3     Running            0              6m50s
cluster-2-vttablet-zone1-2469782763-b8c5852e                   3/3     Running            0              6m50s
cluster-2-vttablet-zone1-2548885007-7477da87                   3/3     Running            0              6m50s
cluster-2-zone1-vtctld-819a43a1-8564fbbd7b-5vzwt               1/1     Running            0              6m51s
cluster-2-zone1-vtgate-fd740669-59c489f9cd-p8c4m               1/1     Running            0              6m50s
example-90089e05-vitessbackupstorage-subcontroller             1/1     Running            0              36m
example-commerce-x-x-vtbackup-init-c6db73c9                    0/1     CrashLoopBackOff   10 (24s ago)   36m
example-commerce-x-x-zone1-vtorc-c13ef6ff-747599bc4d-l87qt     1/1     Running            1 (27m ago)    36m
example-etcd-faf13de3-1                                        1/1     Running            0              36m
example-etcd-faf13de3-2                                        1/1     Running            0              36m
example-etcd-faf13de3-3                                        1/1     Running            0              36m
example-vttablet-zone1-0790125915-4e37d9d5                     3/3     Running            0              36m
example-vttablet-zone1-2469782763-bfadd780                     3/3     Running            0              36m
example-vttablet-zone1-2548885007-46a852d0                     3/3     Running            0              36m
example-zone1-vtctld-1d4dcad0-6d65696dd4-tmtmn                 1/1     Running            1 (27m ago)    36m
example-zone1-vtgate-bc6cde92-695477d5b-fn9xp                  1/1     Running            1 (27m ago)    36m
vitess-operator-586755589d-59h5f                               1/1     Running            0              36m

[GenLN]

it was supposed to be different k8s projects with different namespaces on single k8s cluster... so we had that issue with listing backups for other vitess cluster, the first cluster deployed will return list of backups, but second one would not...