Multicluster environments

[eedorenko]

Consider the following use case:
An organization has a few K8s clusters across the globe. Let's say a cluster per region (EMEA, Asia, America). There are also a few environments (Dev, Qa, Prod) distributed across these clusters. So each environment is represented on all clusters. When an application is being deployed to a Dev environment, for example, it's being deployed to all three clusters (in Dev environment). However, on each cluster the application behaves slightly differently (e.g. it communicates to external services hosted in that region). The desire is to have a single GitOps setup of the application (aka bundle in Rancher Fleet) declaring that it should be deployed across all available clusters in the environment with a specific configuration for each cluster (e.g. different DB connection strings in different clusters, different load balancing settings, even different image tags).

The key point is that the sync process on all clusters should be considered as a single unit. If a deployment on one cluster fails, the whole deployment/sync process should fail (specifying that it was successful on Asia and America clusters, but failed in EMEA). I want to see the state of this GitOps setup in one place (not jumping on clusters) saying that it is in sync on these clusters, healthy on that cluster and out of sync on that one.

Is there any design considerations addressing this multi cluster use case?

[stealthybox]

Flux 2 supports these use-cases via its Remote Apply Feature, Path+Ref specific Sources, and Events/Metrics.

Remote Sync

Flux can ensure workloads are applied, health-checked, and garbage-collected in remote clusters.
See these docs for Cluster API considerations:

• Kustomization: spec.kubeConfig
• HelmRelease: spec.kubeConfig

These two demos do remote applies from a central management cluster:

1. ClusterAPI + Flux + GPG verification
   Repo: https://github.com/stealthybox/capi-flux-demo
   GitOpsDays Video: https://www.youtube.com/watch?v=7W27tAv7Tvs

2. Multi-Cluster CNI management /w DNS Mesh & Route Sharing
   Repo: https://github.com/stealthybox/multicluster-gitops
   KubeCon Video: https://youtu.be/as4NU-Z8dlE?t=1244

Configuring apps for environments/namespaces/clusters

We have this example regarding releasing apps to multiple environments and clusters

• Power of GitOps Part 3
  Repo: https://github.com/stealthybox/flux-app-demo
  Video: https://www.youtube.com/watch?v=N_K5g7o9JKg

Separate paths in a Source can be used to modify a kustomize base (or other manifest generator).
This can be done per-environment.

Environment specific tags can be used for the GitRepository ref alongside those paths to release the desired version of each environment separately.
This allows you to commit changes to all environments at the same time while using tags to rollout changes incrementally.

Viewing all sync info from a single place

When using a management cluster /w remote sync via the Kustomization or HelmRelease spec.kubeConfig, all sync state (object names/errors/commit-shas/etc) can be read within a single cluster.
If desired, all of these sync related resources can be in a single Namespace and be viewed simultaneously using a tool like Octant, Lens, k9s, the upcoming Flux UI, or even just kubectl.

This data can also be scraped from the controllers via Prometheus and displayed in a Grafana Dashboard.
If you prefer, to run a Flux install per-cluster without any central management cluster, these metrics could still be aggregated into a central reporting Prometheus/Grafana.

You can also hook up Alerts for Providers using notification-controller's API's to be notified of any issues via a chat channel or other paging service.

[eedorenko]

@stealthybox thanks for the thorough response! I am digging into the resources you provided. The central management cluster approach looks promising.