Iterating all clusters to create global resources

[thecodeassassin]

We want to create a NATS supercluster and for that we need each NATS instance provisioned by sveltos to know about other clusters.

One way to achieve this is to simply iterate all eligible clusters and add them to each other:

            {{ range $i, $cluster := .Clusters }}
            gateways:
              - name: $cluster.metadata.name
                url: $cluster.metadata.name.sd1213.ts.net:7222
            {{ end }}

Other use-cases for this could include any distributed system that needs to know about its neigbors, for example RAFT-based systems such as consul.

[gianlucam76]

Thank you. This is an excellent use case. It can be achieved using templating and event framework.

Sveltos templating lets you define add-ons as templates and fill in their values at deployment time using resources within the management cluster.

So we need a resource in the management cluster containing list of managed clusters. Event framework can help us here.

Identifying Managed Clusters

In this example, the management cluster has a label type:mgmt. We can leverage Sveltos' event framework to dynamically identify all managed clusters (of type SveltosCluster) that are different from the management cluster (only cluster with label type:mgmt):

apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventSource
metadata:
 name: detect-clusters
spec:
 collectResources: false
 resourceSelectors:
 - group: "lib.projectsveltos.io"
   version: "v1alpha1"
   kind: "SveltosCluster"
   labelFilters:
   - key: sveltos-agent
     operation: Equal
     value: present
   - key: type
     operation: Different
     value: mgmt
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventTrigger
metadata:
 name: detect-cluster
spec:
 sourceClusterSelector: type=mgmt
 eventSourceName: detect-cluster
 oneForEvent: false

This process generates an EventReport resource named detect-clusters within the projectsveltos namespace. This report stores information about all discovered managed clusters. Sveltos templating can then reference this resource to retrieve the list of managed clusters, allowing for dynamic configuration based on your cluster environment.

apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventReport
...
spec:
  eventSourceName: detect-clusters
  matchingResources:
  - apiVersion: lib.projectsveltos.io/v1alpha1
    kind: SveltosCluster
    name: cluster-1
    namespace: civo
  - apiVersion: lib.projectsveltos.io/v1alpha1
    kind: SveltosCluster
    name: cluster-2
    namespace: civo

Automatic Service Creation

Now, let's say you have a NATS supercluster (with the label type:nat) and you want to deploy a Service for each managed cluster. Here's how Sveltos achieves this with templating and referencing the EventReport resource the event framework created:

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  clusterSelector: type=nats
  templateResourceRefs:
  - resource:
      apiVersion: lib.projectsveltos.io/v1alpha1
      kind: EventReport
      name: detect-clusters
      namespace: projectsveltos
    identifier: ClusterData
  policyRefs:
  - kind: ConfigMap
    name: nats-services
    namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nats-services
  namespace: default
  annotations:
      projectsveltos.io/template: "true"
data:
  services.yaml: |
    {{ range $cluster := (index .MgmtResources "ClusterData").spec.matchingResources }}
      apiVersion: v1
      kind: Service
      metadata:
        labels:
          app: nats
          tailscale.com/proxy-class: default
        annotations:
          tailscale.com/tailnet-fqdn: nats-{{ $cluster.name }}
        name: nats-{{ $cluster.name }}
      spec:
        externalName: placeholder
        type: ExternalName
    ---
    {{ end }}

For instance I have two managed clusters (on top of the management cluster)

kubectl get sveltoscluster -A --show-labels
NAMESPACE  NAME    READY  VERSION    LABELS
civo    cluster-1  true  v1.29.2+k3s1  sveltos-agent=present
civo    cluster-2  true  v1.28.7+k3s1  sveltos-agent=present
mgmt    mgmt    true  v1.30.0    sveltos-agent=present,type=mgmt

Two Service instances will be deployed in the NATS super cluster

KUBECONFIG=test/fv/workload_kubeconfig kubectl get service -n default
NAME       TYPE      CLUSTER-IP  EXTERNAL-IP  PORT(S)  AGE
...
nats-cluster-1  ExternalName  <none>    placeholder  <none>  68s
nats-cluster-2  ExternalName  <none>    placeholder  <none>  68s
...

Automatic Updates

The beauty of Sveltos is its automatic reaction to changes. If you create or destroy managed clusters, Sveltos will automatically:

1. Detect the change through the detect-clusters EventSource.
2. Re-evaluate the nats-services template with the updated data.
3. Add or remove Service instances in the NATS supercluster as needed.

This ensures your NATS supercluster remains dynamically updated based on your managed cluster configuration.