Kyma Environment Broker integration with Infrastructure Manager

[PK85]

Description

In order to provide support for dynamic kubeconfigs in the Control Plane, KEB must integrate with the Cluster Inventory.

Reasons

In the current architecture, KEB takes kubeconfig from the Provisioner, and creates a secret. The kubeconfig returned by the Provisioner is static, and should be no longer used by other components. Cluster Inventory is a new component that is responsible for dynamic kubeconfig management. KEB must delegate the task of kubeconfig secret creation to the Cluster Inventory by creating cluster CR.

High-Level Diagram

Acceptance criteria

• 1) KEB must create Cluster CR containing the following data:
  • shoot name
  • secret name
• 2) Cluster CR must contain the same set of labels as defined here
• 3) Cluster CR name contains RuntimeID
• 4) The functionality in KEB is enabled by a feature flag
• 5) Decide how to pass responsibility for secret management to the infrastructure manager.
• 6) Check if KEB jobs require adjustments
• 7) Activate feature on dev and check if e2e are green
• 8) KEB generates RuntimeID, instead of using the one generated by the Provisioner (we need to double check that if this is needed now. Maybe we should start to generate runtimeID when provisioned will be gone)

Info

• The exact structure of the CR will be defined in this issue: Infrastructure Manager - create initial project structure infrastructure-manager#9 .
• Provisioner will keep functionality by getting kubeconfig via getRuntimeStatus

[piotrmiskiewicz]

Technical details:

1. A new step which creates a Cluster CR with provided shoot name and other important fields:

• example of the CR: https://github.com/kyma-project/infrastructure-manager/blob/main/config/samples/infrastructuremanager_v1_gardenercluster.yaml
• the step is enabled with a feature flag, by default, the step is disabled
• the step is executed after the provisioner interaction, then wait for expected status (next step), those steps should be reused in the update process
• the deprovisioning process removes the CR - new step, this is not guarded by feature flag!

NOTE: the existing clusters will be handled by migration tool written by Frogs

2. The RuntimeID generation by KEB will be postponed and run when Provisioner is finally detached

[piotrmiskiewicz]

Tasks:

• Implement a step which creates a CR with proper labels and spec: Gardener Cluster step #182
• Implement a step, which checks an expected state: Check gardener cluster #197
• Add those steps to the provisioning process, add a test in cmd/broker for it (assertion for GardenerCluster, simulate infrastructure manager), add a feature flag which is disabled by default [in progress] infrastructure manager feature flag #201 https://github.tools.sap/kyma/management-plane-config/pull/4650
• Implement and add deprovisioning steps (remove cluster CR and wait for it). This step must be run before deleteKubeconfig. Finally deleteKubeconfig can be removed. PR: Check gardenercluster deletion #213
• syncKubeconfig must not update the secret, just create if needed
• All getting kubeconfig steps (except getkubconfig in provisioning process) must rely on the secret
• create the kubeconfig secret only in it was not created before
• skr go tests: https://github.com/kyma-project/control-plane/blob/main/tests/e2e/provisioning/pkg/client/runtime/runtime_client.go#L55
• All jobs (like runtime reconciler) should use the secret
• RBAC: GardenerCluster RBAC for KEB #212

[ralikio]

Done