eee

[neo-liang-sap]

Release Notes:

Bumped up the custom image version to v3.4.13-bootstrap-11

aaaaaa
ccccccc

`AllMembersReady` condition has now been fixed to eventually show the correct overall readiness of an etcd cluster.

Bump builder image from `golang:1.20.5` to `golang:1.20.7`

A bug preventing `prometheus` ingress to use `wildcard-certificate` is fixed.

Add CVE categorization for etcd-backup-restore.

dddddd
ddddd
adadadad
/command1
/command2
/comand13 error
/var/root/log

:warning: `etcd.Status.ClusterSize`, `etcd.Status.ServiceName`, `etcd.Status.UpdatedReplicas` have been marked as deprecated and users should refrain from depending on these fields.

A bug causing incorrect volume mount path for `Etcd`s and `EtcdCopyBackupsTask`s using `Local` snapshot storage provider while using distroless etcd-backup-restore image `v0.25.x` has been resolved.

A new feature gate named `ContainerdRegistryHostsDir` is introduced to gardenlet. When enabled, the `/etc/containerd/certs.d` directory is created on the Node and containerd is configured to look up for registries/mirrors configuration in this directory (if there is any configuration applied). In future, the [registry-cache extension](https://github.com/gardener/gardener-extension-registry-cache/) will add such registries/mirrors configuration under this directory (via OperatingSystemConfig mutation).

`default-domain`, `internal-domain`, `alerting` and `openvpn-diffie-hellman` secrets are removed from `gardener-controlplane` Helm chart. Please ensure to update them in a different way before upgrading Gardener. If you would like to prevent Helm from deleting these secret during the upgrade, you could annotate them with `"helm.sh/resource-policy": keep`.

skip hibernated seeds for credentials-rotation

A bug preventing `plutono` ingress to use `wildcard-certificate` is fixed.

Etcd-backup-restore now uses the user home directory to create files.

`UseEtcdWrapper` feature gate has been introduced to allow users to opt for the new [etcd-wrapper](https://github.com/gardener/etcd-wrapper) image.

Kubernetes patch versions `1.27.3`, `1.26.6`, `1.25.11` and `1.24.15` do now transition from `preview` to `supported`. Corresponding previous patch releases (`1.26.5`, `1.25.10` and `1.24.14`) are now marked as `deprecated` and set to expire on 2023-12-15, 23:59:59 UTC.

`hack/generate.sh` has been renamed to `hack/generate-sequential.sh`.

Bumps [github.com/gardener/gardener](https://github.com/gardener/gardener) from 1.75.0 to 1.76.2.

Print build version and go runtime info.

Ensure dns-controller-manager is restarted on CA rotation for remote-access server

`gardenlet` no longer reports the `Bootstrapped` condition on `Seed`s. Instead, it now reports the progress in `.status.lastOperation`, similar to how it's done for `Shoot`s.

The `Shoot` maintenance controller now updates the CRI of worker pools from `docker` to `containerd` when force-upgrading from Kubernetes `v1.22` to `v1.23`.

Etcd-related secrets will now be mounted onto the `/var/` directory instead of `/root/`.

Reduce memory footprint for secrets.

Fixed a bug that caused HVPA reconciliation to fail with `expected pointer, but got v2beta1.MetricSpec type` when the HPA spec had changed.

Operators can now configure alicloud seeds with 3 zones and increased CIDR ranges.

Gardenlet can now set feature gates for `etcd-druid`. They can be specified via the gardenlet configuration `GardenletConfiguration.EtcdConfig.FeatureGates`

The deprecated `extensions/pkg/controller/worker.{Options,ApplyMachineResources{ForConfig}}` symbols have been dropped since `gardenlet` takes over management of the `machine.gardener.cloud/v1alpha1` API CRDs since `gardener/gardener@v1.73`.

`kubectl get garden` now features additional printer columns providing more information about the substantial configuration values and statuses.

Package `pkg/utils/managedresources` now works with immutable secrets for managed resources under the hood. Existing secrets will be marked for garbage collection and replaced with immutable ones during the first reconciliation of the managed resource.

Bumps golang from 1.20.5 to 1.21.0.

Druid now exposes metrics related to snapshot compaction, on default port 8080. Please expose the desired metrics port via the etcd-druid service to allow metrics to be scraped by a Prometheus instance.

Gardener Scheduler's Minimal Distance strategy can take scheduling decisions based on region distances configured by operators. This especially improves the allocation for shoots of providers regions for which the standard Levenshtein distance is inappropriate. Please see `docs/concepts/scheduler.md` for more information.

An issue has been fixed for highly-available `Shoot`s whose `etcd` clusters didn't get ready in the `Completing` phase of a CA credentials rotation.

Operators can now view and manage dashboards for compaction jobs running in shoot control plane.

`pkg/utils/chart` does now support embedded charts. The already deprecated methods in the `ChartApplier` and `ChartRenderer` will be removed in a few releases, so extensions should adapt to embedded charts.

Prolong 1.23.17 expirationDate to 2023-08-20

`gardener-operator` now takes over management of `fluent-operator` and `vali`.

gardenlet: A regression causing metering related recording rules for the aggregate-prometheus not to be applied is now fixed.

Fix set_dependency_version for `workerlessSupported`

Refactored `statefulset`, `service`, `poddisruptionbudget`, `lease`, and `configmap` components to use default labels and owner references from `etcd`.

`maintenance-controller` now disables `PodSecurityPolicy` admission controller when forcefully upgrading the Kubernetes version of a `Shoot` to `v1.25`. It also ensures maximum workers of each for group is greater or equal to its number of zone for forceful upgrades to `v1.27`.

The `security.gardener.cloud/pod-security-enforce` annotation in the ControllerRegistration is set to `baseline`. With this, the pods running in the extension namespace should comply with `baseline` pod-security standard.

The `gardener-apiserver` now drops expired `Kubernetes` and `MachineImage` versions from `Cloudprofile`s during creation.

Two additional labels `worker.gardener.cloud/image-name` and `worker.gardener.cloud/image-version` are attached to worker nodes to identify which operating system they are running. This can then be used in selectors that target only workers with a specific operating system and is helpful for e.g. driver deployment.

fix GCP Project lookup in gcp-limit-alerts

gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed.

`extension-shoot-dns-service` no longer supports Shoots with Кubernetes version < 1.22.

Etcd-druid will now deploy distroless `etcd-wrapper` and `etcd-backup-restore` images. Please refer to [etcd-wrapper](https://github.com/gardener/etcd-wrapper) for more information.

While scaling up a non-HA etcd cluster to HA skipping the scale-up checks for first member of etcd cluster as first member can never be a part of scale-up scenarios.

The garbage collection controller now also considers managed resources when deciding if secrets/configmaps should be garbage collected.

Extensions running on seed clusters can get access to the garden cluster by using the injected kubeconfig specified by the `GARDEN_KUBECONFIG` environment variable. You can read about the details in this [doc](https://github.com/gardener/gardener/blob/master/docs/extensions/garden-api-access.md).

File ownership for `var/etcd/data` will be changed to non-root user (65532).

Bump builder image golang from `1.20.4` to `1.20.6` 

Developer Action Required: The `make deploy` command has been replaced with `make deploy-via-kustomize`. Please update your deployment workflows accordingly.

When `Shoot`s were updated from non high-availability to `zone` high-availability, it could happen that the control-plane was scheduled to two instead of three zones. This issue is relevant for cloud providers with an inconsistent zone naming (`Azure` is currently the only candidate to our knowledge).
Existing shoots with the before mentioned problem must be fixed manually be operators if required. An automatic move of `etcd`s and their volumes is not part of this fix due to availability reasons.

Add CVE categorization for etcd-druid.

Feature gates have been introduced in etcd-druid, and can be specified using CLI flag `--feature-gate`.

The `charts/images.yaml` file was moved to `imagevector/images.yaml`.

Backup-restore waits for its etcd to be ready before attempting to update peerUrl

The `virtual-garden-kube-apiserver` service (for the `virtual-garden` cluster) was switched from type `LoadBalancer` to `ClusterIP`. Please make sure to migrate all DNS records from the `virtual-garden-kube-apiserver` to the `istio-ingressgateway` endpoint before upgrading to this Gardener version.

The `github.com/golang/mock/gomock` dependency is replaced by `go.uber.org/mock`.

Makefile has been updated to use `Skaffold` for deploying `etcd-druid` with the `make deploy` target, simplifying the deployment process and eliminating the need to push the image to the container registry for each local development testing.

If you are using `provider-extension` setup you should adapt your files in `example/provider-extensions/garden/controlplane` because `default-domain` and `internal-domain` secrets are removed from `gardener-controlplane` Helm chart.

Update controller-manager-library dependency to fix panic on api-resources discovery.

A bug causing `EtcdCopyBackupsTask` jobs to fail to create temp snapshot directory while using distroless etcd-backup-restore image `v0.25.x` has been resolved.

Components `gardener-operator` and `dashboard` now create required DNS records with the help of the centrally running `dns-management` controller in the Garden Runtime cluster. While `gardener-operator` already requires an existing `DNSProvider` for the Istio-Ingress namespace, an additional provider for the `garden` namespace is needed. This can be achieved via `.components.dnsManagement.providers` in `landscape.yaml`. Please a provider that is permitted to manage records for `.clusters.externalDomain` and `.clusters.dns.accessDomain`.
In addition, a `defaultTTL` setting can be passed to `DNSProviders`. It configures the default TTL value for all `DNSEntries` that don't specify otherwise.

The `Secrets` type as well as the `Delete` functions for secrets were removed from `pkg/utils/managedresources/builder` since their usage was prone to errors. The higher level package `pkg/utils/managedresources` should be used instead.

Add failure tolerance option to the `CreateShoot` test.

Etcd-backup-restore now uses a distroless image as its base image. It is no longer compatible with [etcd-custom-image](https://github.com/gardener/etcd-custom-image), and must be used with [etcd-wrapper](https://github.com/gardener/etcd-wrapper) instead. 

When scaling from single-node to multi-node etcd cluster, Etcd Druid will now first ensure that any change to the peer URL (e.g TLS enablement)  is seen by the existing etcd process running within the etcd member pod. Once that is confirmed then it will scale up the Etcd StatefulSet and add relevant annotations.