Envoy Gateways are restarted when a resource is updated

[alexandermarston]

Description:

Envoy Gateway is restarting a lot when we roll out configuration changes to HTTPRoutes, SecurityPolicies etc.

Merge Gateways is not enabled.

Potentially similar to #2965 and #2637

Repro steps:

1. Modify/Delete/Create a SecurityPolicy resource
2. Observe the following in the Envoy Gateway controller logs:

2024-05-15T08:55:51.328Z	INFO	provider	kubernetes/controller.go:141	reconciling gateways	{"runner": "provider"}
2024-05-15T08:55:51.328Z	INFO	provider	kubernetes/controller.go:714	processing Gateway	{"runner": "provider", "namespace": "envoy-gateway", "name": "dev"}
.....

3. Observe that for each Gateway, two of the three replicas have been terminated and new pods re-created

➜  ~ kgpo
NAME                                                READY   STATUS    RESTARTS   AGE
envoy-envoy-gateway-dev-73146645-7d9bc4c49b-6dt5s   2/2     Running   0          14m
envoy-envoy-gateway-dev-73146645-7d9bc4c49b-qxkzh   2/2     Running   0          3m12s <- NEW PODS
envoy-envoy-gateway-dev-73146645-7d9bc4c49b-vgx74   2/2     Running   0          3m12s <- NEW PODS
envoy-envoy-gateway-qa-81cea4ab-d7cf98866-8qfq7     2/2     Running   0          3m5s <- NEW PODS
envoy-envoy-gateway-qa-81cea4ab-d7cf98866-9vt5c     2/2     Running   0          14m
envoy-envoy-gateway-qa-81cea4ab-d7cf98866-rwpnx     2/2     Running   0          3m6s <- NEW PODS
envoy-gateway-865b76c664-vg256                      1/1     Running   0          11m

Environment:

Envoy Gateway: v1.0.1

Helm Values for Envoy Gateway installation:

    deployment:
      replicas: 1
      pod:
        annotations:
          fluentbit.io/exclude: "true"
      envoyGateway:
        resources:
          limits:
            cpu: 250m
            memory: 3Gi
          requests:
            cpu: 250m
            memory: 3Gi

Logs:

[arkodg]

@alexandermarston can you try this out on v0.0.0-latest ? we've added some fixes #3130 to avoid this