Changes to Gateway infrastructure labels fail to propagate to the service and pods #3666

christiancadieux · 2024-06-24T17:36:32Z

Description:
Changes to Gateway infrastructure labels do not propagate to the service and pods

Repro steps:

create a gateway with infrastruture labels - the corresponding envoy-proxy and service created do include the labels.
update the gateway infrastructure labels - nothing changes in the envoy-proxy/service.

Note:
maybe related to other 'immutable' bugs like #1818
Deleting the Gateway does delete the envoy-proxy deployment

Environment:

Include the environment like gateway version, envoy version and so on.

Gateway

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
spec:
  gatewayClassName: envoygateway-tenant1
  infrastructure:
    labels:
      infra1-label: infra1-value23
...

PODS

$ kubectl get pod --show-labels
NAME                                                        READY   STATUS    RESTARTS   AGE     LABELS
envoy-gateway-5769559676-8rqh4                              1/1     Running   0          17m     app.kubernetes.io/instance=eg-tenant1,app.kubernetes.io/name=gateway-helm,control-plane=envoy-gateway,pod-template-hash=5769559676,tsf.io/service=service1,tsf.io/tenant=tenant1
envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl   2/2     Running   0          8m22s   app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-value2,pod-template-hash=6979c4cbf5

Logs:
the logs when the gateway labels are updated:

2024-06-24T17:51:48.500Z	INFO	provider	kubernetes/controller.go:165	reconciling gateways	{"runner": "provider"}
2024-06-24T17:51:48.500Z	INFO	provider	kubernetes/controller.go:803	processing Gateway	{"runner": "provider", "namespace": "tenant1-ns1", "name": "envoy-gateway"}
2024-06-24T17:51:48.500Z	INFO	provider	kubernetes/routes.go:268	processing HTTPRoute	{"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z	INFO	provider	kubernetes/controller.go:576	processing OIDC HMAC Secret	{"runner": "provider", "namespace": "tenant1-eg", "name": "envoy-oidc-hmac"}
2024-06-24T17:51:48.501Z	INFO	provider	kubernetes/controller.go:1597	processing envoyproxy	{"runner": "provider", "namespace": "tenant1-eg", "name": "proxy-config-tenant1"}
2024-06-24T17:51:48.501Z	INFO	provider	kubernetes/controller.go:374	processing Backend	{"runner": "provider", "kind": "Service", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z	INFO	provider	kubernetes/controller.go:388	added Service to resource tree	{"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z	INFO	provider	kubernetes/controller.go:436	added EndpointSlice to resource tree	{"runner": "provider", "namespace": "tenant1-ns1", "name": "backend-z8xs8"}
2024-06-24T17:51:48.501Z	INFO	provider	kubernetes/controller.go:313	reconciled gateways successfully	{"runner": "provider"}
2024-06-24T17:51:48.501Z	INFO	gateway-api	runner/runner.go:58	received an update	{"runner": "gateway-api"}
2024-06-24T17:51:48.501Z	INFO	provider	kubernetes/status_updater.go:141	received a status update	{"runner": "provider", "namespace": "", "name": "envoygateway-tenant1"}
2024-06-24T17:51:48.502Z	INFO	provider.envoygateway-tenant1	kubernetes/status_updater.go:105	status unchanged, bypassing update	{"runner": "provider"}
2024-06-24T17:51:48.503Z	INFO	gateway-api	runner/runner.go:111	proxy:
  config:
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyProxy
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"EnvoyProxy","metadata":{"annotations":{},"name":"proxy-config-tenant1","namespace":"tenant1-eg"},"spec":{"logging":{"level":{"default":"warn"}},"provider":{"kubernetes":{"envoyDeployment":{"container":{"image":"hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless"}}},"type":"Kubernetes"}}}
      creationTimestamp: "2024-06-20T23:22:25Z"
      generation: 1
      managedFields:
      - apiVersion: gateway.envoyproxy.io/v1alpha1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:annotations:
              .: {}
              f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:spec:
            .: {}
            f:logging:
              .: {}
              f:level:
                .: {}
                f:default: {}
            f:provider:
              .: {}
              f:kubernetes:
                .: {}
                f:envoyDeployment:
                  .: {}
                  f:container:
                    .: {}
                    f:image: {}
              f:type: {}
        manager: kubectl-client-side-apply
        operation: Update
        time: "2024-06-20T23:22:25Z"
      name: proxy-config-tenant1
      namespace: tenant1-eg
      resourceVersion: "24267218"
      uid: b867d886-6c17-47ef-b535-afa743d49e03
    spec:
      logging:
        level:
          default: warn
      provider:
        kubernetes:
          envoyDeployment:
            container:
              image: hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless
        type: Kubernetes
    status: {}
  listeners:
  - address: null
    name: tenant1-ns1/envoy-gateway/http
    ports:
    - containerPort: 8080
      name: http-8080
      protocol: HTTP
      servicePort: 8080
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: envoy-gateway
      gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1
      infra1-label: infra1-value2243
  name: tenant1-ns1/envoy-gateway
	{"runner": "gateway-api", "infra-ir": "tenant1-ns1/envoy-gateway"}
2024-06-24T17:51:48.504Z	INFO	infrastructure	runner/runner.go:78	received an update	{"runner": "infrastructure"}
2024-06-24T17:51:48.504Z	INFO	gateway-api	runner/runner.go:122	accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: tenant1-ns1/envoy-gateway/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 8080
  routes:
  - destination:
      name: httproute/tenant1-ns1/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 198.19.5.80
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.tenant1.example.com
    isHTTP2: false
    name: httproute/tenant1-ns1/backend/rule/0/match/0/www_tenant1_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
	{"runner": "gateway-api", "xds-ir": "tenant1-ns1/envoy-gateway"}
2024-06-24T17:51:48.504Z	INFO	provider	kubernetes/status_updater.go:141	received a status update	{"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.511Z	INFO	provider	kubernetes/status_updater.go:141	received a status update	{"runner": "provider", "namespace": "tenant1-ns1", "name": "envoy-gateway"}
2024-06-24T17:51:48.524Z	ERROR	infrastructure	runner/runner.go:94	failed to create new infra	{"runner": "infrastructure", "error": "failed to create or update deployment tenant1-eg/envoy-tenant1-ns1-envoy-gateway-d016235c: failed to create/update resource with server-side apply for obj &Deployment{ObjectMeta:{envoy-tenant1-ns1-envoy-gateway-d016235c  tenant1-eg    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[] [] [] []},Spec:DeploymentSpec{Replicas:nil,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/component: proxy,app.kubernetes.io/managed-by: envoy-gateway,app.kubernetes.io/name: envoy,gateway.envoyproxy.io/owning-gateway-name: envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1,infra1-label: infra1-value2243,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[prometheus.io/path:/stats/prometheus prometheus.io/port:19001 prometheus.io/scrape:true] [] [] []} {[{certs {nil nil nil nil nil SecretVolumeSource{SecretName:envoy,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {sds {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:envoy-tenant1-ns1-envoy-gateway-d016235c,},Items:[]KeyToPath{KeyToPath{Key:xds-trusted-ca.json,Path:xds-trusted-ca.json,Mode:nil,},KeyToPath{Key:xds-certificate.json,Path:xds-certificate.json,Mode:nil,},},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil}}] [] [{envoy hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless [envoy] [--service-cluster tenant1-ns1/envoy-gateway --service-node $(ENVOY_POD_NAME) --config-yaml admin:\n  access_log:\n  - name: envoy.access_loggers.file\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\n      path: /dev/null\n  address:\n    socket_address:\n      address: 127.0.0.1\n      port_value: 19000\nlayered_runtime:\n  layers:\n  - name: global_config\n    static_layer:\n      envoy.restart_features.use_eds_cache_for_ads: true\n      re2.max_program_size.error_level: 4294967295\n      re2.max_program_size.warn_level: 1000\ndynamic_resources:\n  ads_config:\n    api_type: DELTA_GRPC\n    transport_api_version: V3\n    grpc_services:\n    - envoy_grpc:\n        cluster_name: xds_cluster\n    set_node_on_first_message_only: true\n  lds_config:\n    ads: {}\n    resource_api_version: V3\n  cds_config:\n    ads: {}\n    resource_api_version: V3\nstatic_resources:\n  listeners:\n  - name: envoy-gateway-proxy-ready-0.0.0.0-19001\n    address:\n      socket_address:\n        address: 0.0.0.0\n        port_value: 19001\n        protocol: TCP\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: eg-ready-http\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: prometheus_stats\n              domains:\n              - \"*\"\n              routes:\n              - match:\n                  prefix: /stats/prometheus\n                route:\n                  cluster: prometheus_stats\n          http_filters:\n          - name: envoy.filters.http.health_check\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck\n              pass_through_mode: false\n              headers:\n              - name: \":path\"\n                string_match:\n                  exact: /ready\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n  clusters:\n  - name: prometheus_stats\n    connect_timeout: 0.250s\n    type: STATIC\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: prometheus_stats\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: 127.0.0.1\n                port_value: 19000\n  - connect_timeout: 10s\n    load_assignment:\n      cluster_name: xds_cluster\n      endpoints:\n      - load_balancing_weight: 1\n        lb_endpoints:\n        - load_balancing_weight: 1\n          endpoint:\n            address:\n              socket_address:\n                address: envoy-gateway\n                port_value: 18000\n    typed_extension_protocol_options:\n      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n        \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n        explicit_http_config:\n          http2_protocol_options:\n            connection_keepalive:\n              interval: 30s\n              timeout: 5s\n    name: xds_cluster\n    type: STRICT_DNS\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          tls_params:\n            tls_maximum_protocol_version: TLSv1_3\n          tls_certificate_sds_secret_configs:\n          - name: xds_certificate\n            sds_config:\n              path_config_source:\n                path: \"/sds/xds-certificate.json\"\n              resource_api_version: V3\n          validation_context_sds_secret_config:\n            name: xds_trusted_ca\n            sds_config:\n              path_config_source:\n                path: \"/sds/xds-trusted-ca.json\"\n              resource_api_version: V3\noverload_manager:\n  refresh_interval: 0.25s\n  resource_monitors:\n  - name: \"envoy.resource_monitors.global_downstream_max_connections\"\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig\n      max_active_downstream_connections: 50000\n --log-level warn --cpuset-threads]  [{http-8080 0 8080 TCP } {metrics 0 19001 TCP }] [] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{100 -3} {<nil>} 100m DecimalSI} memory:{{536870912 0} {<nil>}  BinarySI}] []} [] <nil> [{certs true <nil> /certs  <nil> } {sds false <nil> /sds  <nil> }] [] nil &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19001 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/shutdown/ready,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false} {shutdown-manager hub.comcast.net/k8s-eng/envoyproxy/gateway:v1.0.1 [envoy-gateway] [envoy shutdown-manager]  [] [] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{10 -3} {<nil>} 10m DecimalSI} memory:{{33554432 0} {<nil>}  BinarySI}] []} [] <nil> [] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[envoy-gateway envoy shutdown],},HTTPGet:nil,TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false}] [] Always 0xc0009fe038 <nil> ClusterFirst map[] envoy-tenant1-ns1-envoy-gateway-d016235c  0xc0009fe035  false false false <nil> nil []   nil default-scheduler [] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil> nil <nil> [] []}},Strategy:DeploymentStrategy{Type:RollingUpdate,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:*10,Paused:false,ProgressDeadlineSeconds:*600,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}: Deployment.apps \"envoy-tenant1-ns1-envoy-gateway-d016235c\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/component\":\"proxy\", \"app.kubernetes.io/managed-by\":\"envoy-gateway\", \"app.kubernetes.io/name\":\"envoy\", \"gateway.envoyproxy.io/owning-gateway-name\":\"envoy-gateway\", \"gateway.envoyproxy.io/owning-gateway-namespace\":\"tenant1-ns1\", \"infra1-label\":\"infra1-value2243\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"}
2024-06-24T17:51:48.524Z	ERROR	watchable	message/watchutil.go:56	observed an error	{"runner": "infrastructure", "error": "failed to create or update deployment tenant1-eg/envoy-tenant1-ns1-envoy-gateway-d016235c: failed to create/update resource with server-side apply for obj &Deployment{ObjectMeta:{envoy-tenant1-ns1-envoy-gateway-d016235c  tenant1-eg    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[] [] [] []},Spec:DeploymentSpec{Replicas:nil,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/component: proxy,app.kubernetes.io/managed-by: envoy-gateway,app.kubernetes.io/name: envoy,gateway.envoyproxy.io/owning-gateway-name: envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1,infra1-label: infra1-value2243,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[prometheus.io/path:/stats/prometheus prometheus.io/port:19001 prometheus.io/scrape:true] [] [] []} {[{certs {nil nil nil nil nil SecretVolumeSource{SecretName:envoy,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {sds {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:envoy-tenant1-ns1-envoy-gateway-d016235c,},Items:[]KeyToPath{KeyToPath{Key:xds-trusted-ca.json,Path:xds-trusted-ca.json,Mode:nil,},KeyToPath{Key:xds-certificate.json,Path:xds-certificate.json,Mode:nil,},},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil}}] [] [{envoy hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless [envoy] [--service-cluster tenant1-ns1/envoy-gateway --service-node $(ENVOY_POD_NAME) --config-yaml admin:\n  access_log:\n  - name: envoy.access_loggers.file\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\n      path: /dev/null\n  address:\n    socket_address:\n      address: 127.0.0.1\n      port_value: 19000\nlayered_runtime:\n  layers:\n  - name: global_config\n    static_layer:\n      envoy.restart_features.use_eds_cache_for_ads: true\n      re2.max_program_size.error_level: 4294967295\n      re2.max_program_size.warn_level: 1000\ndynamic_resources:\n  ads_config:\n    api_type: DELTA_GRPC\n    transport_api_version: V3\n    grpc_services:\n    - envoy_grpc:\n        cluster_name: xds_cluster\n    set_node_on_first_message_only: true\n  lds_config:\n    ads: {}\n    resource_api_version: V3\n  cds_config:\n    ads: {}\n    resource_api_version: V3\nstatic_resources:\n  listeners:\n  - name: envoy-gateway-proxy-ready-0.0.0.0-19001\n    address:\n      socket_address:\n        address: 0.0.0.0\n        port_value: 19001\n        protocol: TCP\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: eg-ready-http\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: prometheus_stats\n              domains:\n              - \"*\"\n              routes:\n              - match:\n                  prefix: /stats/prometheus\n                route:\n                  cluster: prometheus_stats\n          http_filters:\n          - name: envoy.filters.http.health_check\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck\n              pass_through_mode: false\n              headers:\n              - name: \":path\"\n                string_match:\n                  exact: /ready\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n  clusters:\n  - name: prometheus_stats\n    connect_timeout: 0.250s\n    type: STATIC\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: prometheus_stats\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: 127.0.0.1\n                port_value: 19000\n  - connect_timeout: 10s\n    load_assignment:\n      cluster_name: xds_cluster\n      endpoints:\n      - load_balancing_weight: 1\n        lb_endpoints:\n        - load_balancing_weight: 1\n          endpoint:\n            address:\n              socket_address:\n                address: envoy-gateway\n                port_value: 18000\n    typed_extension_protocol_options:\n      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n        \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n        explicit_http_config:\n          http2_protocol_options:\n            connection_keepalive:\n              interval: 30s\n              timeout: 5s\n    name: xds_cluster\n    type: STRICT_DNS\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          tls_params:\n            tls_maximum_protocol_version: TLSv1_3\n          tls_certificate_sds_secret_configs:\n          - name: xds_certificate\n            sds_config:\n              path_config_source:\n                path: \"/sds/xds-certificate.json\"\n              resource_api_version: V3\n          validation_context_sds_secret_config:\n            name: xds_trusted_ca\n            sds_config:\n              path_config_source:\n                path: \"/sds/xds-trusted-ca.json\"\n              resource_api_version: V3\noverload_manager:\n  refresh_interval: 0.25s\n  resource_monitors:\n  - name: \"envoy.resource_monitors.global_downstream_max_connections\"\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig\n      max_active_downstream_connections: 50000\n --log-level warn --cpuset-threads]  [{http-8080 0 8080 TCP } {metrics 0 19001 TCP }] [] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{100 -3} {<nil>} 100m DecimalSI} memory:{{536870912 0} {<nil>}  BinarySI}] []} [] <nil> [{certs true <nil> /certs  <nil> } {sds false <nil> /sds  <nil> }] [] nil &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19001 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/shutdown/ready,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false} {shutdown-manager hub.comcast.net/k8s-eng/envoyproxy/gateway:v1.0.1 [envoy-gateway] [envoy shutdown-manager]  [] [] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{10 -3} {<nil>} 10m DecimalSI} memory:{{33554432 0} {<nil>}  BinarySI}] []} [] <nil> [] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[envoy-gateway envoy shutdown],},HTTPGet:nil,TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false}] [] Always 0xc0009fe038 <nil> ClusterFirst map[] envoy-tenant1-ns1-envoy-gateway-d016235c  0xc0009fe035  false false false <nil> nil []   nil default-scheduler [] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil> nil <nil> [] []}},Strategy:DeploymentStrategy{Type:RollingUpdate,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:*10,Paused:false,ProgressDeadlineSeconds:*600,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}: Deployment.apps \"envoy-tenant1-ns1-envoy-gateway-d016235c\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/component\":\"proxy\", \"app.kubernetes.io/managed-by\":\"envoy-gateway\", \"app.kubernetes.io/name\":\"envoy\", \"gateway.envoyproxy.io/owning-gateway-name\":\"envoy-gateway\", \"gateway.envoyproxy.io/owning-gateway-namespace\":\"tenant1-ns1\", \"infra1-label\":\"infra1-value2243\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"}

The text was updated successfully, but these errors were encountered:

arkodg · 2024-06-24T18:09:10Z

seeing field is immutable in the logs, so this is same as #1818

christiancadieux · 2024-06-24T18:54:59Z

I don't think it's the same but it's related. for example with Services, it's important to update the labels of the service and not delete/re-create the service since re-creating would assign a new external-IP to the service, which is not good.
Also, when labels come from the Gateway infrastructure, they could be important labels related to the ownership (tenant)
of the Gateway for example, and it's important that the envoy-proxy pod and the service be updated.

arkodg · 2024-06-24T20:37:02Z

i'll bring this up in the community meeting tomorrow, the issue is the same - should Envoy Gateway recreate resources when it hits this specific error field is immutable by default , or should it be based on an opt in flag

christiancadieux · 2024-06-24T21:49:18Z

no need to re-create resources to update labels. It is possible to update labels with PATCH:

$ kubectl label  service/envoy-tenant1-ns1-envoy-gateway-d016235c infra1-label=infra1-test123 --overwrite  -v6
I0624 15:46:02.121803 1444301 loader.go:395] Config loaded from file:  /home/ccadie883/.kube/config
I0624 15:46:02.504242 1444301 round_trippers.go:553] GET https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/services/envoy-tenant1-ns1-envoy-gateway-d016235c 200 OK in 376 milliseconds
I0624 15:46:02.630137 1444301 round_trippers.go:553] PATCH https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/services/envoy-tenant1-ns1-envoy-gateway-d016235c?fieldManager=kubectl-label 200 OK in 124 milliseconds
service/envoy-tenant1-ns1-envoy-gateway-d016235c labeled

$ kubectl get service --show-labels
NAME                                       TYPE           CLUSTER-IP        EXTERNAL-IP     PORT(S)                         AGE     LABELS
envoy-gateway                              ClusterIP      192.168.235.139   <none>          18000/TCP,18001/TCP,19001/TCP   4h18m   app.kubernetes.io/instance=eg-tenant1,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=gateway-helm,app.kubernetes.io/version=v1.0.1,control-plane=envoy-gateway,helm.sh/chart=gateway-helm-v1.0.1
envoy-tenant1-ns1-envoy-gateway-d016235c   LoadBalancer   192.168.254.13    10.112.182.62   8080:9153/TCP                   4h10m   app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-test123

or pod:

$kubectl label  pod/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl infra1-label=infra1-test123 --overwrite  -v6
I0624 15:47:13.898528 1444420 loader.go:395] Config loaded from file:  /home/ccadie883/.kube/config
I0624 15:47:14.284137 1444420 round_trippers.go:553] GET https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/pods/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl 200 OK in 380 milliseconds
I0624 15:47:14.547887 1444420 round_trippers.go:553] PATCH https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/pods/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl?fieldManager=kubectl-label 200 OK in 138 milliseconds
pod/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl labeled

$kubectl get pod envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl  --show-labels
NAME                                                        READY   STATUS    RESTARTS   AGE     LABELS
envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl   2/2     Running   0          4h11m   app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-test123,pod-template-hash=6979c4cbf5

guydc · 2024-06-26T11:37:43Z

-1 to recreation. As stated, there are many possible side effects, including IP change, disruption to traffic, etc.
If possible to solve this with a different strategy (e.g. patch), that should be fine.

arkodg · 2024-07-02T00:51:20Z

hey @sanposhiho can you help with this one if you have a cycle ?
can we make the Patch API https://github.com/envoyproxy/gateway/blob/9a2a7f607e1db52d7aa22daa4c22749cadbf3a91/internal/infrastructure/kubernetes/infra_client.go#L29C24-L29C66 behave like kubectl --overwrite so it doesnt throw an error of field is immutable when updating labels, and also does this w/o recreating the pod or service

sanposhiho · 2024-07-02T01:08:16Z

/assign

I'll take a look.

sanposhiho · 2024-07-02T10:40:30Z

Had a bit of time checking this issue.

According to the provided logs, looks like it doesn't get a conflict at labels, but get conflicted at deployment's selector.
If we fail at updating deployment here, we don't update other following resources, which is why your service isn't updated.
https://github.com/envoyproxy/gateway/blob/main/internal/infrastructure/kubernetes/infra.go#L72-L87

So, I believe this issue is the same as #1818, as @arkodg mentioned first.

shawnh2 · 2024-08-14T11:51:10Z

closed in favour of #3995

christiancadieux added the triage label Jun 24, 2024

christiancadieux changed the title ~~Changes to Gateway infrastructure labels do not propagate to the service and pods~~ Changes to Gateway infrastructure labels fail to propagate to the service and pods Jun 24, 2024

arkodg added kind/bug Something isn't working labels Jun 25, 2024

arkodg modified the milestone: v1.1.0-rc1 Jun 25, 2024

github-actions bot assigned sanposhiho Jul 2, 2024

sanposhiho mentioned this issue Jul 2, 2024

Modify EnvoyProxy podLabels causes reconcilation errors #1818

Closed

arkodg modified the milestones: v1.1.0-rc1, v1.1.0 Jul 3, 2024

arkodg modified the milestones: v1.1.0, v1.2.0-rc1 Jul 31, 2024

sanposhiho mentioned this issue Aug 3, 2024

fix: not generate selector of deployment/daemonset based on the custom label configuration of EnvoyProxy #3995

Merged

shawnh2 closed this as completed Aug 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes to Gateway infrastructure labels fail to propagate to the service and pods #3666

Changes to Gateway infrastructure labels fail to propagate to the service and pods #3666

christiancadieux commented Jun 24, 2024 •

edited

Loading

arkodg commented Jun 24, 2024 •

edited

Loading

christiancadieux commented Jun 24, 2024 •

edited

Loading

arkodg commented Jun 24, 2024

christiancadieux commented Jun 24, 2024

guydc commented Jun 26, 2024

arkodg commented Jul 2, 2024

sanposhiho commented Jul 2, 2024

sanposhiho commented Jul 2, 2024 •

edited

Loading

shawnh2 commented Aug 14, 2024

Changes to Gateway infrastructure labels fail to propagate to the service and pods #3666

Changes to Gateway infrastructure labels fail to propagate to the service and pods #3666

Comments

christiancadieux commented Jun 24, 2024 • edited Loading

arkodg commented Jun 24, 2024 • edited Loading

christiancadieux commented Jun 24, 2024 • edited Loading

arkodg commented Jun 24, 2024

christiancadieux commented Jun 24, 2024

guydc commented Jun 26, 2024

arkodg commented Jul 2, 2024

sanposhiho commented Jul 2, 2024

sanposhiho commented Jul 2, 2024 • edited Loading

shawnh2 commented Aug 14, 2024

christiancadieux commented Jun 24, 2024 •

edited

Loading

arkodg commented Jun 24, 2024 •

edited

Loading

christiancadieux commented Jun 24, 2024 •

edited

Loading

sanposhiho commented Jul 2, 2024 •

edited

Loading