When there are multiple Gateways having listeners with same port and mergeGateway enabled, using CORS for each Gateway separately, only the first Gateway works.

[NamiKazu]

Description:
When there are multiple Gateways having listeners with same port and mergeGateway enabled, using CORS for each Gateway separately, only to the first Gateway will work.

Repro steps:
Install the Gateway API CRDs and Envoy Gateway

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.0.0-latest -n envoy-gateway-system --create-namespace

Use aill-in-one yaml described below:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: custom-proxy-config
    namespace: envoy-gateway-system

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  mergeGateways: true 
---
apiVersion: v1
kind: Namespace
metadata:
  name: sealos-system
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: sealos-system
spec:
  gatewayClassName: eg
  listeners:
    - hostname: "*.192.168.0.15.nip.io"
      name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: v1
kind: Namespace
metadata:
  name: ns-oehe
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bdkzlmibsivuiqav
  namespace: ns-oehe
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bdkzlmibsivuiqav
  template:
    metadata:
      labels:
        app: bdkzlmibsivuiqav
    spec:
      containers:
      - image: nginx:1.25
        name: nginx
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 64Mi
          requests:
            cpu: 10m
            memory: 16Mi
---
apiVersion: v1
kind: Service
metadata:
  name: bdkzlmibsivuiqav
  namespace: ns-oehe
spec:
  clusterIP: "10.96.0.2"
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: bdkzlmibsivuiqav
  type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bdkzlmibsivuiqav
  namespace: ns-oehe
spec:
  hostnames:
  - ntjxuedx.192.168.0.15.nip.io
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: sealos-system
    sectionName: http
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: bdkzlmibsivuiqav
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
---
apiVersion: v1
kind: Namespace
metadata:
  name: ns-vrpg
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mfqjpuycbgjrtdww
  namespace: ns-vrpg
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mfqjpuycbgjrtdww
  template:
    metadata:
      labels:
        app: mfqjpuycbgjrtdww
    spec:
      containers:
      - image: nginx:1.25
        name: nginx
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 64Mi
          requests:
            cpu: 10m
            memory: 16Mi
---
apiVersion: v1
kind: Service
metadata:
  name: mfqjpuycbgjrtdww
  namespace: ns-vrpg
spec:
  clusterIP: "10.96.0.3"
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: mfqjpuycbgjrtdww
  type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mfqjpuycbgjrtdww
  namespace: ns-vrpg
spec:
  gatewayClassName: eg
  listeners:
  - hostname: qccbahgo.qccbahgo
    name: http
    port: 80
    protocol: HTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mfqjpuycbgjrtdww
  namespace: ns-vrpg
spec:
  hostnames:
  - qccbahgo.qccbahgo
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: mfqjpuycbgjrtdww
    namespace: ns-vrpg
    sectionName: http
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: mfqjpuycbgjrtdww
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: cors-example
  namespace: ns-vrpg
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: mfqjpuycbgjrtdww
  cors:
    allowOrigins:
    - "http://*.foo.com"
    allowMethods:
    - PUT
    - GET
    - POST
    - DELETE
    - PATCH
    - OPTIONS
    maxAge: 600s
    allowCredentials: true

Get the name of the Envoy service created the by the Gateway, because mergeGateway is enabled, the way to obtain the Envoy service differs from the quickstart.

export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gatewayclass=eg -o jsonpath='{.items[0].metadata.name}')

Port forward to the Envoy service

kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &

Verify that the CORS headers are present in the response of the OPTIONS request from http://www.foo.com, and the host is qccbahgo.qccbahgo

curl -H "Origin: http://www.foo.com" \
  -H "Host: qccbahgo.qccbahgo" \
  -H "Access-Control-Request-Method: GET" \
  -X OPTIONS -v -s \
  http://localhost:8888/

The following is the response, indicating that the request was not allowed, which is not reasonable.

...
< HTTP/1.1 405 Method Not Allowed
< server: nginx/1.25.4
...

In this case, I applied the SecurityPolicy to the second Gateway, but the SecurityPolicy did not work, although when I checked the status of the SecurityPolicy, it showed as Accepted.

Furthermore, when I applied the SecurityPolicy only to the first Gateway, it works. The steps are as follows:

1. Delete the existing SecurityPolicy

kubectl delete SecurityPolicy cors-example -n ns-vrpg

2. Apply the new SecurityPolicy to the first Gateway

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: cors-example
  namespace: sealos-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  cors:
    allowOrigins:
    - "http://*.foo.com"
    allowMethods:
    - PUT
    - GET
    - POST
    - DELETE
    - PATCH
    - OPTIONS
    maxAge: 600s
    allowCredentials: true

3. Verify that the CORS headers are present in the response of the OPTIONS request from http://www.foo.com, and the host is ntjxuedx.192.168.0.15.nip.io

curl -H "Origin: http://www.foo.com" \
  -H "Host: ntjxuedx.192.168.0.15.nip.io" \
  -H "Access-Control-Request-Method: GET" \
  -X OPTIONS -v -s \
  http://localhost:8888/

The following is the response, indicating that the SecurityPolicy works.

< HTTP/1.1 200 OK
< access-control-allow-origin: http://www.foo.com
< access-control-allow-credentials: true
< access-control-allow-methods: PUT, GET, POST, DELETE, PATCH, OPTIONS
< access-control-max-age: 600

---

So why does the SecurityPolicy behave differently when applied to different gateways? I tried to find out the reason. I used egctl to analyze the differences in applying the SecurityPolicy to the first Gateway and the second Gateway separately.

• When applying the SecurityPolicy to the first one, compared to the xds without the SecurityPolicy, as shown in the diagram, it adds an HTTP filter to the listener and adds the corresponding filter to the RouteConfiguration.

  • to Listener
  
  • to RouteConfiguration
  

• When applying the SecurityPolicy to the second one, compared to the xds without the SecurityPolicy, as shown in the diagram, it only adds the corresponding filter to the RouteConfiguration.

  • to RouteConfiguration
  

So, I think this is the reason why the SecurityPolicy doesn't work when applied to the second Gateway. It lacks the HTTPFilter that should be added to the Listener.

---

So, why does it lack? Is it because the listener to which the HTTP filter should be added cannot be found or something? I think this is the key to solving the problem.

[arkodg]

thanks for raising this as well as debugging this !
since you've already done the work and showed that the discrepancy is in the CORS filter is missing in hcm
feels like we may need to revisit

gateway/internal/xds/translator/cors.go

Line 37 in 74f86f9

func (*cors) patchHCM(

cc @zhaohuabing

[liorokman]

After recreating the issue I notice these errors in the Envoy Gateway logs:

2024-03-02T06:20:22.225Z	ERROR	infrastructure	runner/runner.go:70	failed to create new infra	{"runner": "infrastructure", "error": "failed to create or update service envoy-gateway-system/envoy-eg-d8c59e83: for Update: Service \"envoy-eg-d8c59e83\" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:\"\", Protocol:\"TCP\", AppProtocol:(*string)(nil), Port:80, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}"}
2024-03-02T06:20:22.225Z	ERROR	watchable	message/watchutil.go:56	observed an error	{"runner": "infrastructure", "error": "failed to create or update service envoy-gateway-system/envoy-eg-d8c59e83: for Update: Service \"envoy-eg-d8c59e83\" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:\"\", Protocol:\"TCP\", AppProtocol:(*string)(nil), Port:80, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}"}

After fixing this, the original problem doesn't reproduce.

[liorokman]

@NamiKazu could you please try to reproduce the issue when the patch in #2751 is applied?

[liorokman]

@NamiKazu could you please try to reproduce the issue when the patch in #2751 is applied?

Nevermind - this was not related.

[liorokman]

The problem is that when mergeGateways is enabled and TLS is not configured, the listener configuration is done on the defaultFilterChain.

In this case there are multiple available configurations (one per each merged gateway), but only one defaultFilterChain to receive them.

This means that everything that can be configured in the defaultFilterChain will only be used once - per the first handled Gateway resource. This happens to include additional filters like the Cors filter, which is why the issue was opened on CORS. But it actually affects anything that can be configured on the listener level.

I think that if mergeGateways is enabled, the defaultFilterChain shouldn't be used at all. The mergeGateways feature is predicated on there being some way to distinguish between the routes attached to different gateways anyways.

But since Envoy Proxy only allows filter matching on serverName to be made based on SNI, I don't think this can be fixed if the merged gateways don't use TLS.

[arkodg]

hey @liorokman @NamiKazu can you share what the IR Xds looks like (available in logs), will be easier to reason once we have that

[NamiKazu]

hi @arkodg @liorokman the following is the xds IR that I found in logs. I hope it can help.

2024-03-04T02:46:48.252Z        INFO    gateway-api     runner/runner.go:102    accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*.192.168.0.15.nip.io'
  isHTTP2: false
  name: sealos-system/eg/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 10080
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/ns-oehe/bdkzlmibsivuiqav/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.244.0.8
          port: 80
        protocol: HTTP
        weight: 1
    hostname: ntjxuedx.192.168.0.15.nip.io
    isHTTP2: false
    name: httproute/ns-oehe/bdkzlmibsivuiqav/rule/0/match/0/ntjxuedx_192_168_0_15_nip_io
    pathMatch:
      distinct: false
      name: ""
      prefix: /
- address: 0.0.0.0
  hostnames:
  - qccbahgo.qccbahgo
  isHTTP2: false
  name: ns-vrpg/mfqjpuycbgjrtdww/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 10080
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    cors:
      allowCredentials: true
      allowMethods:
      - PUT
      - GET
      - POST
      - DELETE
      - PATCH
      - OPTIONS
      allowOrigins:
      - distinct: false
        name: ""
        safeRegex: http://.*\.foo\.com
      maxAge: 10m0s
    destination:
      name: httproute/ns-vrpg/mfqjpuycbgjrtdww/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.244.0.9
          port: 80
        protocol: HTTP
        weight: 1
    hostname: qccbahgo.qccbahgo
    isHTTP2: false
    name: httproute/ns-vrpg/mfqjpuycbgjrtdww/rule/0/match/0/qccbahgo_qccbahgo
    pathMatch:
      distinct: false
      name: ""
      prefix: /
        {"runner": "gateway-api", "xds-ir": "eg"}

[NamiKazu]

Furthermore, if needed, here are the details of the xds I obtained using egctl. Here, I only focus on listener and routeConfiguration, the rest of the parts are identical.

Firstly, here is the xds without using CORS.

xds:
  eg:
    configs:
    ...
    - '@type': type.googleapis.com/envoy.admin.v3.ListenersConfigDump
      dynamicListeners:
      - activeState:
          listener:
            '@type': type.googleapis.com/envoy.config.listener.v3.Listener
            accessLog:
            - filter:
                responseFlagFilter:
                  flags:
                  - NR
              name: envoy.access_loggers.file
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                logFormat:
                  textFormatSource:
                    inlineString: |
                      {"start_time":"%START_TIME%","method":"%REQ(:METHOD)%","x-envoy-origin-path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","response_code_details":"%RESPONSE_CODE_DETAILS%","connection_termination_details":"%CONNECTION_TERMINATION_DETAILS%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","duration":"%DURATION%","x-envoy-upstream-service-time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","x-forwarded-for":"%REQ(X-FORWARDED-FOR)%","user-agent":"%REQ(USER-AGENT)%","x-request-id":"%REQ(X-REQUEST-ID)%",":authority":"%REQ(:AUTHORITY)%","upstream_host":"%UPSTREAM_HOST%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","requested_server_name":"%REQUESTED_SERVER_NAME%","route_name":"%ROUTE_NAME%"}
                path: /dev/stdout
            address:
              socketAddress:
                address: 0.0.0.0
                portValue: 10080
            defaultFilterChain:
              filters:
              - name: envoy.filters.network.http_connection_manager
                typedConfig:
                  '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                  accessLog:
                  - name: envoy.access_loggers.file
                    typedConfig:
                      '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                      logFormat:
                        textFormatSource:
                          inlineString: |
                            {"start_time":"%START_TIME%","method":"%REQ(:METHOD)%","x-envoy-origin-path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","response_code_details":"%RESPONSE_CODE_DETAILS%","connection_termination_details":"%CONNECTION_TERMINATION_DETAILS%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","duration":"%DURATION%","x-envoy-upstream-service-time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","x-forwarded-for":"%REQ(X-FORWARDED-FOR)%","user-agent":"%REQ(USER-AGENT)%","x-request-id":"%REQ(X-REQUEST-ID)%",":authority":"%REQ(:AUTHORITY)%","upstream_host":"%UPSTREAM_HOST%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","requested_server_name":"%REQUESTED_SERVER_NAME%","route_name":"%ROUTE_NAME%"}
                      path: /dev/stdout
                  commonHttpProtocolOptions:
                    headersWithUnderscoresAction: REJECT_REQUEST
                  http2ProtocolOptions:
                    initialConnectionWindowSize: 1048576
                    initialStreamWindowSize: 65536
                    maxConcurrentStreams: 100
                  httpFilters:
                  - name: envoy.filters.http.router
                    typedConfig:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                      suppressEnvoyHeaders: true
                  mergeSlashes: true
                  normalizePath: true
                  pathWithEscapedSlashesAction: UNESCAPE_AND_REDIRECT
                  rds:
                    configSource:
                      ads: {}
                      resourceApiVersion: V3
                    routeConfigName: sealos-system/eg/http
                  serverHeaderTransformation: PASS_THROUGH
                  statPrefix: http
                  useRemoteAddress: true
            drainType: MODIFY_ONLY
            name: sealos-system/eg/http
            perConnectionBufferLimitBytes: 32768
    - '@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
      dynamicRouteConfigs:
      - routeConfig:
          '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
          ignorePortInHostMatching: true
          name: sealos-system/eg/http
          virtualHosts:
          - domains:
            - ntjxuedx.192.168.0.15.nip.io
            name: sealos-system/eg/http/ntjxuedx_192_168_0_15_nip_io
            routes:
            - match:
                prefix: /
              name: httproute/ns-oehe/bdkzlmibsivuiqav/rule/0/match/0/ntjxuedx_192_168_0_15_nip_io
              route:
                cluster: httproute/ns-oehe/bdkzlmibsivuiqav/rule/0
                upgradeConfigs:
                - upgradeType: websocket
          - domains:
            - qccbahgo.qccbahgo
            name: ns-vrpg/mfqjpuycbgjrtdww/http/qccbahgo_qccbahgo
            routes:
            - match:
                prefix: /
              name: httproute/ns-vrpg/mfqjpuycbgjrtdww/rule/0/match/0/qccbahgo_qccbahgo
              route:
                cluster: httproute/ns-vrpg/mfqjpuycbgjrtdww/rule/0
                upgradeConfigs:
                - upgradeType: websocket

Secondly, here is the xds with CORS applied only to the first Gateway sealos-system/eg. You can see that it adds an httpFilter to the Listener and a filter to the RouteConfiguration.

xds:
  eg:
    configs:
    ...
    - '@type': type.googleapis.com/envoy.admin.v3.ListenersConfigDump
      dynamicListeners:
      - activeState:
          listener:
            '@type': type.googleapis.com/envoy.config.listener.v3.Listener
            accessLog:
            - filter:
                responseFlagFilter:
                  flags:
                  - NR
              name: envoy.access_loggers.file
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                logFormat:
                  textFormatSource:
                    inlineString: |
                      {"start_time":"%START_TIME%","method":"%REQ(:METHOD)%","x-envoy-origin-path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","response_code_details":"%RESPONSE_CODE_DETAILS%","connection_termination_details":"%CONNECTION_TERMINATION_DETAILS%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","duration":"%DURATION%","x-envoy-upstream-service-time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","x-forwarded-for":"%REQ(X-FORWARDED-FOR)%","user-agent":"%REQ(USER-AGENT)%","x-request-id":"%REQ(X-REQUEST-ID)%",":authority":"%REQ(:AUTHORITY)%","upstream_host":"%UPSTREAM_HOST%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","requested_server_name":"%REQUESTED_SERVER_NAME%","route_name":"%ROUTE_NAME%"}
                path: /dev/stdout
            address:
              socketAddress:
                address: 0.0.0.0
                portValue: 10080
            defaultFilterChain:
              filters:
              - name: envoy.filters.network.http_connection_manager
                typedConfig:
                  '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                  accessLog:
                  - name: envoy.access_loggers.file
                    typedConfig:
                      '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                      logFormat:
                        textFormatSource:
                          inlineString: |
                            {"start_time":"%START_TIME%","method":"%REQ(:METHOD)%","x-envoy-origin-path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","response_code_details":"%RESPONSE_CODE_DETAILS%","connection_termination_details":"%CONNECTION_TERMINATION_DETAILS%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","duration":"%DURATION%","x-envoy-upstream-service-time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","x-forwarded-for":"%REQ(X-FORWARDED-FOR)%","user-agent":"%REQ(USER-AGENT)%","x-request-id":"%REQ(X-REQUEST-ID)%",":authority":"%REQ(:AUTHORITY)%","upstream_host":"%UPSTREAM_HOST%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","requested_server_name":"%REQUESTED_SERVER_NAME%","route_name":"%ROUTE_NAME%"}
                      path: /dev/stdout
                  commonHttpProtocolOptions:
                    headersWithUnderscoresAction: REJECT_REQUEST
                  http2ProtocolOptions:
                    initialConnectionWindowSize: 1048576
                    initialStreamWindowSize: 65536
                    maxConcurrentStreams: 100
                  httpFilters:
                  - name: envoy.filters.http.cors
                    typedConfig:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
                  - name: envoy.filters.http.router
                    typedConfig:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                      suppressEnvoyHeaders: true
                  mergeSlashes: true
                  normalizePath: true
                  pathWithEscapedSlashesAction: UNESCAPE_AND_REDIRECT
                  rds:
                    configSource:
                      ads: {}
                      resourceApiVersion: V3
                    routeConfigName: sealos-system/eg/http
                  serverHeaderTransformation: PASS_THROUGH
                  statPrefix: http
                  useRemoteAddress: true
            drainType: MODIFY_ONLY
            name: sealos-system/eg/http
            perConnectionBufferLimitBytes: 32768
    - '@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
      dynamicRouteConfigs:
      - routeConfig:
          '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
          ignorePortInHostMatching: true
          name: sealos-system/eg/http
          virtualHosts:
          - domains:
            - ntjxuedx.192.168.0.15.nip.io
            name: sealos-system/eg/http/ntjxuedx_192_168_0_15_nip_io
            routes:
            - match:
                prefix: /
              name: httproute/ns-oehe/bdkzlmibsivuiqav/rule/0/match/0/ntjxuedx_192_168_0_15_nip_io
              route:
                cluster: httproute/ns-oehe/bdkzlmibsivuiqav/rule/0
                upgradeConfigs:
                - upgradeType: websocket
              typedPerFilterConfig:
                envoy.filters.http.cors:
                  '@type': type.googleapis.com/envoy.extensions.filters.http.cors.v3.CorsPolicy
                  allowCredentials: true
                  allowMethods: PUT, GET, POST, DELETE, PATCH, OPTIONS
                  allowOriginStringMatch:
                  - safeRegex:
                      regex: http://.*\.foo\.com
                  maxAge: "600"
          - domains:
            - qccbahgo.qccbahgo
            name: ns-vrpg/mfqjpuycbgjrtdww/http/qccbahgo_qccbahgo
            routes:
            - match:
                prefix: /
              name: httproute/ns-vrpg/mfqjpuycbgjrtdww/rule/0/match/0/qccbahgo_qccbahgo
              route:
                cluster: httproute/ns-vrpg/mfqjpuycbgjrtdww/rule/0
                upgradeConfigs:
                - upgradeType: websocket

Finally, here is the xds with CORS applied only to the second Gateway ns-vrpg/mfqjpuycbgjrtdww. You can observe that it only adds a filter to the RouteConfiguration.

xds:
  eg:
    configs:
    ...
    - '@type': type.googleapis.com/envoy.admin.v3.ListenersConfigDump
      dynamicListeners:
      - activeState:
          listener:
            '@type': type.googleapis.com/envoy.config.listener.v3.Listener
            accessLog:
            - filter:
                responseFlagFilter:
                  flags:
                  - NR
              name: envoy.access_loggers.file
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                logFormat:
                  textFormatSource:
                    inlineString: |
                      {"start_time":"%START_TIME%","method":"%REQ(:METHOD)%","x-envoy-origin-path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","response_code_details":"%RESPONSE_CODE_DETAILS%","connection_termination_details":"%CONNECTION_TERMINATION_DETAILS%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","duration":"%DURATION%","x-envoy-upstream-service-time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","x-forwarded-for":"%REQ(X-FORWARDED-FOR)%","user-agent":"%REQ(USER-AGENT)%","x-request-id":"%REQ(X-REQUEST-ID)%",":authority":"%REQ(:AUTHORITY)%","upstream_host":"%UPSTREAM_HOST%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","requested_server_name":"%REQUESTED_SERVER_NAME%","route_name":"%ROUTE_NAME%"}
                path: /dev/stdout
            address:
              socketAddress:
                address: 0.0.0.0
                portValue: 10080
            defaultFilterChain:
              filters:
              - name: envoy.filters.network.http_connection_manager
                typedConfig:
                  '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                  accessLog:
                  - name: envoy.access_loggers.file
                    typedConfig:
                      '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                      logFormat:
                        textFormatSource:
                          inlineString: |
                            {"start_time":"%START_TIME%","method":"%REQ(:METHOD)%","x-envoy-origin-path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","response_code_details":"%RESPONSE_CODE_DETAILS%","connection_termination_details":"%CONNECTION_TERMINATION_DETAILS%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","duration":"%DURATION%","x-envoy-upstream-service-time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","x-forwarded-for":"%REQ(X-FORWARDED-FOR)%","user-agent":"%REQ(USER-AGENT)%","x-request-id":"%REQ(X-REQUEST-ID)%",":authority":"%REQ(:AUTHORITY)%","upstream_host":"%UPSTREAM_HOST%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","requested_server_name":"%REQUESTED_SERVER_NAME%","route_name":"%ROUTE_NAME%"}
                      path: /dev/stdout
                  commonHttpProtocolOptions:
                    headersWithUnderscoresAction: REJECT_REQUEST
                  http2ProtocolOptions:
                    initialConnectionWindowSize: 1048576
                    initialStreamWindowSize: 65536
                    maxConcurrentStreams: 100
                  httpFilters:
                  - name: envoy.filters.http.router
                    typedConfig:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                      suppressEnvoyHeaders: true
                  mergeSlashes: true
                  normalizePath: true
                  pathWithEscapedSlashesAction: UNESCAPE_AND_REDIRECT
                  rds:
                    configSource:
                      ads: {}
                      resourceApiVersion: V3
                    routeConfigName: sealos-system/eg/http
                  serverHeaderTransformation: PASS_THROUGH
                  statPrefix: http
                  useRemoteAddress: true
            drainType: MODIFY_ONLY
            name: sealos-system/eg/http
            perConnectionBufferLimitBytes: 32768
    - '@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
      dynamicRouteConfigs:
      - routeConfig:
          '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
          ignorePortInHostMatching: true
          name: sealos-system/eg/http
          virtualHosts:
          - domains:
            - ntjxuedx.192.168.0.15.nip.io
            name: sealos-system/eg/http/ntjxuedx_192_168_0_15_nip_io
            routes:
            - match:
                prefix: /
              name: httproute/ns-oehe/bdkzlmibsivuiqav/rule/0/match/0/ntjxuedx_192_168_0_15_nip_io
              route:
                cluster: httproute/ns-oehe/bdkzlmibsivuiqav/rule/0
                upgradeConfigs:
                - upgradeType: websocket
          - domains:
            - qccbahgo.qccbahgo
            name: ns-vrpg/mfqjpuycbgjrtdww/http/qccbahgo_qccbahgo
            routes:
            - match:
                prefix: /
              name: httproute/ns-vrpg/mfqjpuycbgjrtdww/rule/0/match/0/qccbahgo_qccbahgo
              route:
                cluster: httproute/ns-vrpg/mfqjpuycbgjrtdww/rule/0
                upgradeConfigs:
                - upgradeType: websocket
              typedPerFilterConfig:
                envoy.filters.http.cors:
                  '@type': type.googleapis.com/envoy.extensions.filters.http.cors.v3.CorsPolicy
                  allowCredentials: true
                  allowMethods: PUT, GET, POST, DELETE, PATCH, OPTIONS
                  allowOriginStringMatch:
                  - safeRegex:
                      regex: http://.*\.foo\.com
                  maxAge: "600"

[arkodg]

thanks for sharing the IR and XDS config @NamiKazu, this is super useful
I have a better understanding of the bug now -

• for http listeners on the same port, we reuse the filter chain
• this is fine, but we set addFilterChain to false
• but the logic to add http filters patchHCMWithFilters

  gateway/internal/xds/translator/listener.go

  Line 257 in 5ca939e

  // Add HTTP filters to the HCM, the filters have already been sorted in the

  lives inside addXdsHTTPFilterChain

Looks like the solution here is to patchHCMWithFilters w/o having to create a new filter chain for the case where there are two non https / http IR listeners for the same port

[liorokman]

@arkodg @NamiKazu If the gateways are using TLS, then different filterChains will be configured anyways and the scenario will work. This bug is only triggered if all of the gateways being used are non-TLS-enabled.

The problem is that if the gateways are not TLS-enabled, then it's not possible to create different filters because without SNI, there's no filter-matcher supported by Envoy on the network level that will work in this case.

The problem is bigger than CORS, where a filter needs to be added on the listener level. Take a look at other things done by various policies that affect the HCM configuration itself. For example, the same issue would happen if two non-TLS gateways being merged had different configuration for:

• OriginalIPDetection
• Trailers
• XffNumTrustedHops
• MergeSlashes
• Tracing

[arkodg]

ah yah @liorokman thanks for clarifying, looks like this issue is not specific to mergeGateways, but will also hold true if two different ClientTrafficPolicies apply different policies to different listeners with protocol http on the same port.
Im not sure moving away from the defaultFilterChain solution to a new filterChain with a new hcm will work, i.e. how will filterChain matching work

[arkodg]

sidenote - if we can't fix it for v1.0.0, we should flag this limitation as a issue in status for CTP, BTP and SP, since its a very specific case

[liorokman]

ah yah @liorokman thanks for clarifying, looks like this issue is not specific to mergeGateways, but will also hold true if two different ClientTrafficPolicies apply different policies to different listeners with protocol http on the same port.

If mergeGateways isn't turned on, each Gateway resource will mean a different Envoy Proxy and different IP address for reaching Envoy. There will be no issue.

ClientTrafficPolicies are only applicable on the Gateway level, so it shouldn't be possible to apply two different ClientTrafficPolicies that conflict if mergeGateways is turned off.

Im not sure moving away from the defaultFilterChain solution to a new filterChain with a new hcm will work, i.e. how will filterChain matching work

It doesn't. I tried. Envoy Proxy returns "response_code_details":"filter_chain_not_found" in its logs.

[NamiKazu]

Hi @arkodg @liorokman, I noticed that this issue has been closed, but after reading the corresponding PR and applying the case above, it appears that it hasn't actually addressed my problem. In the case described above, what I expected was for the policy to be successfully applied to the target gateway without affecting other gateways. However, the actual outcome is that the policy is simply being rejected.

I would like to know if this is the final solution, or just a temporary one?

If it's indeed final, it means that for one port, I can only declare one listener, otherwise I won't be able to apply any policy to it. However, what I'd like to emphasize is that in a multi-tenant scenario, each user implies a Gateway, and each Gateway implies a listener with a port. When dealing with tens of thousands of users, assigning each user a separate port is impractical, they should reuse ports. Or is there any other way to meet my requirements?

[arkodg]

@NamiKazu we've rejected the policy because we cannot support it in Envoy today, note this case is specific to different policies attaching to listeners with the same non https / http only port.
@liorokman did we implement first policy wins or no policy wins ? Also hoping its still possible to apply 1 policy to 2 listeners having the same http protocol and port

[liorokman]

@arkodg I implemented the "no policy wins" option. The scenario I was worried about was that there are 2 listeners on the same HTTP protocol and port, and 1 single policy attached to a single listener. Accepting that single policy means that both listeners are affected, even though only one of them should be affected.

It's not about a conflict between two policies, it's about a conflict that a single policy has with two abstract listeners that get mapped to a single concrete one in Envoy Proxy.

@NamiKazu You can have as many different policies as you want that target TLS enabled listeners, and as many policies that you want that target non-TLS listeners using different network ports.. The limitation is about a situation where you have more than one non-TLS listeners that share the same port. In this case, it's not technically possible to have even a single policy that attaches to one listener and ignores the other.

[arkodg]

thanks for clarifying @liorokman , just want to make sure that we continue to support the case where a single policy can continue to affect two http listeners on the same port, not supporting this, feels like a regression

[liorokman]

@arkodg You are right - we do have a regression here. The following should be possible but is not possible right now:

Setting a single ClientTrafficPolicy or a SecurityPolicy that targets a single Gateway, when that Gateway has more than one listener. The policy in this case is intended to apply the same settings to all of the listeners, and is valid, but it is currently being rejected.

If, instead of targeting the entire Gateway, the SecurityPolicy were to target a specific HTTPRoute then the correct behavior would have been to reject the SecurityPolicy since it's not possible to have different configuration for two routes connected to the same listener.

I think that this should be tracked as a separate issue though - WDYT?

[NamiKazu]

@liorokman @arkodg Actually, I have a simple idea, which is not to map two abstract listeners with the same port into one listener xds, but directly two listener xds. More specifically, We can change the following triplet (Address, Port‎, TCPKeepalive‎) into a quadruplet(Address, Port‎, TCPKeepalive‎, listenerName), where the format of the listener's name is namespace/gateway/listenerName).

gateway/internal/xds/translator/translator.go

Line 130 in 366e990

xdsListener := findXdsListenerByHostPort(tCtx, httpListener.Address, httpListener.Port, corev3.SocketAddress_TCP)

It aligns more closely with the intuition of cluster managers and users semantically, and also makes the HTTPListener more "Application Layer" logically. Additionally, it will resolve all the problems discussed earlier.

Moreover, when using envoyPatchPolicy, we actually meet the same problem as above. This is because when using envoyPatchPolicy, we directly modify xDS components such as listeners and RouteConfigurations. Two abstract listeners with the same port would also be mapped into one xDS. However, when declaring the listener to be modified, we attach only the abstract listenerName, which may no longer exist in reality.

This approach will also solve this problem with envoyPatchPolicy. It's quite simple, but note that due to the additional listener xDS, Envoy will consume more memory resources. I'm not sure if this is an appropriate solution. What do you think?

[liorokman]

@NamiKazu I'm not sure I follow. Consider the following required configuration:

1. A single port (e.g. 80) is used
2. No TLS is configured.
3. A single gateway has two listeners, which are differentiated by hostname, e.g. (foo.example.com and bar.example.com).
4. Each of these hostnames are mapped to different routes.

apiVersion: gateway.networking.k8s.io/v1beta1
    kind: Gateway
    metadata:
      name: gateway
    spec:
      gatewayClassName: envoy-gateway-class
      listeners:
        - name: http
          port: 80
          protocol: HTTP
          hostname: foo.example.com
          allowedRoutes:
            namespaces:
              from: Same
        - name: http2
          port: 80
          protocol: HTTP
          hostname: bar.example.com
          allowedRoutes:
            namespaces:
              from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
    name: httproute-1
spec:
      hostnames:
        - foo.example.com
      parentRefs:
        - namespace: default
          name: gateway
          sectionName: http
      rules:
        - matches:
            - path:
                value: "/"
          backendRefs:
            - name: service-1
              port: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
      name: httproute-2
spec:
      hostnames:
        - bar.example.com
      parentRefs:
        - namespace: default
          name: gateway-1
          sectionName: http-2
      rules:
        - matches:
            - path:
                value: "/"
          backendRefs:
            - name: service-2
              port: 8080
---

Envoy Proxy eventually needs to open a single port, and choose which filter chain to use in order to fulfil the configuration. If the required configuration is not TLS enabled, then the filter chain can be only be matched to a port.

The above maps to the following XDS configuration for the listener:

---
- address:
    socketAddress:
      address: 0.0.0.0
      portValue: 10080
  defaultFilterChain:
    filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        commonHttpProtocolOptions:
          headersWithUnderscoresAction: REJECT_REQUEST
        http2ProtocolOptions:
          initialConnectionWindowSize: 1048576
          initialStreamWindowSize: 65536
          maxConcurrentStreams: 100
        httpFilters:
        - name: envoy.filters.http.router
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            suppressEnvoyHeaders: true
        mergeSlashes: true
        normalizePath: true
        pathWithEscapedSlashesAction: UNESCAPE_AND_REDIRECT
        rds:
          configSource:
            ads: {}
            resourceApiVersion: V3
          routeConfigName: gateway/http
        serverHeaderTransformation: PASS_THROUGH
        statPrefix: http
        useRemoteAddress: true

All the routes above, from both listeners, map to the same defaultFilterChain because they are all non-tls and on the same port.

Add a policy (either SecurityPolicy or ClientTrafficPolicy or both) into the mix and attach it only to one section in the Gateway:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: ClientTrafficPolicy
  metadata:
    name: target-gateway
  spec:
    targetRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: gateway
      namespace: default
      sectionName: http
    timeout:
      http:
        requestReceivedTimeout: "5s"

How do you propose to resolve this into a working Envoy Proxy configuration, considering that the requestReceivedTimeout is configured on the HCM level:

1. If the defaultFilterChain is modified then it affects all the listeners defined in the gateway, and not just the one that was targeted by the ClientTrafficPolicy.
2. Adding an additional filter chain that matches the same port into the listener in the filterChains array would require a matcher in order to work, and the only matcher that is possible in the non-TLS case is the port. Adding a port matcher means that the defaultFilterChain becomes unusable, since all non-TLS traffic intended for the matched port would use the filter definition in the filterChains array.

How would you write the Envoy Proxy configuration to distinguish between two identically targeted filter chains in order to provide different HCM configurations?

[NamiKazu]

@liorokman What I mean is, just let different sections generate different independent listener xDS, so each has its own defaultFilterChain. When using policies, it only modifies the defaultFilterChain corresponding to the section(listener) , thus distinguishing between listeners.

In the example you provided, the xDS would become (this is not the actual result generated after I made code changes; I just simulated it).

---
listener:
  '@type': type.googleapis.com/envoy.config.listener.v3.Listener
  address:
    socketAddress:
      address: 0.0.0.0
      portValue: 10080
  defaultFilterChain:
    filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        commonHttpProtocolOptions:
          headersWithUnderscoresAction: REJECT_REQUEST
        http2ProtocolOptions:
          initialConnectionWindowSize: 1048576
          initialStreamWindowSize: 65536
          maxConcurrentStreams: 100
        httpFilters:
        - name: envoy.filters.http.router
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            suppressEnvoyHeaders: true
        mergeSlashes: true
        normalizePath: true
        pathWithEscapedSlashesAction: UNESCAPE_AND_REDIRECT
        rds:
          configSource:
            ads: {}
            resourceApiVersion: V3
          routeConfigName: gateway/http
        serverHeaderTransformation: PASS_THROUGH
        statPrefix: http
        useRemoteAddress: true
  name: default/gateway/http

listener:
  '@type': type.googleapis.com/envoy.config.listener.v3.Listener
  address:
    socketAddress:
      address: 0.0.0.0
      portValue: 10080
  defaultFilterChain:
    filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        commonHttpProtocolOptions:
          headersWithUnderscoresAction: REJECT_REQUEST
        http2ProtocolOptions:
          initialConnectionWindowSize: 1048576
          initialStreamWindowSize: 65536
          maxConcurrentStreams: 100
        httpFilters:
        - name: envoy.filters.http.router
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            suppressEnvoyHeaders: true
        mergeSlashes: true
        normalizePath: true
        pathWithEscapedSlashesAction: UNESCAPE_AND_REDIRECT
        rds:
          configSource:
            ads: {}
            resourceApiVersion: V3
          routeConfigName: gateway/http
        serverHeaderTransformation: PASS_THROUGH
        statPrefix: http
        useRemoteAddress: true
  name: default/gateway/http2

Just two listener xds with different name, the rest of the parts are identical. That's why I mentioned it as more "application-layer" because their characteristics at the transport layer are all the same.

However, based on your statement, I suspect that perhaps Envoy Proxy does not support this method of opening ports? (I should test it first, but whenever I try to test after modifying the code, it always pulls the image with wrong repo and correct tag. I may raise a new issue later to describe this problem.)

[liorokman]

I think this would not work, since both these listeners have the same socketAddress.

Let's say that it's possible to define this configuration. How does Envoy Proxy decide which listener should accept a new connection on this port? All it has to work with at that point in time is the port, and both these listeners define the same one.

I'd be happy to be wrong here :-)

[zhaohuabing]

Looks like I chime in this late :-).

I believe we can patchHCMWithFilters on an existing FilterChian. For shared configuration such as OriginalIPDetection, Trailers, XffNum, TrustedHops, MergeSlashes, Tracing, etc., I suggest we reject the policies with conflicting configuration. @arkodg @liorokman @NamiKazu