Listener is not created when there are multiple multiple listeners having same protocols and ports but different hostname

[mazzy89]

Description:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: infra-gateway
spec:
  gatewayClassName: infra-gateway
  listeners:
    - name: grpc-https
      hostname: 'grpc.example.com'
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: infra-gateway-grpc-tls
        mode: Terminate
    - name: wildcard-https
      hostname: '*.example.com'
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: infra-gateway-wildcard-tls

Having the above configuration should create two different listeners. However it create only one listener grpc-https.

Repro steps:

Apply the above configuration and then run egctl and look for the wildcard listener. It won't be in place.

Environment:

envoy-gateway v0.6.0

A configuration like that it should be possible as recommended by cert-manager https://cert-manager.io/docs/usage/gateway/#two-listeners-with-the-same-secret-name

Also from upstream config https://gateway-api.sigs.k8s.io/v1alpha2/guides/tls/ it should be possible.

[tanujd11]

/assign

[arkodg]

hey @tanujd11 checking in to see if you still plan on working on this ?

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

hey @tanujd11 still planning on working on this one ?

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[cnvergence]

I will take a look

[JuniorJPDJ]

I'm having similar issue, but in my case problem is multiple TLS listeners with other mode and hostnames:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    argocd.argoproj.io/instance: cluster-config
  name: default
  namespace: envoy-gateway-system
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      name: http-0
      port: 80
      protocol: HTTP
    - allowedRoutes:
        namespaces:
          from: All
      name: tls-pass-0
      port: 443
      protocol: TLS
      tls:
        mode: Passthrough
    - allowedRoutes:
        namespaces:
          from: All
      hostname: '*.p.<redacted>'
      name: https-0
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - group: ''
            kind: Secret
            name: <redacted>
        mode: Terminate
    - allowedRoutes:
        namespaces:
          from: All
      hostname: ssh.<redacted>
      name: tls-term-0
      port: 443
      protocol: TLS
      tls:
        certificateRefs:
          - group: ''
            kind: Secret
            name: <redacted>
        mode: Terminate

tls-pass-0 is accepted, https-0 is accepted and tls-term-0 is not accepted, probably when adding more tls-terms or even httpses those won't be accepted too even with other hostnames

Status:

status:
  addresses:
    - type: IPAddress
      value: 192.168.64.209
  conditions:
    - lastTransitionTime: '2024-03-28T11:32:57Z'
      message: The Gateway has been scheduled by Envoy Gateway
      observedGeneration: 5
      reason: Accepted
      status: 'True'
      type: Accepted
    - lastTransitionTime: '2024-03-28T11:32:57Z'
      message: 'Address assigned to the Gateway, 1/1 envoy Deployment replicas available'
      observedGeneration: 5
      reason: Programmed
      status: 'True'
      type: Programmed
  listeners:
    - attachedRoutes: 3
      conditions:
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Sending translated listener configuration to the data plane
          observedGeneration: 5
          reason: Programmed
          status: 'True'
          type: Programmed
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Listener has been successfully translated
          observedGeneration: 5
          reason: Accepted
          status: 'True'
          type: Accepted
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Listener references have been resolved
          observedGeneration: 5
          reason: ResolvedRefs
          status: 'True'
          type: ResolvedRefs
      name: http-0
      supportedKinds:
        - group: gateway.networking.k8s.io
          kind: HTTPRoute
        - group: gateway.networking.k8s.io
          kind: GRPCRoute
    - attachedRoutes: 3
      conditions:
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Sending translated listener configuration to the data plane
          observedGeneration: 5
          reason: Programmed
          status: 'True'
          type: Programmed
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Listener has been successfully translated
          observedGeneration: 5
          reason: Accepted
          status: 'True'
          type: Accepted
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Listener references have been resolved
          observedGeneration: 5
          reason: ResolvedRefs
          status: 'True'
          type: ResolvedRefs
      name: tls-pass-0
      supportedKinds:
        - group: gateway.networking.k8s.io
          kind: TLSRoute
    - attachedRoutes: 0
      conditions:
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Sending translated listener configuration to the data plane
          observedGeneration: 5
          reason: Programmed
          status: 'True'
          type: Programmed
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Listener has been successfully translated
          observedGeneration: 5
          reason: Accepted
          status: 'True'
          type: Accepted
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Listener references have been resolved
          observedGeneration: 5
          reason: ResolvedRefs
          status: 'True'
          type: ResolvedRefs
      name: https-0
      supportedKinds:
        - group: gateway.networking.k8s.io
          kind: HTTPRoute
        - group: gateway.networking.k8s.io
          kind: GRPCRoute
    - attachedRoutes: 1
      conditions:
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Only one TCP/TLS listener is allowed in a given port
          observedGeneration: 5
          reason: ProtocolConflict
          status: 'True'
          type: Conflicted
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: 'Listener is invalid, see other Conditions for details.'
          observedGeneration: 5
          reason: Invalid
          status: 'False'
          type: Programmed
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Listener references have been resolved
          observedGeneration: 5
          reason: ResolvedRefs
          status: 'True'
          type: ResolvedRefs
      name: tls-term-0
      supportedKinds:
        - group: gateway.networking.k8s.io
          kind: TCPRoute

[cnvergence]

Seems like the same issue like here: #2964
I will have some time now to debug this

[cnvergence]

I have tested it against the gateway with multiple listeners

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: infra-gateway
spec:
  gatewayClassName: eg-internal
  listeners:
    - name: company-https
      hostname: '*.company.com'
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        certificateRefs:
        - group: ""
          kind: Secret
          name: company-tls
          namespace: envoy-gateway-system
        mode: Terminate
    - name: example-https
      hostname: www.example.com
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: example-tls
            namespace: envoy-gateway-system
        mode: Terminate

Without merged gateways, that has another bug related to infra provisioning, I could successfully test out this gateway with two different services. I have checked that XDS IR is translated properly to match different apps.

Company listener:

 curl -v -HHost:'*.company.com' --resolve "sub.company.com:443:172.18.255.200" \                                                                                      
--cacert rsa-cert-wildcard.pem  https://'sub.company.com'/get

* Added sub.company.com:443:172.18.255.200 to DNS cache
* Hostname sub.company.com was found in DNS cache
*   Trying 172.18.255.200:443...
* Connected to sub.company.com (172.18.255.200) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: rsa-cert-wildcard.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=Test Inc
*  start date: Mar 19 16:04:47 2024 GMT
*  expire date: Mar 19 16:04:47 2025 GMT
*  subjectAltName: host "sub.company.com" matched cert's "*.company.com"
*  issuer: CN=Test Inc
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://sub.company.com/get
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: *.company.com]
* [HTTP/2] [1] [:path: /get]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET /get HTTP/2
> Host:*.company.com
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/2 200 
< content-type: application/json
< x-content-type-options: nosniff
< date: Fri, 29 Mar 2024 15:13:34 GMT
< content-length: 474
< 
{
 "path": "/get",
 "host": "*.company.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/8.4.0"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "172.18.0.3"
  ],
  "X-Forwarded-Proto": [
   "https"
  ],
  "X-Request-Id": [
   "da1adb99-00a7-401e-bc19-be3d0a1f5203"
  ]
 },
 "namespace": "envoy-gateway-system",
 "ingress": "",
 "service": "",
 "pod": "company-service-78d6759b89-kjp67"
* Connection #0 to host sub.company.com left intact
}

Example listener:

curl -v -HHost:www.example.com --resolve "www.example.com:443:172.18.255.200" \                                                                                       ✔  16:13:34  
--cacert www.example.com.crt https://www.example.com/get
* Added www.example.com:443:172.18.255.200 to DNS cache
* Hostname www.example.com was found in DNS cache
*   Trying 172.18.255.200:443...
* Connected to www.example.com (172.18.255.200) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: www.example.com.crt
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=www.example.com; O=example organization
*  start date: Mar 19 16:13:34 2024 GMT
*  expire date: Mar 19 16:13:34 2025 GMT
*  common name: www.example.com (matched)
*  issuer: O=example Inc.; CN=example.com
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.example.com/get
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.example.com]
* [HTTP/2] [1] [:path: /get]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET /get HTTP/2
> Host:www.example.com
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/2 200 
< content-type: application/json
< x-content-type-options: nosniff
< date: Fri, 29 Mar 2024 15:14:23 GMT
< content-length: 476
< 
{
 "path": "/get",
 "host": "www.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/8.4.0"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "172.18.0.3"
  ],
  "X-Forwarded-Proto": [
   "https"
  ],
  "X-Request-Id": [
   "47eea4cf-afb7-4336-9088-5998b8ab3ddc"
  ]
 },
 "namespace": "envoy-gateway-system",
 "ingress": "",
 "service": "",
 "pod": "example-service-55694dc865-wvn6c"
* Connection #0 to host www.example.com left intact
}

Please note that for the Infra IR there will be only one Infra IR listener for the multiple Gateway listeners configured with the same port value, as it is directly matched to k8s envoy infrastructure service port.
We are working on adding unified port naming for the Envoy infra and probably could consider to somehow extend Infra IR output to give more valuable information to the users on top, when dealing with such scenarios.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[JuniorJPDJ]

bump

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[JuniorJPDJ]

not stale

[arkodg]

hey @cnvergence still looking into this one ?

[cnvergence]

hey, I concluded my debugging here #1861 (comment)
I couldn't reproduce this issue, even with the same protocol and port but a different hostname, I could match proper backend service.
The outcome of it would be to find out if we can serve IR listeners on different ports in the service when they are duplicated

[arkodg]

there are 2 configs provided by users
1)

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: infra-gateway
spec:
  gatewayClassName: infra-gateway
  listeners:
    - name: grpc-https
      hostname: 'grpc.example.com'
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: infra-gateway-grpc-tls
        mode: Terminate
    - name: wildcard-https
      hostname: '*.example.com'
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: infra-gateway-wildcard-tls

and

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    argocd.argoproj.io/instance: cluster-config
  name: default
  namespace: envoy-gateway-system
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      name: http-0
      port: 80
      protocol: HTTP
    - allowedRoutes:
        namespaces:
          from: All
      name: tls-pass-0
      port: 443
      protocol: TLS
      tls:
        mode: Passthrough
    - allowedRoutes:
        namespaces:
          from: All
      hostname: '*.p.<redacted>'
      name: https-0
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - group: ''
            kind: Secret
            name: <redacted>
        mode: Terminate
    - allowedRoutes:
        namespaces:
          from: All
      hostname: ssh.<redacted>
      name: tls-term-0
      port: 443
      protocol: TLS
      tls:
        certificateRefs:
          - group: ''
            kind: Secret
            name: <redacted>
        mode: Terminate

hey @cnvergence are we unable to repro 1. or 2. or both ?

[cnvergence]

hey @arkodg, I tried both of them, like I said the traffic passed in my case properly to the correct backend.
But it might be that other folks want to have those listeners created as separate ports in the Kubernetes service, we don't support it currently, as we share ports (using 443 for all of them).
IMO, this is still something we could take a look at and create new ports on the service when there is a duplicate between listeners under the same gateway, I think Contour has something working for this case.

[arkodg]

@cnvergence did you try on v0.0.0-latest ? I think we made some refactors that improved/removed these constraints
#3271

      conditions:
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: Only one TCP/TLS listener is allowed in a given port
          observedGeneration: 5
          reason: ProtocolConflict
          status: 'True'
          type: Conflicted
        - lastTransitionTime: '2024-03-28T11:32:57Z'
          message: 'Listener is invalid, see other Conditions for details.'
          observedGeneration: 5
          reason: Invalid
          status: 'False'
          type: Programmed

@JuniorJPDJ can you try with v0.0.0-latest to confirm that the issue you are facing is fixed ? the v1.1 minor release should be out in the coming weeks

[cnvergence]

Oh, I did not see these in my example :)

[JuniorJPDJ]

@arkodg it looks like updating to latest version of v0.0.0-latest fixed the issue.
I haven't tried attaching any routes yet, but listener is accepted now.

[arkodg]

awesome thanks for confirming @JuniorJPDJ, there should be a stable image with the fix out soon