Duplicated xRoutes in gatewayapi.Resources when attaching xRoute to multiple Gateways #3165

shuji-2019 · 2024-04-10T10:40:45Z

Description:

When attaching single HTTPRoute to N Gateways, HTTPRoute is added to gatewayapi.Resources N times. It'll lead problems:

HTTP Listener 's attachedRoutes is N, not 1.

Duplicated ir.HTTPRoutes are added to ir.HTTPListener.

Duplicated Endpoints are added to ir.HTTPRoute.Destination. (Envoy will drop duplicated endpoint when handling EDS resource)

When attaching single TCPRoute to N. Gateway, TCPRoute is added to gatewayapi.Resources N times. It'll lead problems:

TCP Listener's attachedRoutes is N, not 1, then Listener will be treated as unaccepted. (EG only allow TCP Listener whose attachedRoutes is 1)

Repro steps:

Do quickstart.

Duplicate the only Gateway in quickstart and increase port by 1.

Attach the new Gateway to the only HTTPRoute in quickstart.

Apply the new Gateway and the new HTTPRoute.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-81
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 81

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Environment:

EG: v1.0.1

Logs:

status:
  conditions:
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute

status:
  conditions:
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute

2024-04-10T09:08:21.351Z	INFO	gateway-api	runner/runner.go:56	received an update	{"runner": "gateway-api"}
2024-04-10T09:08:21.351Z	INFO	gateway-api	runner/runner.go:103	proxy:
  listeners:
  - address: null
    name: default/eg-81/http
    ports:
    - containerPort: 10081
      name: http
      protocol: HTTP
      servicePort: 81
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: eg-81
      gateway.envoyproxy.io/owning-gateway-namespace: default
  name: default/eg-81
	{"runner": "gateway-api", "infra-ir": "default/eg-81"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:103	proxy:
  listeners:
  - address: null
    name: default/eg/http
    ports:
    - containerPort: 10080
      name: http
      protocol: HTTP
      servicePort: 80
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: eg
      gateway.envoyproxy.io/owning-gateway-namespace: default
  name: default/eg
	{"runner": "gateway-api", "infra-ir": "default/eg"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:114	accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: default/eg/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 10080
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
	{"runner": "gateway-api", "xds-ir": "default/eg"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:114	accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: default/eg-81/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 10081
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
	{"runner": "gateway-api", "xds-ir": "default/eg-81"}

The text was updated successfully, but these errors were encountered:

arkodg · 2024-04-12T10:51:29Z

from a first glance, can't understand the issue, can you elaborate @shuji-2019 ?

shuji-2019 · 2024-04-12T11:28:53Z

Follow Quickstart ¹, this will apply Gateway and HTTPRoute showed below, everything works as Gateway API spec defined, Listener's attachedRoutes is 1.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Then another Gateway is applied.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-81
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 81

Update HTTPRoute in Quickstart to attach Gateway eg-81 to it.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

At this time, each Listener is attached by ONLY 1 route. But Listener's attachedRoutes in Gateway Status is wrong as log listed before,
- Gateway eg Status show its Listener http has attached 2 Routes.
- Gateway eg-81 Status show its Listener http has attached 2 Routes.

Root cause is HTTPRoute is stored in gatewayapi.Resources ² as slice, not map.

When process Gateway eg, all HTTPRoutes is iterated and EG find HTTPRoute backend is attaching to this Gateway, HTTPRoute backend is appended to slice.

When process Gateway eg-81, HTTPRoute backend is happened to slice again, by the same logic.

Then gatewayapi module will translate same HTTPRoute twice, leading to problems described in the initial comment.

chr-fritz · 2024-04-17T21:13:35Z

Hi,

I want to add a different aspect about this problem. For some external reasons I need two differnet gateways but all HttpRoutes are should be on both gateways. So I decided not to duplicate them and add both gateways as spec.parentRefs to my HttpRoutes.

My expecation is that the HttpRoutes are registered on every gateway that is refrenced within spec.parentRefs. But the status of my HttpRoute shows that the routes are only registered on the gateway which is the last one in the spec.parentRefs list:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
status:
  parents:
    - conditions:
        - lastTransitionTime: "2024-04-17T20:43:21Z"
          message: Route is accepted
          observedGeneration: 1
          reason: Accepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2024-04-17T20:43:21Z"
          message: Resolved all the Object references for the Route
          observedGeneration: 1
          reason: ResolvedRefs
          status: "True"
          type: ResolvedRefs
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: eg-81
        sectionName: http

If I try to access the route about the first listed gateway, the access is not possible and I get http status 404 not found. The try it on the gateway which is the last within spec.parentRefs is successful.

aoledk · 2024-04-23T11:48:12Z

@chr-fritz I've reproduced steps listed above with eg v1.0.1 and K8s v1.26.14, eg and eg-81 can be accessed and return 200 OK. The complete HTTPRoute including status is:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  creationTimestamp: "2024-04-23T10:25:31Z"
  generation: 2
  name: backend
  namespace: default
  resourceVersion: "5539707"
spec:
  hostnames:
  - www.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg-81
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: backend
      port: 3000
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Route is accepted
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 2
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  - conditions:
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Route is accepted
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 2
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-81

aoledk · 2024-04-25T11:01:03Z

@arkodg I want to pick up this issue opened by myself. (shuji-2019 is me too)

aoledk · 2024-04-29T10:01:23Z

/assign

shuji-2019 added the triage label Apr 10, 2024

arkodg added this to the v1.1.0-rc1 milestone Apr 18, 2024

arkodg added the help wanted Extra attention is needed label Apr 18, 2024

aoledk mentioned this issue Apr 26, 2024

fix: duplicated xroutes are added to gatewayapi.Resources #3282

Merged

github-actions bot assigned aoledk Apr 29, 2024

arkodg closed this as completed in #3282 May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Duplicated xRoutes in gatewayapi.Resources when attaching xRoute to multiple Gateways #3165

Duplicated xRoutes in gatewayapi.Resources when attaching xRoute to multiple Gateways #3165

shuji-2019 commented Apr 10, 2024

arkodg commented Apr 12, 2024

shuji-2019 commented Apr 12, 2024

chr-fritz commented Apr 17, 2024

aoledk commented Apr 23, 2024

aoledk commented Apr 25, 2024

aoledk commented Apr 29, 2024

Duplicated xRoutes in gatewayapi.Resources when attaching xRoute to multiple Gateways #3165

Duplicated xRoutes in gatewayapi.Resources when attaching xRoute to multiple Gateways #3165

Comments

shuji-2019 commented Apr 10, 2024

arkodg commented Apr 12, 2024

shuji-2019 commented Apr 12, 2024

Footnotes

chr-fritz commented Apr 17, 2024

aoledk commented Apr 23, 2024

aoledk commented Apr 25, 2024

aoledk commented Apr 29, 2024