NLB not get deleted after the service etc got deleted

[liyihuang]

Describe the bug
I used the envoy gateway with gateway API to create LoadBalancer type but found after deleting the envoy gateway all config the NLB from AWS is not deleted by the LB controllers

Steps to reproduce

1. setup the lb controller with eks irsa
2. setup the envoy gateway 0.6.0
3. configure envoy gateway
4. delete the envoy gateway config

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: custom-proxy-config
    namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          external-dns.alpha.kubernetes.io/hostname: abc.com
          
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: emojivotoroute
  namespace: emojivoto
spec:
  parentRefs:
    - name: eg
      namespace: default
  hostnames:
    - "abc.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: web-svc
          port: 80
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Expected outcome
I can see the the logs from LB controller saying the security group and targets are deleted but the NLB itself was not deleted from AWS. I expect that NLB got deleted by the LB controller

Environment

• AWS Load Balancer controller version helm 1.7.1 and I believe that's the latest from 2.7.1
• Kubernetes version 1.29
• Using EKS (yes/no), if so version? yes 1.29

Additional Context:

{"level":"info","ts":"2024-02-24T14:09:56Z","logger":"controllers.service","msg":"created targetGroup","stackID":"envoy-gateway-system/envoy-default-eg-e41e7b31","resourceID":"envoy-gateway-system/envoy-default-eg-e41e7b31:80","arn":"arn:aws:elasticloadbalancing:ca-central-1:679388779924:targetgroup/k8s-envoygat-envoydef-662a714b08/af0a72cf569098e8"}
{"level":"info","ts":"2024-02-24T14:09:56Z","logger":"controllers.service","msg":"creating loadBalancer","stackID":"envoy-gateway-system/envoy-default-eg-e41e7b31","resourceID":"LoadBalancer"}
{"level":"info","ts":"2024-02-24T14:09:57Z","logger":"controllers.service","msg":"created loadBalancer","stackID":"envoy-gateway-system/envoy-default-eg-e41e7b31","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:ca-central-1:679388779924:loadbalancer/net/k8s-envoygat-envoydef-d844852579/3f34d754ad6b8335"}
{"level":"info","ts":"2024-02-24T14:09:57Z","logger":"controllers.service","msg":"creating listener","stackID":"envoy-gateway-system/envoy-default-eg-e41e7b31","resourceID":"80"}
{"level":"info","ts":"2024-02-24T14:09:57Z","logger":"controllers.service","msg":"created listener","stackID":"envoy-gateway-system/envoy-default-eg-e41e7b31","resourceID":"80","arn":"arn:aws:elasticloadbalancing:ca-central-1:679388779924:listener/net/k8s-envoygat-envoydef-d844852579/3f34d754ad6b8335/46f8b73b1cf97a8e"}
{"level":"info","ts":"2024-02-24T14:09:57Z","logger":"controllers.service","msg":"creating targetGroupBinding","stackID":"envoy-gateway-system/envoy-default-eg-e41e7b31","resourceID":"envoy-gateway-system/envoy-default-eg-e41e7b31:80"}
{"level":"info","ts":"2024-02-24T14:09:57Z","logger":"controllers.service","msg":"created targetGroupBinding","stackID":"envoy-gateway-system/envoy-default-eg-e41e7b31","resourceID":"envoy-gateway-system/envoy-default-eg-e41e7b31:80","targetGroupBinding":{"namespace":"envoy-gateway-system","name":"k8s-envoygat-envoydef-662a714b08"}}
{"level":"info","ts":"2024-02-24T14:09:57Z","logger":"controllers.service","msg":"successfully deployed model","service":{"namespace":"envoy-gateway-system","name":"envoy-default-eg-e41e7b31"}}
{"level":"info","ts":"2024-02-24T14:09:59Z","msg":"authorizing securityGroup ingress","securityGroupID":"sg-0eeb8e99fb1fa92e7","permission":[{"FromPort":10080,"IpProtocol":"tcp","IpRanges":null,"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":10080,"UserIdGroupPairs":[{"Description":"elbv2.k8s.aws/targetGroupBinding=shared","GroupId":"sg-036e2ab103adf694a","GroupName":null,"PeeringStatus":null,"UserId":null,"VpcId":null,"VpcPeeringConnectionId":null}]}]}
{"level":"info","ts":"2024-02-24T14:09:59Z","msg":"authorized securityGroup ingress","securityGroupID":"sg-0eeb8e99fb1fa92e7"}
{"level":"info","ts":"2024-02-24T14:09:59Z","msg":"registering targets","arn":"arn:aws:elasticloadbalancing:ca-central-1:679388779924:targetgroup/k8s-envoygat-envoydef-662a714b08/af0a72cf569098e8","targets":[{"AvailabilityZone":null,"Id":"10.5.2.168","Port":10080}]}
{"level":"info","ts":"2024-02-24T14:10:00Z","msg":"registered targets","arn":"arn:aws:elasticloadbalancing:ca-central-1:679388779924:targetgroup/k8s-envoygat-envoydef-662a714b08/af0a72cf569098e8"}
{"level":"info","ts":"2024-02-24T14:25:37Z","msg":"deRegistering targets","arn":"arn:aws:elasticloadbalancing:ca-central-1:679388779924:targetgroup/k8s-envoygat-envoydef-662a714b08/af0a72cf569098e8","targets":[{"AvailabilityZone":"ca-central-1d","Id":"10.5.2.168","Port":10080}]}
{"level":"info","ts":"2024-02-24T14:25:37Z","msg":"deRegistered targets","arn":"arn:aws:elasticloadbalancing:ca-central-1:679388779924:targetgroup/k8s-envoygat-envoydef-662a714b08/af0a72cf569098e8"}
{"level":"info","ts":"2024-02-24T14:25:37Z","msg":"revoking securityGroup ingress","securityGroupID":"sg-0eeb8e99fb1fa92e7","permission":[{"FromPort":10080,"IpProtocol":"tcp","IpRanges":null,"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":10080,"UserIdGroupPairs":[{"Description":"elbv2.k8s.aws/targetGroupBinding=shared","GroupId":"sg-036e2ab103adf694a","GroupName":null,"PeeringStatus":null,"UserId":"679388779924","VpcId":null,"VpcPeeringConnectionId":null}]}]}
{"level":"info","ts":"2024-02-24T14:25:37Z","msg":"revoked securityGroup ingress","securityGroupID":"sg-0eeb8e99fb1fa92e7"}

no more logs to delete the LB controller but the NLB is created by this controller you can see from the logs

[M00nF1sh]

@liyihuang
How is the NLB created? there should be a Service object to be able to create a NLB. You need to delete that service object to clean up the NLB.

[liyihuang]

The service is generated by gatewayAPI's Gateway's controller based my understanding

you can see I have the service annotation in my EnvoyProxy

I have deleted all the Gateway and the service generated from Gateway from k8s itself is gone. but the controller didn't clean up the LB and SG in AWS.

I believe if you follow the envoy gateway project getting started(https://gateway.envoyproxy.io/v0.6.0/user/quickstart/) with envoyproxy change(https://gateway.envoyproxy.io/v0.6.0/user/customize-envoyproxy/#customize-envoyproxy-service-annotations) and you can reproduce this issue.

[liyihuang]

@M00nF1sh any chance to look at this?

[liyihuang]

envoyproxy/gateway#2939 cross posting here since I believe that's also something EKS LB controller can improve

[adrianmace]

I think this is also happening with Cilium's Gateway implementation. Same repro steps.

1. Install Cilium with Gateway API enabled and AWS LB Controller
2. Create Gateway object
3. Observe that the NLB and associated resources is created successfully
4. Delete the Gateway object
5. Observe that the NLB is still in the AWS console

[liyihuang]

Have found the issue is caused by envoy gateway.