Cross-Namespace based configuration of an ext auth service for a Gateway

[dadrus]

Description

I'm trying to configure an external auth service by following the example from the docs (https://gateway.envoyproxy.io/v1.0.1/tasks/security/ext-auth/). My real setup differs however in some parts:

• my ext auth service is in a namespace different from the namespace of the Gateway resource.
• I need the configuration on the Gateway level and not on the HTTPRoute level. To have a global behavior (each inbound request handled by the particular gateway should be forwarded to the ext auth service regardless of the route)

While the latter can be achieved by the following SecurityPolicy

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: ext-auth-heimdall
  namespace: gw-ns
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: gw-ns
  extAuth:
    grpc:
      backendRef:
        name: heimdall
        port: 4456
        namespace: heimdall

it works only if the Gateway resource and the ext auth service are in the same namespace. If they are in different namespaces, as also shown in the snippet above, the status stanza of the security policy tells that the policy is invalid:

Status:
  Ancestors:
    Ancestor Ref:
      Group:      gateway.networking.k8s.io
      Kind:       Gateway
      Name:       eg
      Namespace:  gw-ns
    Conditions:
      Last Transition Time:  2024-05-01T23:36:01Z
      Message:               Backend ref to Service heimdall/heimdall not permitted by any ReferenceGrant
      Observed Generation:   1
      Reason:                Invalid
      Status:                False
      Type:                  Accepted
    Controller Name:         gateway.envoyproxy.io/gatewayclass-controller
Events:                      <none>

Similar reasoning message is also shown in the logs of the gateway controller

2024-05-01T23:23:05.374Z	INFO	provider	kubernetes/controller.go:450	no matching ReferenceGrants found	{"runner": "provider", "from": "HTTPRoute", "from namespace": "gw-ns", "target": "Service", "target namespace": "heimdall"}

Obviously, there is a need for a ReferenceGrant. If I create it (with the contents shown below), the controller logs, that it was able to add the reference grant, but it has no effect. The security policy has still the same status.

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: ReferenceGrant
metadata:
  name: ext-auth-heimdall-rg-hr
  namespace: heimdall
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: gw-ns
  to:
    - group: ""
      kind: Service
      name: heimdall

Actually, I would expect, the reference grant is required for the Gateway itself as there is no HTTPRoute resource in place "between" the gateway and the ext auth service. The configuration happens by making use of the extAuth property in the SecurityPolicy. So instead of kind: HTTPRoute , it should be kind: Gateway. Deploying a corresponding reference grant (with kind: Gateway) has however no effect as well.

Do I miss something? Is it maybe even a bug in the controller implementation? Is cross-namespace access for non xRoute ext auth services possible at all? Thank you in advance.

Additional Resources

This issue/challenge/observation has been initially asked in Slack: https://envoyproxy.slack.com/archives/C03E6NHLESV/p1714637933500289

The context of this issue is an integration guide I'm writing for a service I'm maintaining (https://github.com/dadrus/heimdall) and a corresponding example, which you can find under https://github.com/dadrus/heimdall/blob/8927261af52e0e3c2b6b6f35f0d0d8420c17815d/examples/kubernetes/Justfile#L218 (you'll need to build an image for heimdall and load it into the kind cluster if you would like to check it out: just build-image, followed by kind load docker-image heimdall:local -n demo-cluster when the cluster is started). As of now, the gateway and my ext auth server reside in the same namespace to overcome the limitations described above and to have something working. This is however not how this service is typically used in prod.

[arkodg]

thanks for raising this issue !

the decision that needs to be made here is should Cross Namespace Routing for non xRoute Backends ((SecurityPolicy)ext auth, (EnvoyProxy)otel, grpc ALS) be

1. opt in using a ReferenceGrant
2. enabled by default

I'd vote for 2. here, because the persona linking the backendRef is a privileged user (platform admin) and we are not dealing with the app dev persona here

@envoyproxy/gateway-maintainers wdyt ?

[zhaohuabing]

By default seems to make more sense here.

Opt in would cause a lot of confusion for users like this issue. Also, ext auth service is a global resource in a system, and it will certainly be referenced by multiple SecurityPolicy in different NSs. So do the otel, ALS, and other global resources.

[dadrus]

I just checked if it is possible to have cross-namespace reference for an HTTPRoute. Same result even if I add the required ReferenceGrant.

The setup is as follows: envoy and heimdall (ext auth service) run in the same namespace (named heimdall). An app is deployed in another namespace (named quickstarts). Here the corresponding Gateway, HTTPRoute, SecurityPolicy and the ReferenceGrant:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: heimdall
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      hostname: echo-app.local
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: eg-tls
            kind: Secret

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo-app
  namespace: quickstarts
spec:
  parentRefs:
    - name: eg
      namespace: heimdall
  hostnames:
    - "echo-app.local"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: echo-app
          port: 8080
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: ext-auth-heimdall
  namespace: quickstarts
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: echo-app
    namespace: quickstarts
  extAuth:
    grpc:
      backendRef:
        name: heimdall
        port: 4456
        namespace: heimdall

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: ReferenceGrant
metadata:
  name: ext-auth-heimdall-rg-hr
  namespace: heimdall
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: quickstarts
  to:
    - group: ""
      kind: Service
      name: heimdall

Given these observations (incl the initial one) I assume, same issue is present for the GRPCRoute as well. The message "Backend ref to Service heimdall/heimdall not permitted by any ReferenceGrant" in the SecurityPolicy status stanza is also really misleading as there is a corresponding ReferenceGrant (s.o.) in place.

It works if the Gateway, the ext auth service and the app (incl its HTTPRoute) are in the same namespace.

[guydc]

By default seems to make more sense here.

Opt in would cause a lot of confusion for users like this issue. Also, ext auth service is a global resource in a system, and it will certainly be referenced by multiple SecurityPolicy in different NSs. So do the otel, ALS, and other global resources.

Totally Agree, +1 to cross-NS.

[arkodg]

hey @adam-cattermole & @eguzki, thanks for fixing the bug highlighted in #3322 (comment)

for the kuadrant use case, could you share a little more about which persona creates the SecurityPolicy resource ?

It will help us decide on how to proceed here - opt in or opt out knob for requiring ReferenceGrant for cross ns for non xRoute cases

[adam-cattermole]

We have kind of a hybrid where we will have SecurityPolicy resources targetting either xRoutes or gateways:

• App Developer: The auth is configured at the xRoute level and in this case it makes sense for the ReferenceGrant to be required
• Platform Engineer: In this case the auth is configured at the gateway level, and I think we're in agreement that the option for enabled by default makes the most sense here

[guydc]

It would be nice to have cross-namespace support and reference-grant opt-out options for:

• Policy BackendRefs
• Policy TargetRefs

Maybe also:

• xRoutes BackendRefs

[arkodg]

closing this issue, as the bug has been fixed, im creating a new issue to track the work of opting in to not require ReferenceGrants for non xRoute backends