api: wasm extension

[zhaohuabing]

This PR proposes an API for supporting Wasm plugins:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-extensions
spec:
  targetRef:
    kind: HTTPRoute
    name: backend
  wasm:
    - name: my_plugin_1
      code:
        image:  # the oci image that contains wasm code
          url: <oci-registry>.foo.bar.com/wasm-to-oci:v1 
          pullSecret: wasmPullSecret
        # http: https://foo.bar.com/example.wasm  # the HTTP URL containing the wasm code.
      sha256: xxxxxxx # SHA256 checksum that will be used to verify the wasm code.
      failOpen: true
      config:
        foo:
           foo1: value1
           foo2: value2
        bar: value3
      priority: 450   
    - name: my_plugin_2
      ...

[guydc]

Hi @zhaohuabing . We had some initial discussion here: #2546 on extensibility. The discussion was leaning towards a more common interface for all data plane extensions (golang, wasm, lua, ext-proc).

[Xunzhuo]

Also there is a proposal for containing all extensions by #2570, how do we deal with the relationship with that?

BTW, I prefer wasm for a high level API, since it is frequently used for extensions.

[guydc]

Which persona will be using this API? IMHO it's a bit dangerous to let application owners inject code into envoy, even if executed in isolation. If we see this as a policy for Gateway admins, should we support attachment to routes?

[zhaohuabing]

My apologies @guydc , I haven't noticed the existing discussion and the draft PR and didn't mean to add a competing PR. :-)

Before proposing the API, I considered two approaches to support data plane extensions (WASM, Lua, Golang, External Process, etc.)

• A fat EnvoyExtensionPolicy containing all the extensions.
• A set of EnvoyXExtensionPolicy, each supporting a specific extension.

The main problem of the first approach is that it would be difficult to support multiple extensions of the same type, e.g. users may need to define multiple WASM filters for different purposes for a single HTTPRoute. If we do support a list of each type of extensions in the fat EnvoyExtensionPolicy, that policy would become very large.

[zhaohuabing]

Which persona will be using this API? IMHO it's a bit dangerous to let application owners inject code into envoy, even if executed in isolation. If we see this as a policy for Gateway admins, should we support attachment to routes?

It's common that there are multiple applications behind the gateway, and they may need different wasm extensions, just like the other policies.

[guydc]

My apologies @guydc , I haven't noticed the existing discussion and the draft PR and didn't mean to add a competing PR. :-)

All good, I'm very happy that this topic is getting more attention and I hope that we can make this happen for v1.1.0.

Before proposing the API, I considered two approaches to support data plane extensions (WASM, Lua, Golang, External Process, etc.)

I agree that different extensions do have different structures, interfaces, concerns and use patterns. Some support per-route config while other don't, some are more modular and may repeat multiple time in the FC, while others are external and more likely to only be called-out once in the filter chain.

I'm very much open to different extension types being supported for different use cases if other maintainers also support this approach.

[zhaohuabing]

I agree that different extensions do have different structures, interfaces, concerns and use patterns. Some support per-route config while other don't, some are more modular and may repeat multiple time in the FC, while others are external and more likely to only be called-out once in the filter chain.

I'm very much open to different extension types being supported for different use cases if other maintainers also support this approach.

Let's bring this to the EG meeting and discuss further. Even we option to have multiple extension types, these types do need to address the common concerns raised in #2570 to ensure cohensive approach.

• Users may need to control the logical phase of filter execution (e.g. before auth, before rate limit, ... )
• Users may need to control the order of execution of extension filters amongst themselves (within a specific phase)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 64.51%. Comparing base (f699edf) to head (54b0eab).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2877   +/-   ##
=======================================
  Coverage   64.50%   64.51%           
=======================================
  Files         121      121           
  Lines       21381    21388    +7     
=======================================
+ Hits        13792    13798    +6     
- Misses       6718     6720    +2     
+ Partials      871      870    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zirain]

should remove it instead of comment.

[zhaohuabing]

I leave this because it's likely to be used when implementing wasm extension.

Of course, I'll remove it if I find out it's not necessary when implementing it.

[arkodg]

thanks !

[arkodg]

hey @zhaohuabing lets start off with HTTP support, and temporarily pause on the design for the Image feature based on the discussions from envoyproxy/envoy#33212