mark all extension API fields as experimental

[arkodg]

Description:

Mark all top level fields in all our extension APIs as experimental . This allows us to decouple graduating APIs from graduating API fields

[zirain]

Where would you like to see these markings? On doc site, crd annotations?

[arkodg]

On the docs API page

[zirain]

you can just update the comment, or what is the UX you expected?

[arkodg]

ideal UX would be a experimental string, that is hyperlinked into our API graduation/deprecation policy doc

[arkodg]

maybe we should step back a little, had a good discussion in the community meeting with @guydc and @kflynn, let me outline some options

Option 1 - All fields within a v1 API are stable and cannot be removed, only deprecated

Pros

• High velocity of feature additions
• No extra cognitive load for user - No extra user docs or install steps

Cons

• High Maintenance burden + complexity in case the fields are not well accepted by users, creating many deprecated fields with multiple variants existing in the code base for the same feature

Option 2 - Stable and Experimental Release Channel (different CRDs) for v1

Pros

• High velocity of feature additions, adding fields with experimental support into the experimental channel
• Fields can be removed from experimental, if they haven't graduated to stable keeping the API and code lean and simple

Cons

• Increased Cognitive load for user + more install knobs in helm to decide which CRDs to install

[arkodg]

ptal @envoyproxy/gateway-maintainers

[arkodg]

I prefer option 2 to keep it in line with upstream versioning https://gateway-api.sigs.k8s.io/concepts/versioning/
allows the user to even decide whether they want to install the stable or experiemental upstream APIs

[zirain]

either is fine to me.

[Xunzhuo]

@arkodg Option 2 is more reasonable for me. If a stable API frequently is marked as deprecated, the UX is bad for end-users. Instead, separating core and extended like what gwapi does, it reminds end-users that what is more stable to use and what is not. And if they accept to use the experimental ones, they should also accept the possibilities of API breaking changes.

[zirain]

@arkodg Option 2 is more reasonable for me. If a stable API frequently is marked as deprecated, the UX is bad for end-users. Instead, separating core and extended like what gwapi does, it reminds end-users that what is more stable to use and what is not. And if they accept to use the experimental ones, they should also accept the possibilities of API breaking changes.

stable does not mean that it cannot be deprecated, need a workflow to make sure it won't be panic too much.

[Xunzhuo]

Yes, I mean stable APIs should not be making breaking changes frequently.

[zirain]

but actually in kubernetes, we shouldn't promise anything in alpha.

[kflynn]

As discussed in the meeting today, ultimately, the most sane route forward is most likely to introduce stable and experimental release channels:

• Be cautious about taking CRDs to v1, since once you do that you’ll need to support it forever, with no reasonable way to remove a field or change its semantics
• Leave pre-v1 CRDs in the experimental channel
• Mark fields as “likely to change” or “considered very unlikely to change” even in experimental so that users know what to expect as CRDs move to stable
• Allow Envoy Gateway as a whole to go to v1 without taking the CRDs to v1
  • With Envoy Gateway v1, have the installation default to using the experimental CRDs; have an install option to use stable instead
    • We don’t want to start with stable as default because the BackendTrafficPolicy, ClientTrafficPolicy, and SecurityPolicy are very young and shouldn’t start out in stable
  • At some later point when we have more feedback and more confidence with the CRDs, switch the installation to default to stable instead

This is also being discussed in Gateway API itself, since we realized that we hadn't really written anything down about it. Some combination of Rob and I will be writing on it for upstream.

[arkodg]

thanks for writing this down @kflynn ! agree with all of these points except prefer the --experimental-crds helm knob to be disabled by default, so the helm defaults to stable CRDs. However if all our Quickstart and other installation guides, I'd prefer to incorporate --experimental-crds into the hem installation command and explain why to use it and why not to

[kflynn]

Yeah, I could get behind that, too. Just a question of whether we want to go with docs or conventions; I'm fine with either. 🙂

[zirain]

QQ: how can I move forward #2549?

[arkodg]

hey @zirain I have some WIP stashed locally to create standard and experimental CRDs in Helm Charts, will try and push that soon.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[zirain]

@arkodg still need this?

[arkodg]

lets close, not needed atm