release v1beta4

[chendave]

(edited by neolit123)

general action items:

• Add v1beta4 boilerplate:
  kubeadm: add v1beta4 API boilerplate kubernetes#118762

• Add v1beta4 to kubeadm scheme but prevent the --config codepath from using it?
  Add --experimental flag for API migration/validate CLI commands:
  see kubeadm: add v1beta4 API boilerplate kubernetes#118762 (comment)
  kubeadm: add v1beta4 to scheme; add --allow-experimental-api flag kubernetes#118866

• restore TestMigrateOldConfigFromFile in common_test.go once v1beta4 is added. See how we can make this test optional if only one API version is available (e.g. in the future only v1beta4 is avaiable and v1beta3 is removed)
  kubeadm: restore and generalize the TestMigrateOldConfig test kubernetes#118941

• Make dryRun a config item in the config file in v1beta4 for reset/init/join/upgrade.
  kubeadm: Make dry-run is configurable in initConfiguration and joinConfiguration kubernetes#119279 - initConfiguration

• Loose the limitation to skip all in the init/join/... config file
  kubeadm: implementation of ResetConfiguration API types kubernetes#113583 (comment)
  kubeadm: remove the limitation that the 'ignorePreflightErrors' field can not be set to 'all' in kubeadm config file kubernetes#119351

• Deprecate dryRun, ignorePreflightErrors flags and others when they are supported by the all the configuration (reset, upgrade...)
  - kubeadm: Make dry-run is configurable in initConfiguration and joinConfiguration kubernetes#119279

• merge the flag set with the configuration ignorePreflightErrors set or just override the config when flag is used // to be decided.
  kubeadm: implementation of ResetConfiguration API types kubernetes#113583 (comment)

• Support the kubeadm config validation/migrate/print for the kubeamd reset config file, see: kubeadm: implementation of ResetConfiguration API types kubernetes#113583 (comment)
  kubeadm: add kubeadm config print reset-defaults command for ResetConfiguration kubernetes#119345
  kubeadm: Support kubeadm config validate for ResetConfiguration kubernetes#119319
  kubeadm: Support kubeadm config migrate for ResetConfiguration kubernetes#119313

• finalize v1beta4 tasks;
  • document the v1beta4 changes set and prepare a release note
  • move v1beta4 to the first element (highest priority) in api/kubeadm/scheme/scheme.go#AddToScheme()
  • support for v1beta4 'config migrate' and 'config validate'
• kubeadm: enable the v1beta4 API kubernetes#125029

• Start using v1beta4 in kinder e2e tests

• kinder: enable the v1beta4 API for >= 1.31 #3067

---

list of features:

• Support skipping addons image pull - need review / owner: @ruquanzhao
  skip image pull for addons that are skipped #2603
  kubeadm: Support skipping addons image pull  kubernetes#114534

• Implementation of UpgradeConfiguration API types - **need review / owner: @calvin0327 **
  add UpgradeConfiguration/ResetConfiguration API types #2489
  kubeadm: implementation of UpgradeConfiguration API types kubernetes#114062
  kubeadm: implementation of UpgradeConfiguration API types kubernetes#123068

• Implementation of ResetConfiguration API types - need review / owner: @chendave
  add UpgradeConfiguration/ResetConfiguration API types #2489
  kubeadm: implementation of ResetConfiguration API types kubernetes#113583

• Support custom env in kubeadm ControlPlaneComponent - implementable / owner: @chendave
  Support custom env in kubeadm ControlPlaneComponent #2845
  kubeadm: Support custom env in control plane component kubernetes#118867

• add API support for controlling various timeouts during init / join - implementable
  add API support for controlling various timeouts #2463
  [WIP][don't review] kubeadm: add Timeout structure and move TimeoutForControlPlane from APIServer into it kubernetes#103226
  kubeadm: add API support for controlling various timeouts during init / join kubernetes#119015
  kubeadm: add a Timeouts struct to v1beta4 kubernetes#122529
  implementable / owner: @neolit123

• handling of extraArgs which are allowed multiple times handling of extraArgs which are allowed multiple times #1601 - status: implementable / BREAKING change. / owner: @neolit123
  kubeadm add support for structured ExtraArgs kubernetes#119156
  kubeadm: enable the v1beta4 API kubernetes#125029 adds support for non-duplicate --*extra-args flags, but also deprecates them

• Allow overwrite KubeletRunDirectory when init/join Allow overwrite KubeletRunDirectory when init/join  #2104 - status: implementable / low priority / owner: @ruquanzhao

• graduate kubeadm patch support to GA graduate kubeadm patch support to GA #2046 - status: waiting for feedback DROPPED

• Allow usage of KubeletConfiguration next to JoinConfiguration on kubeadm join Allow usage of KubeletConfiguration next to JoinConfiguration on kubeadm join #1682 - status: waiting for feedback

• Switch to generating ECDSA keys Switch to generating ECDSA keys #1535 - status: implementable / need agreement on the approach / BREAKING change. / owner: @neolit123
  Switch to generating ECDSA keys #1535
  kubeadm: add v1beta4.ClusterConfiguration.EncryptionAlgorithm kubernetes#120417
  Provide options to use at least 3072-bits RSA to meet German federal BSI Technical Guideline TR-02102-2 #3003
  kubeadm: enhance encryption algorithm support in v1beta4 kubernetes#123054

• support parallel image pulls
  kubeadm add flag to enable parallel image pulling by default #2780 implementable / owner: @neolit123

• kubeadm: add support for parallel image pulls in v1beta4 kubernetes#122616 add the support in join and init
• support it in upgrade "node" and "apply"

There are already PR ready for the first three, others either need the further discussion or need someone's commitment to own.

---

BREAKING change

• handling of extraArgs which are allowed multiple times handling of extraArgs which are allowed multiple times #1601
  extraArgs in the ControlPlaneComponent is expected to be updated, it is currently map[string]string, it's going to be replaced by a more flexible type to allow duplicated key.

• Switch to generating ECDSA keys Switch to generating ECDSA keys #1535
  FG: PublicKeysECDSA will be dropped but a cluster level field (config item in ClusterConfiguration?) will be used to support the ECDSA key
  NOTE: this will just change the way how to support the ECDSA key, the API or the functionality will be same as before.

• some timeout fields are planned to be moved in v1beta4:
  add API support for controlling various timeouts #2463

---

Design is not clear at the moment

• Allow overwrite KubeletRunDirectory when init/join Allow overwrite KubeletRunDirectory when init/join  #2104
  This is not clear by far, @ruquanzhao is working on a POC and provide a design if necessary. DROPPED

[chendave]

ref: kubernetes/enhancements#970 (comment)

[chendave]

@neolit123 @SataQiu @pacoxu would like to hear your feedback on this. Is this list enough for v1beta4?

[chendave]

ref: kubernetes/kubernetes#114534 (comment)

[chendave]

/assign

[neolit123]

please include all labeled issues
kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API
(even if some are a bit vague / not actionable)

[neolit123]

i've added this to .29. i think if this api code work is to happen it should be at the start of a release cycle. but perhaps we can get all the planning done in .28. what do people think?

[chendave]

please include all labeled issues

I hope I haven't miss anything, the list is updated.

(even if some are a bit vague / not actionable)

Haven't got the chance to go through each of them, but I will evaluate each of them in the near future.

[chendave]

i think if this api code work is to happen it should be at the start of a release cycle. but perhaps we can get all the planning done in .28

+1.
we can also get some of them implemented within the time frame of .28 after the scope if finalized.

[neolit123]

i was thinking that if we want to work on an api for multiple releases, we could try this:

• add the new api boilerplate / clone v1beta3
• prevent the users from using it via cli / wip errors in a few key places
• add all new api changes
• remove the wip errors
• release the api

the tricky part would be getting kubeadm to agree on this wip status.

in this approach, a single release note must be used listing all changes when we are ready - i.e. all api changes must go without release notes.

reminder - we should not iterate new features over a released api, because it causes pain for consumers like capi. ideally, an api should be a released, locked feature set.

[chendave]

add the new api boilerplate / clone v1beta3
prevent the users from using it via cli / wip errors in a few key places

+1, I believe we can start this in .28 release, and part of features could be added in .28 as well, otherwise, it would be a little rush for .29 release.
We will definitely has some warning for v1beta4 before the official release.

[pacoxu]

reminder - we should not iterate new features over a released api, because it causes pain for consumers like capi. ideally, an api should be a released, locked feature set.

Should we include the CAPI team and other consumers in this topic to get more feedback?

[neolit123]

reminder - we should not iterate new features over a released api, because it causes pain for consumers like capi. ideally, an api should be a released, locked feature set.

Should we include the CAPI team and other consumers in this topic to get more feedback?

we could email the sig mailing list.

[chendave]

@neolit123 @pacoxu I am currently triage all those requirements, will send a summarization to the mailing list or join the sig community meeting for the head-ups and ask for review comments.

[ruquanzhao]

/cc

[SataQiu]

work on an api for multiple releases

We can introduce the v1beta4 API, but not register it into the Scheme, prevent users to use it.
https://github.com/kubernetes/kubernetes/blob/18d05b646d09b2971dc5400bc288062b0414e8cf/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go#L42-L46

[chendave]

We can introduce the v1beta4 API, but not register it into the Scheme, prevent users to use it.

that's indeed a good suggestion, thanks @SataQiu !

[neolit123]

FYI. in yesterday's Sig meeting we discussed kubeadm's v1beta4 plan. @fabriziopandini requested that we tag in the above list what is a [BREAKING] change. one example is allowing duplicate extraArgs. please check if there is anything else.

https://docs.google.com/document/d/1Gmc7LyCIL_148a9Tft7pdhdee0NBHdOfHS1SAF0duI4/edit?usp=drivesdk

we also discussed whether this should be a v1 or a v1beta4. my vote was to gather motivation from the kubelet which is moving to v1. perhaps v1beta4 should be our last beta. ideally, this means we should do all the planned breaking changes in this beta4. otherwise we need to plan v2 after the v1.

[chendave]

@neolit123 thanks for bringing this to the sig meeting!

we tag in the above list what is a [BREAKING] change

sure, will do.

[fabriziopandini]

thanks for adding [breaking]; it will echo yesterday discussion today at the CAPI office hours and report back feedback/ask people to provide feedback directly here

[neolit123]

i can try sending the boilerplate pr for v1beta4 later today. copy v3 -> v4, essentially.

work on an api for multiple releases

We can introduce the v1beta4 API, but not register it into the Scheme, prevent users to use it. https://github.com/kubernetes/kubernetes/blob/18d05b646d09b2971dc5400bc288062b0414e8cf/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go#L42-L46

i was thinking about this. it can work partially as long as we make the right changes in the internal api. but it will not work for breaking changes e.g. removing the ecdsa FG.

we may have to add the api changes first, keep the api out of the scheme, think when to add business logic, once ready for release add the api in the scheme, but drop api changes without business logic.

[chendave]

we may have to add the api changes first, keep the api out of the scheme, think when to add business logic, once ready for release add the api in the scheme, but drop api changes without business logic.

SGTM, we can have the boilerplate as the first commit and the api change as an incremental commit.

[neolit123]

boilerplate PR
kubernetes/kubernetes#118762

[pacoxu]

Too many tasks are tracked here. Should we use a simple GitHub board to track v1beta4-related things?

https://github.com/orgs/kubernetes/projects

[chendave]

thanks @pacoxu for the info! I am +1 to create the board.

Most important features which need some attention now is:

• support for structured ExtraArgs: kubeadm add support for structured ExtraArgs kubernetes#119156 (need review)

• upgrade API type: kubeadm: implementation of UpgradeConfiguration API types kubernetes#114062 (need some eyes, no review comments yet)

• support for controlling various timeouts during init / join:

  • https://docs.google.com/document/d/1VNGpGWb-vqblfZQLp9DY1lx777rwhwIFAjDFQNIwWh0 (design doc, I am pretty worried about this, we haven't finalized the way we should go)
  • kubeadm: add API support for controlling various timeouts during init / join kubernetes#119015

• skipping addons image pull: kubeadm: Support skipping addons image pull  kubernetes#114534 (waiting for rebase and review, this should be fine as it is not a big change)

Anything else should not hard to address, e.g. enable the v1beta4 in code base, ignore the errors should be align with other flag/config etc.

[pacoxu]

kubernetes/kubernetes#125029 is merged for v1.31.

• kinder: enable the v1beta4 API for >= 1.31 #3067 is WIP.

[neolit123]

closing this one. thanks everyone.
let's track bugs / feedback in separate new tickets.