RFE: use learner mode for joining etcd members

[fabriziopandini]

Growing a local etcd cluster is a complex operation, and in the past, we already faced some issues like e.g. kubernetes-sigs/kind#588

Now that the implementation of the etcd learner mode is progressing, we should start considering if to use it in kubeadm in order to make join --control-plane implementation more robust.

at a high level what we would like to achieve is:

• a new etcd member should be created as a learner and became a voting member only after the etcd data are fully aligned.
  ideally
• we should also prevent the api-server to read from a learner node

Ref docs:

• https://etcd.io/docs/v3.3.12/learning/learner/
• https://etcd.io/docs/v3.4.0/learning/design-learner/

---

(edit by neolit123)

1.26:

• initial KEP draft
  https://hackmd.io/@DAKGcrh_RpC5vlt8w5bf8A/r1qoLh9zj
• KEP PR + tracking issue at k/e:
  sig-cl/kubeadm: add provisional KEP for etcd learner mode enhancements#3615
  sig-cl/kubeadm: Use etcd's learner mode enhancements#3614

1.27(alpha):

• k/k alpha feature gate code
  kubeadm: enable etcd's learner mode when joining etcd members kubernetes#113318
  kubeadm: fix etcd learner join kubernetes#115038
• CI added
  add e2e test for etcd learner mode #2807
  kubeadm: add e2e test for etcd learner mode test-infra#28574

1.29(beta):

• update etcd learner mode status of kubeadm implementation enhancements#4172
• kubeadm: promote EtcdLearnerMode to beta kubernetes#120228
• kubeadm: EtcdLearnerMode is beta in v1.29 website#42801
• add a new blog for etcd learner mode usage in kubeadm join website#43095

1.32(GA):

• [1.32]kubeadm: promote etcd learner mode to GA  kubernetes#126374
• kinder: remove learner-mode workflow #3096
• kubeadm: remove learner-mode test job test-infra#33316
• kubeadm-init: mark EtcdLearnerMode as GA website#47563
• kubeadm: check whether the peer URL for the added etcd member already exists when the MemberAddAsLearner/MemberAdd fails kubernetes#127491 and kubeadm: fix a bug where the RemoveMember function did not return the correct member list when the member to be removed did not exist kubernetes#127650 (Cherry-picked to v1.28-1.31)

1.33:

• TODO: remove the FG from kubeadm code
• TODO: update the k/website "kubeadm init" page

[SataQiu]

/cc

[rosti]

We have to be careful, but we certainly need to act upon it. The plan is that from etcd 3.5 new members will be joined only as learners.
In etcd 3.5 it will be possible to use a learner node for reading, but still the problem with writing continues. And, as LBs are out of the scope of kubeadm, things might become a bit difficult.
We probably need to direct API servers to healthy leaders and possibly do that via an etcd LB. Another possibility is to not expose the API servers, that have a local learner etcd node from the API LB (not sure if this would actually work though).
In short, we need to experiment a bit with this to find what's viable and easy for use.

[RA489]

/assign

[neolit123]

xref #2005 (comment)

https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/runtime-configuration.md

[prksu]

/cc

[ereslibre]

Some context on this: etcd-io/etcd#11640, we might want to wait for an etcd version that includes this patch.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

great feedback. thank you @tobiasgiese

maybe you could provide some feedback, I think you folks still use it right?

We (Mercedes-Benz) are using it already, yes. Also we have backported it to v1.2[4-6] (since kubernetes/kubernetes#115038) and it is working quite well. We have never had any problems with the learner mode and we have alot of nightly builds (about 50 periodic nightly builds and 40 Prow trigger builds/jobs).

[pacoxu]

I suppose that we should graduate this feature later in v1.31+ and get more feedback before GA.

So no action item for v1.30.

[neolit123]

I suppose that we should graduate this feature later in v1.31+ and get more feedback before GA.

So no action item for v1.30.

i got contacted in slack by a person that had feedback about learner mode in kubeadm, but they never send me the info.
learner mode was broken for them in some way.

i will see if i can message them about this after NY.

[pacoxu]

One issue that I may imagine is timeout for a step of promotion ready waiting or promotion may be a problem.

[pacoxu]

#2997 (comment)
We have a short discussion about if we need to add progress percentage of synced in logging.

• It needs change in etcd side.

#2997 (comment)
Another potentiel improvement is adding a configurable timeout for etcd learner ready for promoting. There are already a lot of timeout configuration in v1beta4 timeouts structs. (+0 for this as 2 min should be enough for most scanerios.)

[neolit123]

We have a short discussion about if we need to add progress percentage of synced in logging.

ok, i don't think it's GA blocking.

Another potentiel improvement is adding a configurable timeout for etcd learner ready for promoting. There are already a lot of timeout configuration in v1beta4 timeouts structs. (+0 for this as 2 min should be enough for most scanerios.)

+0 as well from me.
our 2 minutes timeout will apply to all etcd client calls by default.

[neolit123]

i got contacted in slack by a person that had feedback about learner mode in kubeadm, but they never send me the info.
learner mode was broken for them in some way.

they did not log an issue...

[pacoxu]

I updated beta related PRs in this issue description.

I think we may wait for at least another 1 or 2 release cycles for feedbacks to make this GA, as most users are not using v1.29 yet, which make it beta, by default enabled.

[neolit123]

40s timeout for waiting to be ready to promote a learner.

should we increase this time to 2 minutes, or more by default?

[pacoxu]

https://github.com/kubernetes/kubernetes/blob/227c2e7c2b2c05a9c8b2885460e28e4da25cf558/cmd/kubeadm/app/util/etcd/etcd.go#L531-L557

already 2m.

I miss the log that The learner was promoted as a voting member success finally. Sorry for disturb.

[neolit123]

the flakes on https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-kinder-upgrade-addons-before-controlplane-1-29-latest
seem like slow infra problems, 5 minutes should be plenty of time for a few nodes to join and be ready :/

[neolit123]

I think we may wait for at least another 1 or 2 release cycles for feedbacks to make this GA, as most users are not using v1.29 yet, which make it beta, by default enabled.

@pacoxu should we GA this in 1.32?

[pacoxu]

Agree