Control plane should scale up serially, not in parallel

[dlipovetsky]

We recently implemented control plane scale up. If the desired number of control plane machines exceeds the actual number, and at least one control plane machine exists, the controller will create multiple machines in parallel. (Once created, each machine runs kubeadm join --control-plane).

I think we should scale up control planes serially. Before creating an additional control plane machine, we should verify that every etcd member has started. We could also verify that the etcd cluster has quorum (if it does not have quorum, creating a new machine might be a waste of time and resources. On the other hand, if it does have quorum, it might lose it after we create the machine)

Today, etcd still recommends that the cluster be scaled up or down one member at a time. Moreover, there are known issues with running kubeadm join --control-plane in parallel.

In the future, we will likely be able to scale up in parallel by using etcd non-voting members (learners). Kubeadm is already exploring this idea.

/cc @detiber @randomvariable @chuckha

[dlipovetsky]

#1902 is required to scale the control plane serially.

[dlipovetsky]

I think we meant to spell this out in the proposal; I will file a PR to update it.

[chuckha]

Some interesting information regarding this issue: kubernetes/kubeadm#2001 (comment)

It's tracked upstream here: kubernetes/kubeadm#1793

Until that is fixed we should indeed join serially or block on kubeadm join call and only run at concurrency level 1.

[ncdc]

Fixed by #2335
/close

[k8s-ci-robot]

@ncdc: Closing this issue.

In response to this:

Fixed by #2335
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.