setting tls-min-version to TLSv1.3 will result in kube-apiserver constant restarts

[zhzhuang-zju]

Referring to #15156, etcd can configure TLS protocol configuration through parameter --tls-min-version, and supports TLSv1.3, so I tried to config etcd with - --tls-min-version=TLS1.3 in a k8s cluster installed by kind, and found that kube-apiserver keeps restarting.
How to reproduce

1. created a k8s cluster with kind
   imageRef: registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/kube-apiserver:v1.27.3
2. setting tls-min-version to TLSv1.3

    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --tls-min-version=TLS1.3

3. view etcd log

2023-11-09T03:14:10.131121822Z stderr F {"level":"warn","ts":"2023-11-09T03:14:10.130955Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:55268","server-name":"","error":"tls: client offered only unsupported versions: [303]"}
2023-11-09T03:14:16.063110628Z stderr F {"level":"warn","ts":"2023-11-09T03:14:16.062847Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:38058","server-name":"","error":"tls: client offered only unsupported versions: [303]"}
2023-11-09T03:14:16.109019988Z stderr F {"level":"warn","ts":"2023-11-09T03:14:16.108888Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:38068","server-name":"","error":"tls: client offered only unsupported versions: [303]"}
2023-11-09T03:14:16.314482421Z stderr F {"level":"warn","ts":"2023-11-09T03:14:16.314339Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:38080","server-name":"","error":"tls: client offered only unsupported versions: [303]"}

4. view kube-apiserver log

2023-11-09T03:14:16.109079654Z stderr F }. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: protocol version not supported"
2023-11-09T03:14:16.314613059Z stderr F W1109 03:14:16.314542       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
2023-11-09T03:14:16.314639555Z stderr F   "Addr": "127.0.0.1:2379",
2023-11-09T03:14:16.314642037Z stderr F   "ServerName": "127.0.0.1",
2023-11-09T03:14:16.314644226Z stderr F   "Attributes": null,
2023-11-09T03:14:16.314646089Z stderr F   "BalancerAttributes": null,
2023-11-09T03:14:16.314648032Z stderr F   "Type": 0,
2023-11-09T03:14:16.31465014Z stderr F   "Metadata": null
2023-11-09T03:14:16.31465246Z stderr F }. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: protocol version not supported"
2023-11-09T03:14:20.008472589Z stderr F E1109 03:14:20.008338       1 run.go:74] "command failed" err="context deadline exceeded"

When upgrading etcd to v3.5.10 with gcr.io/etcd-development/etcd:v3.5.10, the restart problem still exists.

[zhzhuang-zju]

That's confusing, I don't know if there is something wrong with my configuration. If you can assist, I'd appreciate it! Thank you~

[jmhbnz]

Hey @zhzhuang-zju - Thanks for your question. It looks like your kubernetes cluster api server has not been configured to use or support TLS 1.3. I believe both etcd and the api server will need to be configured for this?

For api server I believe the flag is --tls-min-version=VersionTLS13, refer https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver.

[zhzhuang-zju]

Thanks for your suggestion~, I tried as you suggested but still failed. I guess this configuration can only limit it as the server's cipher suites. The server supports tlsv1.3 but the client may not support it.

[zhzhuang-zju]

In these error messages, 303 is TLS 1.2, 302 is TLS 1.1, 301 is TLS 1.0 and 300 is SSL 3.0. This means that kube-apiserver can only support tls1.2 when connecting to etcd. Therefore, I upgraded the version of the component kube-apiserver and found that when its release version was v1.28.0, the restart problem was solved.