TLS under container environment with log: x509: certificate specifies an incompatible key usage

[walleliu1016]

I want to deploy etcd in kubernetes use etcd-operator with tls enabled.
the first node start successfully, but the second container start with err as the first etcd node reject connection.
the running etcd node err log

{"level":"warn","ts":"2024-01-19T16:12:27.873+0800","caller":"embed/config_logging.go:279","msg":"rejected connection","remote-addr":"127.0.0.1:58670","server-name":"","error":"tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage"}

my etcd version: 3.4.0

my etcd config:

    Command:
      /usr/local/bin/etcd
      --data-dir=/var/etcd/data
      --name=test-02-6kzs4hfq6c
      --initial-advertise-peer-urls=https://test-02-6kzs4hfq6c.test-02.etcd.svc.cluster.local:2380
      --listen-peer-urls=https://0.0.0.0:2380
      --listen-client-urls=https://0.0.0.0:2379
      --advertise-client-urls=https://test-02-6kzs4hfq6c.test-02.etcd.svc.cluster.local:2379
      --initial-cluster=test-02-6kzs4hfq6c=https://test-02-6kzs4hfq6c.test-02.etcd.svc.cluster.local:2380
      --initial-cluster-state=new
      --peer-client-cert-auth=true
      --peer-trusted-ca-file=/etc/etcdtls/member/peer-tls/peer-ca.crt
      --peer-cert-file=/etc/etcdtls/member/peer-tls/peer.crt
      --peer-key-file=/etc/etcdtls/member/peer-tls/peer.key
      --client-cert-auth=true
      --trusted-ca-file=/etc/etcdtls/member/server-tls/server-ca.crt
      --cert-file=/etc/etcdtls/member/server-tls/server.crt
      --key-file=/etc/etcdtls/member/server-tls/server.key
      --initial-cluster-token=9a2f1e58-c123-4442-a5c0-7545f2767531

some stdout log in running pod:

{"level":"warn","ts":"2024-01-19T16:12:22.631+0800","caller":"embed/config_logging.go:279","msg":"rejected connection","remote-addr":"127.0.0.1:58560","server-name":"","error":"tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage"}
{"level":"warn","ts":"2024-01-19T16:12:22.631+0800","caller":"grpclog/grpclog.go:60","msg":"grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0  <nil>}. Err :connection error: desc = \"transport: authentication handshake failed: remote error: tls: bad certificate\". Reconnecting..."}
{"level":"warn","ts":"2024-01-19T16:12:23.637+0800","caller":"embed/config_logging.go:279","msg":"rejected connection","remote-addr":"127.0.0.1:58578","server-name":"","error":"tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage"}
{"level":"warn","ts":"2024-01-19T16:12:23.637+0800","caller":"grpclog/grpclog.go:60","msg":"grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0  <nil>}. Err :connection error: desc = \"transport: authentication handshake failed: remote error: tls: bad certificate\". Reconnecting..."}
{"level":"warn","ts":"2024-01-19T16:12:25.225+0800","caller":"embed/config_logging.go:279","msg":"rejected connection","remote-addr":"127.0.0.1:58622","server-name":"","error":"tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage"}
{"level":"warn","ts":"2024-01-19T16:12:25.225+0800","caller":"grpclog/grpclog.go:60","msg":"grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0  <nil>}. Err :connection error: desc = \"transport: authentication handshake failed: remote error: tls: bad certificate\". Reconnecting..."}
{"level":"info","ts":"2024-01-19T16:12:26.877+0800","caller":"raft/raft.go:1530","msg":"82015e02429f711e switched to configuration voters=(9367872063708033310 10675801162557415164)"}
{"level":"info","ts":"2024-01-19T16:12:26.878+0800","caller":"membership/cluster.go:392","msg":"added member","cluster-id":"4242cc1a22fac1a6","local-member-id":"82015e02429f711e","added-peer-id":"9428109832efdefc","added-peer-peer-urls":["https://test-02-kc9zdmvsfr.test-02.etcd.svc.cluster.local:2380"]}

some err log in err pod

{"level":"warn","ts":"2024-01-19T16:12:30.472+0800","caller":"etcdserver/cluster_util.go:76","msg":"failed to get cluster response","address":"https://test-02-6kzs4hfq6c.test-02.etcd.svc.cluster.local:2380/members","error":"Get https://test-02-6kzs4hfq6c.test-02.etcd.svc.cluster.local:2380/members: EOF"}
{"level":"info","ts":"2024-01-19T16:12:30.472+0800","caller":"embed/etcd.go:360","msg":"closing etcd server","name":"test-02-kc9zdmvsfr","data-dir":"/var/etcd/data","advertise-peer-urls":["https://test-02-kc9zdmvsfr.test-02.etcd.svc.cluster.local:2380"],"advertise-client-urls":["https://test-02-kc9zdmvsfr.test-02.etcd.svc.cluster.local:2379"]}
{"level":"info","ts":"2024-01-19T16:12:30.472+0800","caller":"embed/etcd.go:364","msg":"closed etcd server","name":"test-02-kc9zdmvsfr","data-dir":"/var/etcd/data","advertise-peer-urls":["https://test-02-kc9zdmvsfr.test-02.etcd.svc.cluster.local:2380"],"advertise-client-urls":["https://test-02-kc9zdmvsfr.test-02.etcd.svc.cluster.local:2379"]}
{"level":"warn","ts":"2024-01-19T16:12:30.473+0800","caller":"etcdmain/etcd.go:176","msg":"failed to start etcd","error":"cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs"}
{"level":"fatal","ts":"2024-01-19T16:12:30.473+0800","caller":"etcdmain/etcd.go:271","msg":"discovery failed","error":"cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs","stacktrace":"go.etcd.io/etcd/etcdmain.startEtcdOrProxyV2\n\t/tmp/etcd-release-3.4.0/etcd/release/etcd/etcdmain/etcd.go:271\ngo.etcd.io/etcd/etcdmain.Main\n\t/tmp/etcd-release-3.4.0/etcd/release/etcd/etcdmain/main.go:46\nmain.main\n\t/tmp/etcd-release-3.4.0/etcd/release/etcd/main.go:28\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}

the tls config as follower

ca-config.json

{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

ca-csr.json

{
    "CN": "My own CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "O": "My Company Name",
            "ST": "San Francisco",
            "OU": "Org Unit 1"
        }
    ]
}

server.json

{
    "CN": "etcd server",
    "hosts": [
        "*.test-02.etcd.svc.cluster.local",
        "test-02-client.etcd.svc.cluster.local",
        "localhost"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}

peer.json

{
    "CN": "etcd peer",
    "hosts": [
        "*.test-02.etcd.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}

client.json

{
    "CN": "etcd client",
    "hosts": [""],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}

[jmhbnz]

Hey @walleliu1016 - Thanks for your question, based on https://etcd.io/docs/v3.5/op-guide/security/#im-seeing-a-sslv3-alert-handshake-failure-when-using-tls-client-authentication I think your ca-config.json server profile requires the client auth usage.

The Verify function in the crypto/x509 logic implements a common, but non-standard extension - it requires that CA & intermediate certificates either define no extended key usage, or a superset of those on the end-entity certificates. If certificates in your chain define any extended key usages, they must also include serverAuth and/or clientAuth.

Otherwise, you may see an error like unsuitable certificate purpose (OpenSSL) or certificate specifies an incompatible key usage (Go).

Please give that a go and report back.

[walleliu1016]

hi @jmhbnz thanks for your reply.
After added the client auth into the server profile the ecertificate specifies an incompatible key usage err disappeared.
But still got some error, the second etcd pod cannot join the etcd cluster as the first pod rejected the connection.
here the logs for both pod,

error.log is the pod try to join the cluster, but end with cannot get the members with err:

{"level":"warn","ts":"2024-01-20T18:56:37.180+0800","caller":"etcdserver/cluster_util.go:76","msg":"failed to get cluster response","address":"https://test-02-9z9bmcg9g5.test-02.etcd.svc.cluster.local:2380/members","error":"Get https://test-02-9z9bmcg9g5.test-02.etcd.svc.cluster.local:2380/members: EOF"}

error.log

runnning.log is the pod aleady running, and rejected the connection.

running.log

ca-config.json

{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

pod details:

[kwohlfahrt]

{"level":"warn","ts":"2024-01-20T18:56:37.179+0800","caller":"embed/config_logging.go:270","msg":"rejected connection","remote-addr":"8.0.1.136:39984","server-name":"test-02-9z9bmcg9g5.test-02.etcd.svc.cluster.local","ip-addresses":[],"dns-names":["*.test-02.etcd.svc","*.test-02.etcd.svc.cluster.local"],"error":"tls: \"8.0.1.136\" does not match any of DNSNames [\"*.test-02.etcd.svc\" \"*.test-02.etcd.svc.cluster.local\"] (lookup 136.1.0.8.in-addr.arpa. on 8.30.0.10:53: no such host)"}

It looks like your server is trying to identify itself by IP address, but there is no entry for this in the certificate (only the DNS name).

I don't remember off the top of my head whether a certificate for the IP is always required, or whether it depends on the peer URL or some other setting.

[walleliu1016]

kubernetes will create a domain name for each Pod, such as test-02-xxx.test-02.etcd.svc.cluster.local, so that it can be judged by wildcards when generating certificates, such as *test in my attachment -02.etcd. svc.cluster.local.
https://etcd.io/docs/v3.5/op-guide/security/#im-seeing-a-sslv3-alert-handshake-failure-when-using-tls-client-authentication
This article also states that DNS wildcards are supported from version 3.2.5 onwards.
From my analysis of the current situation of the two etcd pods, the reverse resolution failed because the second etcd node failed to start, causing the domain name to be deleted, so the first etcd pod tried to determine whether it was a valid client through IP.

But no reason was found why the second node failed to call the interface of the first etcd node https://test-02-9z9bmcg9g5.test-02.etcd.svc.cluster.local:2380/members

[walleliu1016]

i have read many issue about this problem
coreos/etcd-operator#1962
coreos/etcd-operator#1384
#8268
#8803
#9398
@kwohlfahrt @jmhbnz looking forward your reply