etcd query range API took > 200ms, how to optimize existing etcd cluster? #15775

ningyougang · 2023-04-25T06:57:46Z

ningyougang
Apr 25, 2023

Send query range request to etcd cluster

$ curl -H "Authorization: Basic xxx" -w "@/tmp/curl-format.txt" -o /dev/null -s -L -X GET  "http://our-company-inner-server/xxx/xxx"
time_namelookup[time-consuming for DNS to resolve the domain name]:                 0.004
time_connect[time-consuming for the client and server establish a TCP connection]:  0.007
time_pretransfer[time-consuming from request to start response]:                    0.007
time_starttransfer[time-consuming from request to start of transmission]:           0.263
---------------------------------------------------------------------------------------------------------
time_total[total time-consuming]:                                                   0.263

Our company inner server will send like below request to etcd cluster

etcdClient.Get(con, resultKey.SearchKey(), clientv3.WithPrefix())

From above result, we can see, the response time is about 263ms，sometime, the response time is about 500ms
Obviously, the response time is very long, my target is reduce the response time to < 100ms

Btw, our etcd cluster has about 150, 000 keys.

etcd cluster's startup command

/usr/local/bin/etcd \
  --name $HOSTNAME \
  --listen-client-urls "https://$IP:10379,http://127.0.0.1:10379" \
  --listen-peer-urls "https://$IP:2380,http://127.0.0.1:2380" \
  --advertise-client-urls "https://$HOSTNAME.nfra.io:10379" \
  --initial-cluster-token adl1-etcd-1 \
  --initial-advertise-peer-urls "https://$HOSTNAME.nfra.io:2380" \
  --initial-cluster "a_server=https://a_server:2380,b_server=https://b_server:2380,c_server=https://c_server:2380,d_server=https://d_server:2380,e_server=https://e_server:2380" \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file=./certs/ca.pem \
  --cert-file=./certs/$HOSTNAME.pem \
  --key-file=./certs/$HOSTNAME-key.pem \
  --peer-auto-tls \
  --auto-compaction-retention=1 \
  --quota-backend-bytes=$((6*1024*1024*1024))\
  --logger=zap \
  --log-outputs=stderr,a.log

Execute like above command to all etcd nodes

How to optimize existing etcd cluster?

How to find the issue on existing etcd cluster?

As we already know, when execute mysql statement, we can find the execute path and find the issue,
But for etcd, when execute like etcdctl get /..../.... --prefix, there has any way to find where the issue?

Other info

Endpoint health status

# ETCDCTL_API=3 etcdctl endpoint health --write-out=table --endpoints=https://a_server:2379,https://b_server:2379,https://c_server:2379,https://d_server:2379,https://e_server:2379,https://f_server:2379  --cacert=./etcd-cert/ca.pem --cert=./etcd-cert/client-cert.pem --key=./etcd-cert/client-key.pem
+-----------------------------+--------+--------------+-------+
|          ENDPOINT           | HEALTH |     TOOK     | ERROR |
+-----------------------------+--------+--------------+-------+
|    https://e_server:2379 |   true | 335.963958ms |       |
|    https://f_server:2379 |   true | 342.358299ms |       |
| https://a_server:2379 |   true | 342.301271ms |       |
| https://b_server:2379 |   true | 357.998212ms |       |
|  https://c_server:2379 |   true | 361.076328ms |       |
|  https://a_server:2379 |   true | 366.871719ms |       |
+-----------------------------+--------+--------------+-------+

ahrtr · 2023-04-25T07:14:47Z

ahrtr
Apr 25, 2023
Maintainer

The possible bottleneck:

network delay between peer communication; Please check the metrics peer_round_trip_time_seconds
slow disk; Please check metrics wal_fsync_duration_seconds and backend_commit_duration_seconds
big DB size, and accordingly big memory usage. If you try to perform defragmentation using etcd-defrag

Please read How to debug performance issue? .

Also check the etcdserver's log, check how big the range response is; note the clientv3.WithPrefix() might cause the etcdserver to return a huge number of records.

0 replies

ningyougang · 2023-04-26T01:39:12Z

ningyougang
Apr 26, 2023
Author

@ahrtr

metric check

histogram_quantile(0.99, etcd_network_peer_round_trip_time_seconds_bucket)

Seems it is ok
sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[$__rate_interval])) by (instance) and sum(rate(etcd_disk_backend_commit_duration_seconds_sum[$__rate_interval])) by (instance)

Seems it is ok too.

etcd endpoint health check

I used below command to check

ETCDCTL_API=3 etcdctl endpoint status --write-out=json --endpoints=https://a_server:2379,https://b_server:2379,https://c_server:2379,https://d_server:2379,https://e_server:2379,https://f_server:2379  --cacert=./etcd-cert/ca.pem --cert=./etcd-cert/client-cert.pem --key=./etcd-cert/client-key.pem

It is ok too.

Do etcd-defrag tool to perform defragmentation

After applied, invoke range api's response time is still > 200ms

etcd startup style

use docker to startup. key info as below

Env

"Env": [
  "ETCD_DATA_DIR=/var/lib/etcd",
  "ETCD_ADVERTISE_CLIENT_URLS=https://xxx.xxx.xxx.xxx:2379",
  "ETCD_INITIAL_ADVERTISE_PEER_URLS=https://xxx.xxx.xxx.xxx:2380",
  "ETCD_INITIAL_CLUSTER_STATE=existing",
  "ETCD_METRICS=basic",
  "ETCD_LISTEN_CLIENT_URLS=https://xxx.xxx.xxx.xxx:2379,https://127.0.0.1:2379",
  "ETCD_ELECTION_TIMEOUT=5000",
  "ETCD_HEARTBEAT_INTERVAL=250",
  "ETCD_INITIAL_CLUSTER_TOKEN=elb-etcd-1",
  "ETCD_LISTEN_PEER_URLS=https://xxx.xxx.xxx.xxx:2380",
  "ETCD_NAME=aee001",
  "ETCD_PROXY=off",
  "ETCD_INITIAL_CLUSTER=aee001=https://xxx.xxx.xxx.xxx:2380,aee002=https://xxx.xxx.xxx.xxx:2380,cee001=https://xxx.xxx.xxx.xxx:2380,cee002=https://xxx.xxx.xxx.xxx:2380,xee001=https://xxx.xxx.xxx.xxx:2380,xee002=http://xxx.xxx.xxx.xxx:2380",
  "ETCD_AUTO_COMPACTION_MODE=periodic",
  "ETCD_AUTO_COMPACTION_RETENTION=10m",
  "ETCD_SNAPSHOT_COUNT=10000",
  "ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem",
  "ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-aee001.ncc.pem",
  "ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-aee001.ncc-key.pem",
  "ETCD_CLIENT_CERT_AUTH=true",
  "ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem",
  "ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-aee001.ncc.pem",
  "ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-aee001.ncc-key.pem",
  "ETCD_PEER_CLIENT_CERT_AUTH=true",
  "ETCD_QUOTA_BACKEND_BYTES=8589934592",
],

mounts

"Mounts": [
    {
        "Type": "bind",
        "Source": "/var/lib/etcd",
        "Destination": "/var/lib/etcd",
        "Mode": "rw",
        "RW": true,
        "Propagation": "rprivate"
    },
    {
        "Type": "bind",
        "Source": "/etc/ssl/certs",
        "Destination": "/etc/ssl/certs",
        "Mode": "ro",
        "RW": false,
        "Propagation": "rprivate"
    },
    {
        "Type": "bind",
        "Source": "/etc/ssl/etcd/ssl",
        "Destination": "/etc/ssl/etcd/ssl",
        "Mode": "ro",
        "RW": false,
        "Propagation": "rprivate"
    }
],

use hdd or ssd to storage data(after checked, the disk type is `ssd`)

# df -h
...
/dev/sda2        64G  6.3G   58G  10% /
/dev/sda1       300M   12M  289M   4% /boot/efi
/dev/sda4       3.0T  2.9G  3.0T   1% /var
...

[root@aee001.ncc member]# fdisk -l /dev/sda4

Disk /dev/sda4: 3287.2 GB, 3287155343360 bytes, 6420225280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 917504 bytes

[root@aee001.ncc member]# cat /sys/block/sda/queue/rotational
0

Use ssd to storage etcd data to /var/lib/etcd

etcd records check

the total number is 163, 694

when execute query range api, the return number is small. e.g.

{"log":"2023-04-25 09:12:02.819485 W | etcdserver: read-only range request \"key:\\\"/dns/com/company/io/us/band/\\\" range_end:\\\"/dns/com/company/io/us/band0\\\" \" with result \"range_response_count:32 size:3045\" took too long (346.62893ms) to execute\n","stream":"stderr","time":"2023-04-25T09:12:02.819529991Z"}
{"log":"2023-04-25 09:12:02.819668 W | etcdserver: read-only range request \"key:\\\"/dns/com/company/io/scv-game/\\\" range_end:\\\"/dns/com/company/io/scv-game0\\\" \" with result \"range_response_count:4 size:356\" took too long (569.466744ms) to execute\n","stream":"stderr","time":"2023-04-25T09:12:02.819704929Z"}

There has other clue to check?

0 replies

ahrtr · 2023-04-26T01:56:35Z

ahrtr
Apr 26, 2023
Maintainer

Please look at the trace log, please provide more logs around the two log entries you provided.

The range request returned 3045 and 356 records respectively. Also please try to read a key prefix with less number of keys.

1 reply

ahrtr Apr 26, 2023
Maintainer

Note that there are some performance improvements in newer versions, please also try 3.4.25 or 3.5.8 if possible.

ningyougang · 2023-05-22T03:12:53Z

ningyougang
May 22, 2023
Author

After added more logs in our application, like this

func GetWithRecordTime(etcdClient Client, ctx context.Context, key string, opts ...clientv3.OpOption) (*clientv3.GetResponse, error) {
	startTime := common.Timestamp()
	logrus.Infof("Starting function execution [key: %s]", key)
	response, err := etcdClient.Get(ctx, key, opts...)
	endTime := common.Timestamp()
	took := endTime - startTime
	logrus.Infof("End function execution [key: %s, took: %dms]", key, took)
	return response, err
}

etcd itself has no problem(Btw, only consumes < 10ms for above logs)
The problem is network data transmission from server to client, because the response data has about 500+ records, so it may consume much time for network transmission

0 replies

ningyougang · 2023-05-22T03:14:20Z

ningyougang
May 22, 2023
Author

I will close this.
Anyway, thanks for @ahrtr for your kind reply.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcd query range API took > 200ms, how to optimize existing etcd cluster? #15775

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

etcd query range API took > 200ms, how to optimize existing etcd cluster? #15775

ningyougang Apr 25, 2023

Send query range request to etcd cluster

etcd cluster's startup command

How to optimize existing etcd cluster?

Other info

Replies: 5 comments · 1 reply

ahrtr Apr 25, 2023 Maintainer

ningyougang Apr 26, 2023 Author

metric check

etcd endpoint health check

Do etcd-defrag tool to perform defragmentation

etcd startup style

use hdd or ssd to storage data(after checked, the disk type is ssd)

etcd records check

ahrtr Apr 26, 2023 Maintainer

ahrtr Apr 26, 2023 Maintainer

ningyougang May 22, 2023 Author

ningyougang May 22, 2023 Author

ningyougang
Apr 25, 2023

Replies: 5 comments 1 reply

ahrtr
Apr 25, 2023
Maintainer

ningyougang
Apr 26, 2023
Author

use hdd or ssd to storage data(after checked, the disk type is `ssd`)

ahrtr
Apr 26, 2023
Maintainer

ahrtr Apr 26, 2023
Maintainer

ningyougang
May 22, 2023
Author

ningyougang
May 22, 2023
Author