Why is the timeout for loading snapshot 10 second (HARDCODE)

[shuangchengsun]

I have changed quota-backend-bytes to 36G, however, here comes some error when the etcd has been restarted. the error message is 「db file is flocked by another process, or taking too long」. I known it's my fault, but i still don't known why the timeout is hardcode. Or, is here some way I can restart the etcd node?

[serathius]

This is just a warning wrote if reading etcd db file takes longer than 10 seconds. It doesn't have any effect on the process. Etcd will still continue to try to read the file for as long as it takes.

[shuangchengsun]

Oh, sorry, I think the db was loaded finished. But the NODE is restart again and again. The Log like this .

{"level":"warn","ts":"2024-02-01T19:29:36.462+0800","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} {"level":"info","ts":"2024-02-01T19:29:36.462+0800","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","resto red-compact-revision":17611158} {"level":"info","ts":"2024-02-01T19:29:59.309+0800","caller":"etcdmain/config.go:339","msg":"loaded server configuration, other configuration command line flags and environment variables will be ign ored if provided","path":"/etc/sysconfig/etcd.yaml"} {"level":"info","ts":"2024-02-01T19:29:59.309+0800","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/bin/etcd","--config-file=/etc/sysconfig/etcd.yaml"]} {"level":"info","ts":"2024-02-01T19:29:59.309+0800","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/media/ssd1/etcd/data","dir-type":"member"} {"level":"info","ts":"2024-02-01T19:29:59.309+0800","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:4010"]} {"level":"info","ts":"2024-02-01T19:29:59.310+0800","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:4012"]} {"level":"info","ts":"2024-02-01T19:29:59.310+0800","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"08407ff76","go-version":"go1.16.15","go-os":"linux" ,"go-arch":"amd64","max-cpu-set":64,"max-cpu-available":64,"member-initialized":true,"name":"kce-Etcd1","data-dir":"/media/ssd1/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/media/ss d1/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1.5s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"in itial-advertise-peer-urls":["http://10.50.75.94:4010"],"listen-peer-urls":["http://0.0.0.0:4010"],"advertise-client-urls":["http://10.50.75.94:4012"],"listen-client-urls":["http://0.0.0.0:4012"],"li sten-metrics-urls":["http://0.0.0.0:4013"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":1099511627776,"pre-vot e":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"4h0m0s","auto-compaction-interval":"4h0m0s","discovery-url":"" ,"discovery-proxy":"","downgrade-check-interval":"5s"} {"level":"info","ts":"2024-02-01T19:30:09.311+0800","caller":"etcdserver/backend.go:85","msg":"db file is flocked by another process, or taking too long","path":"/media/ssd1/etcd/data/member/snap/db ","took":"10.000804538s"} {"level":"info","ts":"2024-02-01T19:30:34.222+0800","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":29505205,"snapshot-size":"15 kB"} {"level":"info","ts":"2024-02-01T19:30:34.222+0800","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":39872241664,"backend-size":"40 GB","backend-si ze-in-use-bytes":21685284864,"backend-size-in-use":"22 GB"} {"level":"info","ts":"2024-02-01T19:31:16.448+0800","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"71446a3cab9553d7","local-member-id":"b18209dd886cf1f2","commit-ind ex":29543831} {"level":"info","ts":"2024-02-01T19:31:16.449+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b18209dd886cf1f2 switched to configuration voters=(2584225942489656687 327065102033752 8953 3276783544776782025 12768609750960732515 12790796738766893554)"} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b18209dd886cf1f2 became follower at term 10"} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft b18209dd886cf1f2 [peers: [23dd03cd8971ed6f,2d63afc63b7cc079,2d797945d843f0c9,b 13336eae19a7963,b18209dd886cf1f2], term: 10, commit: 29543831, applied: 29505205, lastindex: 29543831, lastterm: 10]"} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"71446a3cab9553d7","local-member-id":"b18209dd886cf1f2 ","recovered-remote-peer-id":"2d63afc63b7cc079","recovered-remote-peer-urls":["http://10.50.55.159:4010"]} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"71446a3cab9553d7","local-member-id":"b18209dd886cf1f2 ","recovered-remote-peer-id":"2d797945d843f0c9","recovered-remote-peer-urls":["http://10.50.75.158:4010"]} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"71446a3cab9553d7","local-member-id":"b18209dd886cf1f2 ","recovered-remote-peer-id":"b13336eae19a7963","recovered-remote-peer-urls":["http://10.50.55.117:4010"]} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"71446a3cab9553d7","local-member-id":"b18209dd886cf1f2 ","recovered-remote-peer-id":"b18209dd886cf1f2","recovered-remote-peer-urls":["http://10.50.75.94:4010"]} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"71446a3cab9553d7","local-member-id":"b18209dd886cf1f2 ","recovered-remote-peer-id":"23dd03cd8971ed6f","recovered-remote-peer-urls":["http://10.50.55.246:4010"]} {"level":"info","ts":"2024-02-01T19:31:16.450+0800","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"} {"level":"warn","ts":"2024-02-01T19:31:16.450+0800","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} {"level":"info","ts":"2024-02-01T19:31:16.451+0800","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","resto red-compact-revision":17611158} {"level":"info","ts":"2024-02-01T19:31:39.310+0800","caller":"etcdmain/config.go:339","msg":"loaded server configuration, other configuration command line flags and environment variables will be ign ored if provided","path":"/etc/sysconfig/etcd.yaml"} {"level":"info","ts":"2024-02-01T19:31:39.310+0800","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/bin/etcd","--config-file=/etc/sysconfig/etcd.yaml"]} {"level":"info","ts":"2024-02-01T19:31:39.310+0800","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/media/ssd1/etcd/data","dir-type":"member"} {"level":"info","ts":"2024-02-01T19:31:39.310+0800","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:4010"]} {"level":"info","ts":"2024-02-01T19:31:39.310+0800","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:4012"]} {"level":"info","ts":"2024-02-01T19:31:39.310+0800","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"08407ff76","go-version":"go1.16.15","go-os":"linux" ,"go-arch":"amd64","max-cpu-set":64,"max-cpu-available":64,"member-initialized":true,"name":"kce-Etcd1","data-dir":"/media/ssd1/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/media/ss d1/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1.5s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"in itial-advertise-peer-urls":["http://10.50.75.94:4010"],"listen-peer-urls":["http://0.0.0.0:4010"],"advertise-client-urls":["http://10.50.75.94:4012"],"listen-client-urls":["http://0.0.0.0:4012"],"li sten-metrics-urls":["http://0.0.0.0:4013"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":1099511627776,"pre-vot e":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"4h0m0s","auto-compaction-interval":"4h0m0s","discovery-url":"" ,"discovery-proxy":"","downgrade-check-interval":"5s"} {"level":"info","ts":"2024-02-01T19:31:49.311+0800","caller":"etcdserver/backend.go:85","msg":"db file is flocked by another process, or taking too long","path":"/media/ssd1/etcd/data/member/snap/db ","took":"10.000152281s"}