Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: etcdserver: member not found, after upgrading helm chart etcd replicas 1 to 3 #1659

Closed
todeb opened this issue May 16, 2024 · 3 comments · Fixed by openebs/mayastor-extensions#536
Closed

Error: etcdserver: member not found, after upgrading helm chart etcd replicas 1 to 3 #1659

todeb opened this issue May 16, 2024 · 3 comments · Fixed by openebs/mayastor-extensions#536

Comments

@todeb
Copy link

todeb commented May 16, 2024

Describe the bug
After upgrading chart with mayastor.etcd.replicaCount=3 from 1.

etcd-0 member crashloopbackoff with log:

etcd 09:39:00.03 Welcome to the Bitnami etcd container
etcd 09:39:00.03 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 09:39:00.04 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 09:39:00.04
etcd 09:39:00.05 INFO  ==> ** Starting etcd setup **
etcd 09:39:00.07 INFO  ==> Validating settings in ETCD_* env vars..
etcd 09:39:00.07 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 09:39:00.09 INFO  ==> Initializing etcd
etcd 09:39:00.09 INFO  ==> Generating etcd config file using env variables
etcd 09:39:00.15 INFO  ==> Detected data from previous deployments
etcd 09:39:00.49 INFO  ==> Updating member in existing cluster
{"level":"warn","ts":"2024-05-16T09:39:00.595Z","logger":"etcd-client","caller":"v3@v3.5.6/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00037a8c0/openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
Error: etcdserver: member not found

other new deployed etcd members logs:

Defaulted container "etcd" out of: etcd, volume-permissions (init)
etcd 09:26:48.36
etcd 09:26:48.36 Welcome to the Bitnami etcd container
etcd 09:26:48.37 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 09:26:48.37 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 09:26:48.37
etcd 09:26:48.37 INFO  ==> ** Starting etcd setup **
etcd 09:26:48.39 INFO  ==> Validating settings in ETCD_* env vars..
etcd 09:26:48.39 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 09:26:48.40 INFO  ==> Initializing etcd
etcd 09:26:48.40 INFO  ==> Generating etcd config file using env variables
etcd 09:26:48.45 INFO  ==> There is no data from previous deployments
etcd 09:26:48.45 INFO  ==> Bootstrapping a new cluster
etcd 09:26:48.48 INFO  ==> Obtaining cluster member ID
etcd 09:26:48.50 INFO  ==> Starting etcd in background
etcd 09:26:51.69 INFO  ==> Stored member ID: 6f5be5a5cdec82d9
etcd 09:26:51.70 INFO  ==> Stopping etcd
Terminated
etcd 09:26:52.72 INFO  ==> ** etcd setup finished! **

etcd 09:26:52.74 INFO  ==> ** Starting etcd **
{"level":"info","ts":"2024-05-16T09:26:52.780Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2379,http://openebs-etcd.openebs.svc.cluster.local:2379"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTH_TOKEN","variable-value":"jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"100"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"openebs-etcd-0=http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380,openebs-etcd-1=http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380,openebs-etcd-2=http://openebs-etcd-2.openebs-etcd-headless.openebs.svc.cluster.local:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"new"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"openebs-etcd-1"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"8589934592"}
{"level":"warn","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=openebs-etcd-headless.openebs.svc.cluster.local"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2024-05-16T09:26:52.784Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2024-05-16T09:26:52.784Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2024-05-16T09:26:52.784Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2024-05-16T09:26:52.784Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"info","ts":"2024-05-16T09:26:52.784Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"warn","ts":"2024-05-16T09:26:52.785Z","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"member_id","data-dir":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-05-16T09:26:52.785Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"info","ts":"2024-05-16T09:26:52.785Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.785Z","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2024-05-16T09:26:52.786Z","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.6","git-sha":"cecbe35ce","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":10,"max-cpu-available":10,"member-initialized":true,"name":"openebs-etcd-1","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2379","http://openebs-etcd.openebs.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":8589934592,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"100ns","auto-compaction-interval":"100ns","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2024-05-16T09:26:52.789Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/bitnami/etcd/data/member/snap/db","took":"229.366µs"}
{"level":"info","ts":"2024-05-16T09:26:52.789Z","caller":"etcdserver/server.go:530","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2024-05-16T09:26:52.795Z","caller":"etcdserver/raft.go:529","msg":"restarting local member","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","commit-index":8}
{"level":"info","ts":"2024-05-16T09:26:52.798Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 switched to configuration voters=()"}
{"level":"info","ts":"2024-05-16T09:26:52.798Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 became follower at term 3"}
{"level":"info","ts":"2024-05-16T09:26:52.798Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6f5be5a5cdec82d9 [peers: [], term: 3, commit: 8, applied: 0, lastindex: 8, lastterm: 3]"}
{"level":"info","ts":"2024-05-16T09:26:52.827Z","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2024-05-16T09:26:52.828Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":8589934592,"quota-size":"8.6 GB"}
{"level":"info","ts":"2024-05-16T09:26:52.828Z","caller":"etcdserver/server.go:854","msg":"starting etcd server","local-member-id":"6f5be5a5cdec82d9","local-server-version":"3.5.6","cluster-version":"to_be_decided"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/bitnami/etcd/data/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/bitnami/etcd/data/member/snap","suffix":"snap","max":5,"interval":"30s"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/bitnami/etcd/data/member/wal","suffix":"wal","max":5,"interval":"30s"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"etcdserver/server.go:754","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 switched to configuration voters=(8024259661377864409)"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","added-peer-id":"6f5be5a5cdec82d9","added-peer-peer-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 switched to configuration voters=(8024259661377864409 8763204770994443348)"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","added-peer-id":"799d284f4d21d454","added-peer-peer-urls":["http://openebs-etcd-2.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454","remote-peer-urls":["http://openebs-etcd-2.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 switched to configuration voters=(8024259661377864409 8763204770994443348 16092638608179729187)"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","added-peer-id":"df54898d3b4a1323","added-peer-peer-urls":["http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.832Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.834Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.834Z","caller":"embed/etcd.go:275","msg":"now serving peer/client/metrics","local-member-id":"6f5be5a5cdec82d9","initial-advertise-peer-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2379","http://openebs-etcd.openebs.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2024-05-16T09:26:52.835Z","caller":"embed/etcd.go:586","msg":"serving peer traffic","address":"[::]:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.835Z","caller":"embed/etcd.go:558","msg":"cmux::serve","address":"[::]:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323","remote-peer-urls":["http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","cluster-version":"3.5"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"6f5be5a5cdec82d9","to":"799d284f4d21d454","stream-type":"stream Message"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6f5be5a5cdec82d9 elected leader 799d284f4d21d454 at term 3"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"6f5be5a5cdec82d9","to":"799d284f4d21d454","stream-type":"stream MsgApp v2"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.839Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"warn","ts":"2024-05-16T09:26:52.841Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"info","ts":"2024-05-16T09:26:52.842Z","caller":"embed/serve.go:100","msg":"ready to serve client requests"}
{"level":"warn","ts":"2024-05-16T09:26:52.842Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"info","ts":"2024-05-16T09:26:52.842Z","caller":"etcdserver/server.go:2054","msg":"published local member to cluster through raft","local-member-id":"6f5be5a5cdec82d9","local-member-attributes":"{Name:openebs-etcd-1 ClientURLs:[http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2379 http://openebs-etcd.openebs.svc.cluster.local:2379]}","request-path":"/0/members/6f5be5a5cdec82d9/attributes","cluster-id":"f2b3b9f758cdc80","publish-timeout":"7s"}
{"level":"info","ts":"2024-05-16T09:26:52.842Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2024-05-16T09:26:52.842Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
{"level":"info","ts":"2024-05-16T09:26:52.844Z","caller":"embed/serve.go:146","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"[::]:2379"}
{"level":"warn","ts":"2024-05-16T09:26:52.845Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:52.845Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:52.948Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:52.949Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.045Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.045Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.149Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.154Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.251Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.251Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.347Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.348Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.450Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.458Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.546Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.547Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.648Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.649Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.757Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.761Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.851Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.851Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.947Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.948Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:54.049Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:54.050Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 [term 3] received MsgTimeoutNow from 799d284f4d21d454 and starts an election to get leadership."}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 is starting a new election at term 3"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 became candidate at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 received MsgVoteResp from 6f5be5a5cdec82d9 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 [logterm: 3, index: 9] sent MsgVote request to 799d284f4d21d454 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 [logterm: 3, index: 9] sent MsgVote request to df54898d3b4a1323 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6f5be5a5cdec82d9 lost leader 799d284f4d21d454 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 received MsgVoteResp from 799d284f4d21d454 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 has received 2 MsgVoteResp votes and 0 vote rejections"}
{"level":"info","ts":"2024-05-16T09:26:54.063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 became leader at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6f5be5a5cdec82d9 elected leader 6f5be5a5cdec82d9 at term 4"}
{"level":"error","ts":"2024-05-16T09:26:54.067Z","caller":"rafthttp/util.go:99","msg":"request sent was ignored due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-cluster-id":"f2b3b9f758cdc80","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.checkPostResponse\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/util.go:99\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).post\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:163\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).handle\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:100"}
{"level":"error","ts":"2024-05-16T09:26:54.077Z","caller":"rafthttp/util.go:99","msg":"request sent was ignored due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-cluster-id":"f2b3b9f758cdc80","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.checkPostResponse\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/util.go:99\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).post\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:163\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).handle\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:100"}
{"level":"error","ts":"2024-05-16T09:26:54.103Z","caller":"rafthttp/util.go:99","msg":"request sent was ignored due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-cluster-id":"f2b3b9f758cdc80","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.checkPostResponse\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/util.go:99\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).post\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:163\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).handle\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:100"}
{"level":"warn","ts":"2024-05-16T09:26:54.146Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:54.150Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:54.162Z","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454","error":"EOF"}
{"level":"warn","ts":"2024-05-16T09:26:54.166Z","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454","error":"EOF"}
...
{"level":"warn","ts":"2024-05-16T09:45:46.015Z","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version","remote-member-id":"df54898d3b4a1323","error":"Get \"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version\": dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:46.015Z","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"df54898d3b4a1323","error":"Get \"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version\": dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:48.208Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"df54898d3b4a1323","rtt":"3.654257ms","error":"dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:48.224Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"df54898d3b4a1323","rtt":"8.09826ms","error":"dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:50.022Z","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version","remote-member-id":"df54898d3b4a1323","error":"Get \"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version\": dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:50.022Z","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"df54898d3b4a1323","error":"Get \"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version\": dial tcp 10.42.4.22:2380: connect: connection refused"}

Current helm values:

release:
  version: "4.0.1"
mayastor:
  enabled: true
  etcd:
    replicaCount: 3

Previous helm values:

release:
  version: "4.0.1"
mayastor:
  enabled: true
  etcd:
    replicaCount: 1

To Reproduce

  • install openebs helm with mayastor etcd replica 1
  • upgrade chart with etcd replica 3

Expected behavior
all etcd members are healthy

Screenshots
If applicable, add screenshots to help explain your problem.

OS info (please complete the following information):
openebs.io/version: 2.6.1

Additional context
Add any other context about the problem here.
@tiagolobocastro
Copy link
Contributor

I don't think this will work automatically.
Can you provide some points @datacore-tilangovan ?

@todeb
Copy link
Author

todeb commented May 16, 2024 •
edited
Loading

did also a test scalling to 2 replicas, in that case etcd-0 was healthy but etcd-1 show errors:

k logs -n openebs openebs-etcd-1
Defaulted container "etcd" out of: etcd, volume-permissions (init)
etcd 10:38:13.72
etcd 10:38:13.72 Welcome to the Bitnami etcd container
etcd 10:38:13.72 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 10:38:13.72 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 10:38:13.72
etcd 10:38:13.72 INFO  ==> ** Starting etcd setup **
etcd 10:38:13.76 INFO  ==> Validating settings in ETCD_* env vars..
etcd 10:38:13.77 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 10:38:13.78 INFO  ==> Initializing etcd
etcd 10:38:13.78 INFO  ==> Generating etcd config file using env variables
etcd 10:38:13.81 INFO  ==> Detected data from previous deployments
etcd 10:38:14.00 INFO  ==> Updating member in existing cluster
{"level":"warn","ts":"2024-05-16T10:38:14.099Z","logger":"etcd-client","caller":"v3@v3.5.6/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000e6a80/openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
Error: etcdserver: member not found

Also after 1st case when I rollback from 3 to 1 replica for etcd. etcd-0 lost its all data. Fortunately I made a snapshot of etcd before and restored it.

In 2nd case when scalling from 1 to 2 and rollback from 2 to 1. etcd-0 was healthy entire time so data was kept.

@tiagolobocastro
Copy link
Contributor

I haven't had the time yet, but I wonder if the etcd-operator would help here? #1630

This was referenced Aug 31, 2024
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(chart): allow scaling etcd via replicaCount and Helm upgrade michaelbeaumont/mayastor-extensions
3 participants
@tiagolobocastro @GlennBullingham @todeb