etcdctl cluster-health and member list commands do not work correctly

[kelseyhightower]

etcd server version

/opt/bin/etcd --version
etcd version 2.0.9

etcd client version

/usr/local/bin/etcdctl --version
etcdctl version 2.0.9

Start a 3 node etcd cluster

vmrun list
Total running VMs: 3
/Users/kelseyhightower/Documents/Virtual Machines.localized/core0.vmwarevm/core0.vmx
/Users/kelseyhightower/Documents/Virtual Machines.localized/core1.vmwarevm/core1.vmx
/Users/kelseyhightower/Documents/Virtual Machines.localized/core2.vmwarevm/core2.vmx

etcdctl cluster-health
cluster is healthy
member 5ae3067007f7fb85 is healthy
member 7931e79c0d8b47c5 is healthy
member 987146e8925f10e5 is healthy

etcdctl member list
5ae3067007f7fb85: name=etcd2 peerURLs=http://192.168.12.52:2380 clientURLs=http://192.168.12.52:2379
7931e79c0d8b47c5: name=etcd0 peerURLs=http://192.168.12.50:2380 clientURLs=http://192.168.12.50:2379
987146e8925f10e5: name=etcd1 peerURLs=http://192.168.12.51:2380 clientURLs=http://192.168.12.51:2379

Poweroff one of the etcd members

vmrun stop /Users/kelseyhightower/Documents/Virtual\ Machines.localized/core0.vmwarevm/core0.vmx

vmrun list
Total running VMs: 2
/Users/kelseyhightower/Documents/Virtual Machines.localized/core1.vmwarevm/core1.vmx
/Users/kelseyhightower/Documents/Virtual Machines.localized/core2.vmwarevm/core2.vmx

The member list commands fails

etcdctl -C http://192.168.12.50:2379,http://192.168.12.51:2379,http://192.168.12.52:2379 member list
context deadline exceeded

The cluster is reported healthy, but no nodes are marked unhealthy even though member 7931e79c0d8b47c5 is powered off.

etcdctl -C http://192.168.12.50:2379,http://192.168.12.51:2379,http://192.168.12.52:2379 cluster-health
cluster is healthy
member 5ae3067007f7fb85 is healthy
member 7931e79c0d8b47c5 is healthy
member 987146e8925f10e5 is healthy

[yichengq]

This is basically the same one as what we met in 2.0.3: #2340

[mariusgrigaitis]

Can confirm this.

[durzo]

Running v2.1.0-alpha.1, shutting down a node still has cluster-health returning that all members are healthy.

2015/06/24 09:29:30 etcdserver: failed to reach the peerURL(http://etcd2:7001) of member 7a9767de17ea4500 (Get http://etcd2:7001/version: net/http: request canceled while waiting for connection)

root@etcd3:~$ etcdctl cluster-health
cluster is healthy
member 7a9767de17ea4500 is healthy
member cb1f485859524c11 is healthy
member d555fc8f72be9146 is healthy

[xiang90]

/cc @yichengq

[yichengq]

@durzo Did it changes to unhealthy finally? How long did you see this false healthy info?

So far, we know that the implementation has some delay(around minutes) on healthy status for hard-kill machine, and we plan to improve it in 2.2. Internal details are that etcd 2.0 sends MsgApp async on HTTP stream, which cannot reflect whether the receive side works.