etcdctl: cluster-health reports has delay

[kelseyhightower]

etcd version: v2.0.3

How to reproduce:

• Start a 3 node etcd cluster.
• Kill a node (hard poweroff the machine -- no clean shutdown)
• run etcdctl cluster-health

Expected results:

etcd reports the kill node as unhealthy

Actual Results:

$ etcdctl cluster-health
cluster is healthy
member 5ae3067007f7fb85 is healthy
member 7931e79c0d8b47c5 is healthy 
member 987146e8925f10e5 is healthy <- This node is powered off

[kelseyhightower]

Seems after 5 mins or so, the cluster-health command reports the node unhealthy:

$ etcdctl cluster-health
cluster is healthy
member 5ae3067007f7fb85 is healthy
member 7931e79c0d8b47c5 is healthy
member 987146e8925f10e5 is unhealthy

[xiang90]

@kelseyhightower
The current detection is not very reliable since it is using the old 0.4.x endpoint. The member healthy checking is based on the underlying tcp connection. If you just power off the machine, the connection will not be dropped immediately.

This command achieves our current goal for detection the healthiness after migration.
We will improve it.

[philips]

What will the approach be on this? Are we going to add a threshold for RPC latency and then mark something as unhealthy at that point?

[xiang90]

I am moving this to 2.2.

This is more in line with our goal for 2.2: etcdctl uses etcd/client and new endpoints if possible.

[xiang90]

This is kind of duplicated with #2711 and @yichengq is working on it. So I am closing this one if favor of #2711.