[EKS] [bug] Removed kube-apiservers return 401 Unauthorized instead of closing connection

[mhulscher]

Around the time of the release of EKS 1.23 we started noticing that EKS is more aggressively scaling out/in its kube-apiservers. We are seeing them being replaced more frequently. What we also notice is that whenever a kube-apiserver is removed (it no longer appears in kubectl get endpoints kubernetes -n default) it doesn't close existing connections. Instead, whenever a client makes a request to this removed kube-apiserver over an existing connection, the kube-apiserver returns a 401 Unauthorized. This seems to happen every time a kube-apiserver is scaled down. Applications might not be triggered by this 401 Unauthorized to re-establish their connection to the kube-apiserver. Instead, they might think that certain API resources are not available and act accordingly. This happens for example with the latest release of cilium.

I believe that whenever a kube-apiserver is removed as an endpoint, it should also immediately close all of its client connections; forcing the clients to establish a new connection.

Related issue: cilium/cilium#20915

[mhulscher]

I wanted to provide a small update. It seems that since about a week ago, kube-apiservers that are in the process of being removed, still accept client requests while the connection from the kube-apiserver to etcd is already gone. What I'm also seeing is in-flight requests not being gracefully handled. This leads to clients reporting errors.

Clients can work around this by retrying failed requests, but it strikes me as "not nice" that kube-apiserver is still accepting requests, even though it can't possibly handle them. As a result, we are now forced to add more and more retry logic to our e2e testing of our EKS clusters. This is a good practice regardless, but I think EKS can improve here.

This as well, only started happening around the time of the EKS 1.23 release.

When trying to create a namespace:

Trying to delete a pod:

Applying a statefulset:

[kanor1306]

It would be nice to get an explanation from the EKS team to know whether this is something that must be expected when dealing with the EKS kube-api. It really is a weird situation having to retry 401s to prevent issues with the availability of a rotating/scaling API.