Rotating cloud instances with PVCs in a StatefulSet

[joekohlsdorf]

Online you can find a bunch of examples (even in the official docs) which show how to use the local-volume-provisioner in combination with PhysicalVolumeClaims in a StatefulSet.

All works fine until a node goes away and your cloud provider brings up a new one, be it due to an issue on their side or due to you bringing up some new nodes because you are upgrading K8s.

What happens in this case is that the PVC stays bound to a PV which no longer exists. This prohibits the pod in the StatefulSet from coming up until you manually delete the PVC. Now this makes sense because there is no way of knowing if the node was shut down for maintenance and if it will come back later or if it's gone forever.

However I'd just like the node to be assumed dead because I'm never going to reboot nodes intentionally, I'll just roll the cluster. If the pod can be scheduled on another node I know 100% that the node was replaced (due to my affinity settings).
Is there any official way of dealing with this or any config option I'm overseeing?

I can write a job which takes care of this but surely others must have hit this issue?!

[nerddelphi]

@joekohlsdorf I guess we are facing the same issue: #65 (comment)

How do you plan solve that?

[cofyc]

There is no Kubernetes-official way right now because Kubernetes will not unbind or delete PVCs. It's up to the users to recover from this situation. I have a plan to write a cloud controller to handle this automatically.

When a new node with a different name is created to replace the old node (e.g. auto-scaling group in AWS), PVs belonging to the old node are invalid. PVCs must be deleted, then the scheduler can find feasible PVs on other nodes. By the way, if pods have already been recreated on node deletion and are stuck at pending, they must be recreated to trigger StatefulSet to create PVCs again.

In GKE, this is a little different because the managed instance group recreates the underlying instance but uses the old node name.

[joekohlsdorf]

What I did is I wrote a janitor which every 20 seconds looks for pending pods which have PVCs bound to PVs on dead hosts and will remove the PVC if necessary. It then deletes the pending pod to get it scheduled again.

My nodes for this service are static and have labels, this way I can be sure that the host isn't just rebooting. I know that my service runs on X nodes but if I see X nodes online and a PV on a node that doesn't exist I know it's dead and no coming back.

If this doesn't happen on GKE maybe some workaround could be found with custom node tags. You could have an ASG for every node so tags would stay the same even if a node dies.

[nerddelphi]

Thank for your answers, @cofyc and @joekohlsdorf .

@joekohlsdorf could you provide your janitor with us? I'll be glad if you can.

[msau42]

@NickrenREN may have written a similar controller in the past

[joekohlsdorf]

Well I certainly would strongly advise against doing what I did but here is the unedited janitor I hacked up. Please only use it as a reference, I had to get this done in a time crunch.
https://gist.github.com/joekohlsdorf/2658f03b1e1b6194ebe6b61bd8770647

[nerddelphi]

Hi, @joekohlsdorf . Thank you for script.

[NickrenREN]

There is an issue that is similiar to this.
Some guys and I propose to introduce NodeFencing to solve this because it suit for both Cloud Providers and Bare metals and the reaction is relatively simple.
But others decide to take NodeShutdown taint method, there is an ongoing proposal: kubernetes/enhancements#1116.

Actually we have implemented NodeFencing feature (external controller and agent) in our own production environment.

[nerddelphi]

@NickrenREN Are you using that implementation https://github.com/kvaps/kube-fencing ? If yes, what kind of agent to dealing with PV/PVC issues?
My clusters are on GKE.

Thank you.

[NickrenREN]

@nerddelphi No, our fencing controller and agent are implemented by ourselves.
Agent is designed to shut down machines forcefully, the control logic, race conditions and cleanup work are done by controller.

[NickrenREN]

The design above is for bare metals, and for cloud providers, it may be a little bit different

[rsoika]

I am sorry that I am entering this discussion even though I am not a Kubernetes expert as you.
But I have been dealing with this problem for a some weeks and I also followed this long running discussion.

I am running a simple Kubernetes Cluster with only a view Nodes. I guess this is a complete different environment as that ones you discuss here, but let me describe my scenario to give you different view on the problem:

• I have setup a distributed storage based on Ceph or Longhorn (same behavior for both).
• I deploy a PostgresDB using a persistence volume claim.
• I kill (for testing) the node on which the Database POD is running
• Now I run into this problem that Kubernetes gets stuck while restarting the database POD on a new node, because the broken POD get not detached from the volume.
• I have to manually delete the volumeattachment to get rid of this situation

I understand all your concerns about the data and what can happen to it if a volume is automatically detached.
But I - as the administrator of my cluster - trust in my Longhorn or Ceph Cluster. And of course, something can always go wrong, but that's my job to secure my data.

From my point of view, it is not Kubernetes' job to interfere in my data management. PLEASE give us a switch with which we can switch off this behavior and get terminating pods detached from a volume.

[NickrenREN]

@rsoika Thanks for your input.
IIUIC, your scenario is the case NodeFencing can solve. If the node is dead (or Unknown), it will be forced to shut down and we do not expect it to be back again. As you described, data management isn't kubernetes' job, so the reaction is easy: go ahead and detach the volume forcefully.
And of course, if you want to bring you node back, you need to do the cleanup work first (this is also the work of kubernetes relevant team).

[rsoika]

@NickrenREN Thanks for your clarification. So there is no self-healing mechanism in Kubernetes for this scenario?