CrashLoopBackOff after draining node

[mkhazamipour]

Hi,
As we planned to upgrade nodes on cluster, after draining nodes with --ignore-daemonsets and --delete-local-data switches, we are getting CrashLoopBackOff on csi plugin

[upgrade/versions] Cluster version: v1.13.1
[upgrade/versions] kubeadm version: v1.13.1
[upgrade/versions] Latest stable version: v1.13.1
[upgrade/versions] Latest version in the v1.13 series: v1.13.1

kubectl logs csi-hc-node-j4vfx -n kube-system -c csi-hc-plugin                 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xc0 pc=0x85840b]

goroutine 1 [running]:
github.com/zenjoy/csi-hetzner/driver.NewDriver(0x7ffd880f62dc, 0x14, 0x7ffd880f62f9, 0x40, 0x7ffd880f6340, 0x1a, 0xc000055260, 0x0, 0x0)
        /Users/peter/Development/Go/src/github.com/zenjoy/csi-hetzner/driver/driver.go:91 +0x25b
main.main()
        /Users/peter/Development/Go/src/github.com/zenjoy/csi-hetzner/cmd/hc-csi-plugin/main.go:43 +0x19f

Deleting whole csi plugin and recreating it didnt help.
Thank you

[mkhazamipour]

With further investigation it seems CSI on hetzner is not compatible with v1.13+ yet.
More information : digitalocean/csi-digitalocean#113
Any ETA for update ?

[dedene]

I'm unfortunately not able to reproduce the issue yet. I setup a test cluster on v0.12.3, installed the CSI Driver, upgrade to k8s v0.13.1 and played around with draining and undraining nodes: no issues with both the CSI controller as the nodes using PVCs.

Also setting up directly on a v0.13.1 cluster does not seem to give any issues.

I'm wondering, as zenjoy/csi-hetzner/driver/driver.go:91 (https://github.com/zenjoy/csi-hetzner/blob/master/driver/driver.go#L91) points to a part in the code related to a response from the Hetzner API: maybe there was a glitch in the API? Because it's very weird to get a nullpointer exception at that point in the code unless the Hetzner API returned no results when querying for the server details?

Btw, @thcyron from Hetzner confirmed they will release an officially supported CSI Driver after the holidays: apricote/hcloud-csi-driver#1 (comment) It might be a good idea at that point to migrate to using their driver as this was only meant as a simple port of the DigitalOcean driver to get started and do some testing during the Hetzner Volumes Beta.

[mkhazamipour]

That was a problem with Hetzner API as you said.
Thank you