Dealing with Hetzner CSI issues

[JustinGuese]

Description

Hey, we are experiencing a lot of Hetzner CSI issues. 3 clusters, all with the same problem since ~3 weeks... Did you update the Hetzner CSI plugin by chance?
What happens is:

1. Pod runs on node A with a hetzner volume
2. Pod is rescheduled to node B
3. Pod is stuck at "ContainerCreation" as the volume does not detach from Node A. This is particularly bad because the Pod is already terminated in Node A, meaning the service is unavailable

The solution is to either manually detach the volume in the Hetzner Dashboard, or restart Node A to force a detach, but of course requires manual intervention...

Kube.tf file

The default one, no longhorn

Screenshots

No response

Platform

Linux

[mysticaltech]

@JustinGuese Sorry to hear. The cluster itself does not update the extra components like this on its own, you would have to do that manually.

When installing a new cluster, we fetch the latest version from Github, but you can always pin to a specific version via hetzner_csi_version.

Basically, we cannot do anything about this, you need to open an issue with https://github.com/hetznercloud/csi-driver, and hit the Hetzner support telling them that their CSI is misbehaving! This is on them.

[mysticaltech]

@JustinGuese Also super useful would be to store and review the Hetzner CSI-associated pods logs! Without those, we are all blind.

More info about this is more than welcome for other affected users!

[hypery2k]

I'm having a similar issue. Seems that the wrong node is selected:

- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      volume.beta.kubernetes.io/storage-provisioner: csi.hetzner.cloud
      volume.kubernetes.io/selected-node: production-agent-large-srd
      volume.kubernetes.io/storage-provisioner: csi.hetzner.cloud
    creationTimestamp: "2023-03-24T05:54:30Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app.kubernetes.io/component: primary
      app.kubernetes.io/instance: pcf-app
      app.kubernetes.io/name: postgresql
    name: data-pcf-app-postgresql-0
    namespace: pen-testing
    resourceVersion: "4164426"
    uid: 0c39bdac-5540-4a34-b274-151a6409cdbf
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 8Gi
    storageClassName: hcloud-volumes
    volumeMode: Filesystem
  status:
    phase: Pending
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      meta.helm.sh/release-name: reconmap-app
      meta.helm.sh/release-namespace: pen-testing
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: csi.hetzner.cloud
      volume.kubernetes.io/selected-node: production-storage-yhq
      volume.kubernetes.io/storage-provisioner: csi.hetzner.cloud
    creationTimestamp: "2023-03-22T08:09:04Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: mysql
      app.kubernetes.io/managed-by: Helm
    name: reconmap-app-mysql-pv-claim
    namespace: pen-testing
    resourceVersion: "3367563"
    uid: e355ac30-2136-4193-8264-04e33bc335c8
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
    storageClassName: hcloud-volumes
    volumeMode: Filesystem
    volumeName: pvc-e355ac30-2136-4193-8264-04e33bc335c8
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 20Gi
    phase: Bound

Seeing this logs:

17m         Normal    WaitForFirstConsumer   persistentvolumeclaim/data-pcf-app-postgresql-0   waiting for first consumer to be created before binding
12m         Normal    ExternalProvisioning   persistentvolumeclaim/data-pcf-app-postgresql-0   waiting for a volume to be created, either by external provisioner "csi.hetzner.cloud" or manually created by system administrator
12m         Normal    Provisioning           persistentvolumeclaim/data-pcf-app-postgresql-0   External provisioner is provisioning volume for claim "pen-testing/data-pcf-app-postgresql-0"
12m         Warning   ProvisioningFailed     persistentvolumeclaim/data-pcf-app-postgresql-0   failed to provision volume with StorageClass "hcloud-volumes": error generating accessibility requirements: no topology key found on CSINode production-agent-large-srd
10m         Normal    WaitForFirstConsumer   persistentvolumeclaim/data-pcf-app-postgresql-0   waiting for first consumer to be created before binding
6s          Normal    ExternalProvisioning   persistentvolumeclaim/data-pcf-app-postgresql-0   waiting for a volume to be created, either by external provisioner "csi.hetzner.cloud" or manually created by system administrator
61s         Normal    Provisioning           persistentvolumeclaim/data-pcf-app-postgresql-0   External provisioner is provisioning volume for claim "pen-testing/data-pcf-app-postgresql-0"
61s         Warning   ProvisioningFailed     persistentvolumeclaim/data-pcf-app-postgresql-0   failed to provision volume with StorageClass "hcloud-volumes": error generating accessibility requirements: no topology key found on CSINode production-agent-large-srd

[hypery2k]

When updating manually the volume.kubernetes.io/selected-node annotation to production-storage-yhq it's working

[mysticaltech]

@hypery2k Probably a good idea to open an issue with https://github.com/hetznercloud/csi-driver. And personally, a lot of users use Longhorn without issues, it probably is more robust than Hetzner CSI.

[hypery2k]

thanks, putting the infos here: hetznercloud/csi-driver#400