Volume cannot be mounted on Pod (iSCSI login failure)

[benjamin-gentner-fnt]

From time to time we receive the following error message during the startup of a Pod when mounting the volume. We're using the Synology CSI driver. Access mode is ReadWriteOnly for the PVC. The issue can be fixed by manual intervention when accessing the node which still has the volume mounted and then unmounting this volume. The root cause of the issue might be related to a situation when 2 or more Pods at the same time try to mount the volume which is not possible. Deleting the Pod or scaling down all pods won't unmount the volume of the old node in probably 10% of the cases. Then scaling up the Pod again shows the issue as shown below:

Events:
Type Reason Age From Message

---

Warning FailedMount 22m (x28 over 67m) kubelet MountVolume.MountDevice failed for volume "pvc-2b9b73b4-049f-46e5-abbc-27f92eb71ca9" : rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:nasellvicloud.pvc-2b9b73b4-049f-46e5-abbc-27f92eb71ca9], err: iscsiadm: Could not login to [iface: default, target: iqn.2000-01.com.synology:nasellvicloud.pvc-2b9b73b4-049f-46e5-abbc-27f92eb71ca9, portal: 172.27.2.211,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
Logging in to [iface: default, target: iqn.2000-01.com.synology:nasellvicloud.pvc-2b9b73b4-049f-46e5-abbc-27f92eb71ca9, portal: 172.27.2.211,3260]
(exit status 19)
Warning FailedMount 2m22s (x29 over 65m) kubelet Unable to attach or mount volumes: unmounted volumes=[minio], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition

[bplein]

You mean ReadWriteOnce.

If you have more than one node, you should not be using RWO PVCs with an app that has many pods using the same PVC which is what you seem to be describing.

I may have been misunderstanding your problem statement.

[benjamin-gentner-fnt]

No, we want to mount the RWO PVCs from just one node. But sometimes the issue might happen when there are 2 pods using the same PVC on different nodes during a Pod's RollingUpdate or ReplicaSet's re-spawn of the pod (when the pod is deleted and recreated by the ReplicaSet). Then the new Pod cannot start which is valid, but sometimes when we scale down all pods, then start one pod again, we get into the mentioned issue and receive the error message from above.

Apart from that, I guess it would be good to change either to a StatefulSet (from Deployment) or to update the updateStrategy (ensuring there can be just one pod at a time)