Volume Encryption

[mysticaltech]

Discussed in #287

^{Originally posted by JustinGuese August 29, 2022}
do we have an idea of how to implement volume encryption?
any ideas how we could solve it?

[mysticaltech]

@JustinGuese, I will address this ASAP.

[mysticaltech]

@JustinGuese We very recently added support for Longhorn volumes and a custom values.yaml file to be added, please see the kube.tf.example.

And this is the how to add encryption, which is probably already possible now that you have complete flexibility over Longhorn's config.

Please try it, and let me know if all is good so we can close this!

[mysticaltech]

I will consider this resolved, but if you tried and it turned out not possible because we are missing something, just let me know, and we will reopen this.

[JustinGuese]

@mysticaltech did you have a thought about making this feature donatable?

also: I'm currently stuck at that the nodes need some libraries installed for longhorn and the encryption to work...
How can I do that? does it even work for an existing installation?

iscsiadm/open-iscsi
cryptsetup
and the dm_crypt has to be loaded

Thanks again for your help!

[mysticaltech]

@JustinGuese We basically just install Longhorn for you, but then up to you to configure it.

However, please have a look at this guide by @ifeulner, it touches on a lot of good subjects about how to use Longhorn optimally: https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b

To install additional packages, you can use the extra_packages_to_install variable which accepts an array of strings like so ["package1", "package2"...].

Then you to apply to terraform again and add an ideally creating new agent nodepools that will mirror your old nodepools. This will temporarily double the size of your cluster. But then you can decommission the old nodes draining them all and then changing their (old) nodepool count to 0 (don't delete them). This method is explained better in the readme.

However, before doing all of this, you better ssh into a node and try to install the above packages manually via transactional-update pkg install just to make sure that they have the same name. Note that MicroOS is a version of Tumbleweed, so when searching for a package on Google search for the name of the package followed by "OpenSUSE Tumbleweed" and that will give you the real name used. Confirming the presence of the package for Tumbleweed is probably enough.

[JustinGuese]

i don't want to be annoying, but maybe someone already did this?
i managed to install cryptsetup as stated by mysticaltech, and longhorn installs nicely and everything...
but it still seems to struggle (longhorn problem).

using the "default" longhorn volume everything works, but as soon as I create an encrypted storageclass as stated in their doc, it fails to mount that volume. it does not throw an error, and I think the issue is specific to microos, so I guess it must be some permission error, or a missing library in microos? anyone experienced something similar?

[ifeulner]

Haven't used encrypted volumes yet, but currently the default install is lacking at least cryptsetupas @mysticaltech stated. I would suggest to check the server logs via journalctl and also longhorn logging...

[JustinGuese]

yeah I'm only getting "disk is not where it was expected to mount", so basically something in the mount part seems to go wrong... Possibly related to the read-only filesystem?

[mysticaltech]

@JustinGuese /var and /etc are read-write and we mount network volumes at /var/longhorn.

[JustinGuese]

alright, so no read only filesystem issue I guess

[JustinGuese]

Okay I got it to work, some findings from my journey:

(preferred) hcloud csi encrypted volumes

The easiest way to get encrypted volumes working is actually to use the new encryption functionality of hcloud csi itself, see https://github.com/hetznercloud/csi-driver.
For this, you just need to create a secret containing the encryption key

   apiVersion: v1
   kind: Secret
   metadata:
     name: encryption-secret
     namespace: kube-system
   stringData:
     encryption-passphrase: foobar

and to create a new storage class

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: hcloud-volumes-encrypted
   provisioner: csi.hetzner.cloud
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   allowVolumeExpansion: true
   parameters:
     csi.storage.k8s.io/node-publish-secret-name: encryption-secret
     csi.storage.k8s.io/node-publish-secret-namespace: kube-system

in addition, you nodes need to have the package cryptsetup installed, which can be done by adding the line extra_packages_to_install = ["cryptsetup"] to the kube.tf file.

If you want to check if the encryption really works (which I can confirm), you can mount the volume using the hetzner console, and then open it with cryptsetup luksOpen /dev/sdb encryptedhetznervolume (might be different to sdb), and then mount /dev/mapper/encryptedhetznervolume /mnt

(not working) longhorn encryption

I tried this, but it did not work for me. For this, you need to install the cryptsetup package as well with the above command, or transactional-update pkg install cryptsetup in existing nodes, install longhorn, and create a new longhorn storage class and secret

apiVersion: v1
kind: Secret
metadata:
  name: longhorn-crypto
  namespace: longhorn-system
stringData:
  CRYPTO_KEY_VALUE: "Your encryption passphrase"
  CRYPTO_KEY_PROVIDER: "secret"

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-encrypted-global
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: "Delete" 
volumeBindingMode: Immediate 
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fromBackup: ""
  fsType: "ext4"
  dataLocality: "disabled"
  encrypted: "true"
  csi.storage.k8s.io/provisioner-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/provisioner-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-publish-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"

there seems to be a problem though, as somehow this volume will not get mounted on the node. Anyways, the hetzner volume csi way is better anyways

[ifeulner]

Need to find out why the longhorn-based setup does not mount (just tried it, fails for me, too, see output below ) cryptsetup should be included in the base installation, absolutely.

k logs longhorn-csi-plugin-lg74l longhorn-csi-plugin -n longhorn-system

time="2022-12-08T18:10:54Z" level=info msg="NodeStageVolume: req: {\"secrets\":\"***stripped***\",\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/997a17410f81802650d9697dbbd9d7f26df7d869557c047a2045b473ffa3d6b4/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":1}},\"volume_context\":{\"diskSelector\":\"nvme\",\"encrypted\":\"true\",\"fromBackup\":\"\",\"numberOfReplicas\":\"3\",\"staleReplicaTimeout\":\"2880\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1670507952420-8081-driver.longhorn.io\"},\"volume_id\":\"pvc-688fba46-aa28-46aa-b948-71f572ce298b\"}"
time="2022-12-08T18:10:54Z" level=info msg="volume pvc-688fba46-aa28-46aa-b948-71f572ce298b using user and longhorn provided ext4 fs creation params: -b4096"
time="2022-12-08T18:10:54Z" level=debug msg="volume pvc-688fba46-aa28-46aa-b948-71f572ce298b device /dev/longhorn/pvc-688fba46-aa28-46aa-b948-71f572ce298b contains filesystem of format crypto_LUKS"
time="2022-12-08T18:10:54Z" level=debug msg="volume pvc-688fba46-aa28-46aa-b948-71f572ce298b requires crypto device /dev/mapper/pvc-688fba46-aa28-46aa-b948-71f572ce298b"
time="2022-12-08T18:10:54Z" level=debug msg="device /dev/longhorn/pvc-688fba46-aa28-46aa-b948-71f572ce298b is already opened at /dev/mapper/pvc-688fba46-aa28-46aa-b948-71f572ce298b"
time="2022-12-08T18:10:54Z" level=debug msg="trying to ensure mount point /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/997a17410f81802650d9697dbbd9d7f26df7d869557c047a2045b473ffa3d6b4/globalmount"
time="2022-12-08T18:10:54Z" level=debug msg="mount point /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/997a17410f81802650d9697dbbd9d7f26df7d869557c047a2045b473ffa3d6b4/globalmount try reading dir to make sure it's healthy"
I1208 18:10:54.780338    2546 mount_linux.go:449] Disk "/dev/mapper/pvc-688fba46-aa28-46aa-b948-71f572ce298b" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/mapper/pvc-688fba46-aa28-46aa-b948-71f572ce298b]
E1208 18:10:54.781369    2546 mount_linux.go:455] format of disk "/dev/mapper/pvc-688fba46-aa28-46aa-b948-71f572ce298b" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/997a17410f81802650d9697dbbd9d7f26df7d869557c047a2045b473ffa3d6b4/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.43.8 (1-Jan-2018)
The file /dev/mapper/pvc-688fba46-aa28-46aa-b948-71f572ce298b does not exist and no size was specified.
)
time="2022-12-08T18:10:54Z" level=error msg="NodeStageVolume: err: rpc error: code = Internal desc = format of disk \"/dev/mapper/pvc-688fba46-aa28-46aa-b948-71f572ce298b\" failed: type:(\"ext4\") target:(\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/997a17410f81802650d9697dbbd9d7f26df7d869557c047a2045b473ffa3d6b4/globalmount\") options:(\"defaults\") errcode:(exit status 1) output:(mke2fs 1.43.8 (1-Jan-2018)\nThe file /dev/mapper/pvc-688fba46-aa28-46aa-b948-71f572ce298b does not exist and no size was specified.\n) "

Anyone an idea about this? (there seems to be an issue with multipathd, see here but this service is not installed on MicroOS)

[ifeulner]

if you are including cryptsetup anyways, we could maybe offer both storage classes by default? asking for the encryption password as variable (or does it get saved somewhere?)

It should work in general and we should provide an example, but as the user-specific configuration via storage-classes and encryptions keys (for hcloud-volumes and / or longhorn volumes) could quite vary, I don't think it makes sense to support this directly via the terraform setup.

[mysticaltech]

Great finding @ifeulner! Basically, it's completely doable to add multipath and start it too, as we do that for another service. I will add both this and cryptsetup. And fully agree that should be enough, to just add compatibility, later folks can configure it the way they want with yaml files, that's not up to us.

[ifeulner]

I think multipathd is only a problem IF it is installed, but it isn't in the current setup. So I don't think that this is our problem.

[mysticaltech]

Ah my bad, had misunderstood that part! Ok, good. So we still need to figure out why Longhorn with encryption is not mounting volumes.

[valkenburg-prevue-ch]

I may have nailed it. #638
Longhorn encryption works.

[ifeulner]

Great finding @valkenburg-prevue-ch ! Thanks! Will try it asap.