External ceph connection is requiring more permissions that needed for Ceph CSI

[gustavosr98]

Summary

Microk8s integration with external Ceph using ceph-rook add-on is requiring more permissions that the ones needed for Ceph CSI

Thus, I am ending up using root admin credentials which turns into a security concern

What Should Happen Instead?

The connection should be able to work just using RBD role permissions rather than root.
The same way that it is described on Ceph CSI documentation
And the same way that works for LXD integration with Ceph just with RBD role permissions to a target pool

Reproduction Steps

# A - RBD only access credentials
sudo microk8s connect-external-ceph \
  --ceph-conf /var/snap/microceph/current/conf/ceph.conf \
  --keyring /var/snap/microceph/current/conf/ceph.client.management-microk8s.keyring \
  --rbd-pool management-microk8s \
  --no-rbd-pool-auto-create

Creating namespace rook-ceph-external
namespace/rook-ceph-external created
Configuring Ceph CSI secrets
Traceback (most recent call last):
  File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 184, in <module>
    main()
  File "/snap/microk8s/6529/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/snap/microk8s/6529/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/snap/microk8s/6529/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/microk8s/6529/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 169, in main
    import_external_ceph_cluster(ceph_conf, keyring, namespace, rbd_pool)
  File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 109, in import_external_ceph_cluster
    p = subprocess.run(
  File "/snap/microk8s/6529/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/snap/microk8s/6529/usr/bin/python3', PosixPath('/var/snap/microk8s/common/plugins/.rook-create-external-cluster-resources.py'), '--format=bash', '--rbd-data-pool-name=management-microk8s', '--ceph-conf=/var/snap/microceph/current/conf/ceph.conf', '--keyring=/var/snap/microceph/current/conf/ceph.client.management-microk8s.keyring']' returned non-zero exit status 1

# Same permissions required for Ceph CSI
# https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/
cat /var/snap/microceph/current/conf/ceph.client.management-microk8s.keyring 
[client.management-microk8s]
        key = HERE_IS_MY_KEY
        caps mgr = "profile rbd pool=management-microk8s"
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=management-microk8s"

# B - Ceph Admin / Root credentials (Which I cannot use for production)
sudo microk8s connect-external-ceph   \
  --ceph-conf /var/snap/microceph/current/conf/ceph.conf   \
  --keyring /var/snap/microceph/current/conf/ceph.keyring   \
  --rbd-pool $MICROK8S_CEPH_POOL   \
  --no-rbd-pool-auto-create

Creating namespace rook-ceph-external
Error from server (AlreadyExists): namespaces "rook-ceph-external" already exists
Configuring Ceph CSI secrets
Successfully configured Ceph CSI secrets
Importing Ceph CSI secrets into MicroK8s
secret/rook-ceph-mon created
configmap/rook-ceph-mon-endpoints created
secret/rook-csi-rbd-node created
secret/rook-csi-rbd-provisioner created
storageclass.storage.k8s.io/ceph-rbd created
Importing external Ceph cluster

=================================================

Successfully imported external Ceph cluster. You can now use the following storageclass
to provision PersistentVolumes using Ceph CSI:

NAME       PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
ceph-rbd   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   0s