mayastor-io-engine pods stuck on error 403 trying to create diskpools

[przemeklal]

Summary

On my setup all 3 mayastor-io-engine pods got stuck in Init:2/3, hitting error 403 when trying to create diskpools.
I ran the curl command from the init pod manually and got:

# curl --cacert "$CACERT" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -XPOST -d "$BODY" "https://kubernetes.default.svc/apis/openebs.io/v1alpha1/namespaces/$NAMESPACE/diskpools?fieldManager=kubectl-create"
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "diskpools.openebs.io is forbidden: User \"system:serviceaccount:mayastor:default\" cannot create resource \"diskpools\" in API group \"openebs.io\" in the namespace \"mayastor\"",
  "reason": "Forbidden",
  "details": {
    "group": "openebs.io",
    "kind": "diskpools"
  },
  "code": 403
}

After modifying the clusterrolebinding like this:

kubectl edit clusterrolebindings.rbac.authorization.k8s.io mayastor-io-engine-sa-cluster-role-binding
...
# added to subjects:
...
- kind: ServiceAccount
  name: default
  namespace: mayastor

It got automatically unblocked, all io-engine 3 pods are now running and diskpools were created correctly.

What Should Happen Instead?

It shouldn't get stuck, it should use the right service account and RBAC rules should be correct.

Reproduction Steps

1. Deploy microk8s 1.27.4 cluster (3 nodes) using microk8s latest/edge charm rev 115.
2. Enable addons: dns ingress rbac metallb
3. sudo microk8s addons repo add core --force https://github.com/canonical/microk8s-core-addons --reference 1.27
4. sudo microk8s enable core/mayastor --default-pool-size 900G
5. Watch: sudo microk8s get pods -n mayastor and microk8s kubectl logs -n mayastor mayastor-io-engine-nwxvc initialize-pool

I also tried disabling and enabling mayastor addon but then I ran into the same issue again.

Introspection Report

inspection-report-20230809_074438.tar.gz

Can you suggest a fix?

If io-engine pods could use mayastor-io-engine-sa SA instead of default that would fix the issue I believe.
Alternatively, what I did with clusterrolebinding can also work.

Are you interested in contributing with a fix?

No.

[neoaggelos]

Should be closed by #225 and #226