Ignore namespaces when dealing with ClusterRole or ClusterRoleBindings

[Bhashit]

While installing Istio with CNI, I had a manifest generated using istioctl. The manifest incorrectly creates a ClusterRoleBinding that specifies a namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-cni-repair-rolebinding
  namespace: kube-system
  labels:
    k8s-app: istio-cni-repair
subjects:
- kind: ServiceAccount
  name: istio-cni
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-cni-repair-role

This refers to another ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-cni-repair-role
  labels:
    app: istio-cni
    release: istio
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch", "delete", "patch", "update" ]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["get", "list", "watch", "delete", "patch", "update", "create" ]

Because the ClusterRoleBinding specifies a namespace, it looks like this provider is trying to look for a resource in that namespace. It fails with the following error.

module.cluster-services.kustomization_resource.current["rbac.authorization.k8s.io_v1_ClusterRoleBinding|kube-system|istio-cni-repair-rolebinding"]: Creating...
 Error: ResourceCreate: creating 'rbac.authorization.k8s.io/v1, Resource=clusterrolebindings' failed: the server could not find the requested resource

This may be an easy fix, and I'll attempt to send a PR once I have fixed this issue from happening in my CI pipeline.

[pst]

Thanks for reporting. I've seen this error a couple of times. The solution is to not send a namespaced ClusterRolebinding, which is as you note invalid. The error seems to come from the kustomizationResourceCreate method.

Kubectl, from what I understand, silently removes the namespace for non namespaced resources. But I don't think that's the correct approach here. The provider should apply exactly what is specified in the kustomization. Not some magically altered version of it.

This error most likely only surfaces during apply, not plan. So #24 may help.

[riccardomc]

The same thing happens for manifests generated by the prometheus-operator helm chart and PodSecurityPolicy resources.

However, I believe the problem is in the chart. Since these resources are cluster scoped shouldn't not have a namespace field to begin with. I will file an issue to the prometheus-operator helm chart.

I think @Bhashit, you should do the same for istio.

[riccardomc]

FYI: helm/charts#22946

[Bhashit]

Kubectl, from what I understand, silently removes the namespace for non namespaced resources. But I don't think that's the correct approach here. The provider should apply exactly what is specified in the kustomization. Not some magically altered version of it.

@pst Maybe one last argument: shouldn't the tools that kinda replace kubectl should also work like kubectl?

[pst]

I do understand your argument @Bhashit, but writing a Terraform provider I am in between two worlds. And this kubectl behavior does not seem acceptable in a Terraform provider to me.

If the Prometheus helm chart mentioned by @riccardomc and istioctl generate invalid Kubernetes manifests, then fixing that seems the better approach to me.

[Bhashit]

Got it. I understand. I'll close the issue