github.com/kbst/terraform-provider-kustomize/kustomize.kustomizationResourceCreate > timed out waiting for: <resource>: the server has asked for the client to provide credentials

[europ]

SUMMARY

Unable to apply configuration via kbst/kustomization using kubeconfig_raw option.

Apply for any Kubernetes object type managed by kustomization results in an error. Provider kustomization has an issue with kubeconfig processing.

error example (CLICK TO SHOW)

Error: github.com/kbst/terraform-provider-kustomize/kustomize.kustomizationResourceCreate: "rbac.authorization.k8s.io/ClusterRole/_/AAAAAA": timed out waiting for: "rbac.authorization.k8s.io/v1, Kind=ClusterRole": the server has asked for the client to provide credentials

   with module.BBBBBB.kustomization_resource.main["rbac.authorization.k8s.io/ClusterRole/_/AAAAAA"],
   on ../../../modules/BBBBBB/main.tf line 93, in resource "kustomization_resource" "main":
   93: resource "kustomization_resource" "main" {

This issue is similar to #90, where the explicit definition of context provided a sufficient workaround for the given issue. In our case, this workaround does not work.

Provider kustomization did not work with explicit kubeconfig_raw and context definition.

provider "kustomization" {
  kubeconfig_raw = "aaa" # kubeconfig example is included below
  context        = "bbb"
}

terraform-provider-kustomization/kustomize/provider.go

Lines 90 to 95 in 0b49602

	if raw != "" {
	config, err = getClientConfig([]byte(raw), context)
	if err != nil {
	return nil, fmt.Errorf("provider kustomization: kubeconfig_raw: %s", err)
	}
	}

DETAILS

Versions

• registry.terraform.io/kbst/kustomization 0.9.4
• registry.terraform.io/oracle/oci 5.1.0
• terraform 1.4.6
• terragrun 0.47.0

Use case

• Kubernetes cluster 1.24.1 hosted in Oracle Cloud Infrastructure (OCI)

• Kubeconfig is generated via OCI

  kubeconfig content (CLICK TO SHOW), that is assigned to "kubeconfig_raw" parameter

  ---
  apiVersion: v1
  kind: ""
  clusters:
    - name: cluster_name_example
      cluster:
        server: https://0.1.2.3:4
        certificate-authority-data: xxxxxxxxxxxxxxxx
  users:
    - name: user_name_example
      user:
        exec:
          apiVersion: client.authentication.k8s.io/v1beta1
          command: oci
          args:
            - ce
            - cluster
            - generate-token
            - --cluster-id
            - ocid1.cluster.oc1.reg.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
            - --region
            - region_name_example
            - --profile
            - profile_name_example
          env: []
  contexts:
    - name: context_name_example
      context:
        cluster: cluster_name_example
        user: user_name_example
  current-context: context_name_example

Ideas

Is it possible that this specific kubeconfig type is not supported? E.g. included command execution in kubeconfig (see /users/0/user/exec/command) can be a problem to the kustomization provider?

[pst]

I've used the provider with exec before. so I don't think it's that. And timed out doesn't sound like an error thrown when client-go can't read the kubeconfig.

Does your kubeconfig work with kubectl? What happens if you call the command specified in the exec manually?

[europ]

FYI: Our Terraform code uses Kubernetes provider and Kustomize provider with the exact same Kubeconfig. The issue occurs only in the Kustomize provider.

Does your kubeconfig work with kubectl?

Yes, it works. It works with Kubernetes provider as well.
Kubernetes provider URL: https://registry.terraform.io/providers/hashicorp/kubernetes/2.22.0

What happens if you call the command specified in the exec manually?

I will recheck it and let you know. However, we have dozens of Kubernetes resources managed via Kubernetes provider, which works properly. I think it will work, but I will double check just to be sure.

[europ]

Does your kubeconfig work with kubectl?

Yes, it works as expected. I have verified it and it works fine with the following tools:

1. kubectl
2. terraform helm provider
3. terraform kubernetes provider

What happens if you call the command specified in the exec manually?

It works and returns a valid token. Command is defined here: https://docs.oracle.com/en-us/iaas/tools/oci-cli/3.29.4/oci_cli_docs/cmdref/ce/cluster/generate-token.html. Here (below) is an result example:

{
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "kind": "ExecCredential",
    "status": {
        "token": "xxxxxxxxxxxxxxxxxxx",
        "expirationTimestamp": "2023-07-20T07:26:27.953812Z"
    }
}

SUMMARY: We can totally exclude possible kubeconfig misconfiguration. Kubeconfig is definitely valid.

[godeater]

For what it's worth - I'm seeing this issue now as well - with a kubectl generated by the aws cli.

kubeconfig content

``` apiVersion: v1 clusters: - cluster: certificate-authority-data: *** server: https://1.2.3.4 name: cluster_name_example contexts: - context: cluster: cluster_name_example user: user_name_example name: context_name_example current-context: context_name_example kind: Config preferences: {} users: - name: user_name_example user: exec: apiVersion: client.authentication.k8s.io/v1beta1 args: - --region - ap-southeast-2 - eks - get-token - --cluster-name - cluster_name_example - --output - json command: aws ```

It's giving this error:

Error: github.com/kbst/terraform-provider-kustomize/kustomize.kustomizationResourceExists: "apiextensions.k8s.io/CustomResourceDefinition/_/awxrestores.awx.ansible.com": timed out waiting for: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition": an error on the server ("getting credentials: exec: executable aws failed with exit code 1") has prevented the request from succeeding

And the same as the other reporter it works fine with:

• kubectl
• terraform helm provider
• terraform kubernetes provider

Versions:

• registry.terraform.io/kbst/kustomization 0.9.5
• registry.terraform.io/hashicorp/aws 5.24.0
• terraform v1.6.6

Use case

• Kubernetes cluster 1.28 running on Amazon EKS
• Kubeconfig is generated by aws eks update-kubeconfig

[pst]

"getting credentials: exec: executable aws failed with exit code 1"

If the CLI used in the exec part does error, I do not think this is an issue with the provider. I will close this issue now.