Error: Post https://xxxx.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp xxx:443: i/o timeout

[max-rocket-internet]

On every example on first run:

Error: Post https://C39B1571479D46ED13A85BC0105510E2.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp 44.229.104.205:443: i/o timeout

  on ../../aws_auth.tf line 45, in resource "kubernetes_config_map" "aws_auth":
  45: resource "kubernetes_config_map" "aws_auth" {

This is because theres a small between when the cluster endpoint is returned after creation but is not available.

If you rerun terraform apply the configmap is created fine.

Quite annoying 😅

[dpiddockcmp]

Is this an EKS bug that's worth reporting to AWS container team, or an interplay between the aws and kubernetes providers?

[xposix]

I keep on having the same issue. IMO, the kubernetes provider should wait / retry few times before coming back with an error.

[max-rocket-internet]

Related: hashicorp/terraform-provider-kubernetes#96

[max-rocket-internet]

Is this an EKS bug that's worth reporting to AWS container team, or an interplay between the aws and kubernetes providers?

I think a socket error should be retried by the provider.

In the mean time, could we make a work around somehow? Like a null resource that sleeps for 10 seconds after cluster creation?

[ivanich]

I'm using kubectl hack for this but it requires kubectl and a lot of depends_on and in general it ugly

  provisioner "local-exec" {
    command = "until kubectl --kubeconfig ${path.root}/kubeconfig_${var.cluster_id} -n kube-system get pods >/dev/null 2>&1;do echo 'Waiting for EKS API';sleep 5;done"
  }
}

in the meantime I'm going to fork eks module with kubernetes provider commits reverted because it's not usable for our usecases.

[max-rocket-internet]

in the meantime I'm going to fork eks module with kubernetes provider commits reverted because it's not usable for our usecases.

You can just use release v7.0.1, no?

[ivanich]

You can just use release v7.0.1, no?

7.0.1 has faulty kubernetes provider commit, so staying with 7.0.0 for now, and if it's will be an issue for a while fork is the way to go at least we can have new features provided by upstream.

[max-rocket-internet]

7.0.1 has faulty kubernetes provider commit

No it doesn't 🙂 I made it specifically the commit before: #630

https://github.com/terraform-aws-modules/terraform-aws-eks/releases/tag/v7.0.1

[ivanich]

No it doesn't 🙂 I made it specifically the commit before: #630

Nice, thanks for the tip, haven't checked commits since kubernetes provider merge. 👍

[dpiddockcmp]

This is definitely a "feature" of the AWS EKS API. It returns creation complete before the cluster is actually ready to be used 😞

This requires all tools creating clusters to have a retry on their first access. e.g.:

• eksctl's WaitForControlPlane
• logic in the "quickstart" example run_command
• v7 of this module local_exec

We need to go back to having a retry loop hack. Which kind of defeats the point of moving to the kubernetes provider 😞

I was comparing with GCP's kubernetes offering. Their service only returns ready once the cluster is actually usable. You can easily chain GKE creation with the kubernetes terraform provider.
Test code here

Raised an issue in the containers roadmap here . See what they say

[barryib]

FYI, I opened a PR in the AWS provider to wait for kubernetes endpoint hashicorp/terraform-provider-aws#11426. I don't know if it the right quick win before aws/containers-roadmap#654 get solved. Feedbacks are welcome.

[dpiddockcmp]

Hit another issue with this damn provider when fiddling about with node_groups:
EKS Node Groups modify, and if necessary, create the aws-auth ConfigMap. This causes the kubernetes provider to always fail if you create the cluster and node groups from scratch in one run. The k8s provider needs resources to not exist or to be imported. There's no force create option.

Can we go back to kubectl and forget this whole exercise?

Edit: Raised an issue with the k8s provider. See what they say. hashicorp/terraform-provider-kubernetes#719