Module gke-cluster with "forces replacement" due to deletion of default node pool

[shashlikxp]

My versions:

Terraform v1.4.1
+ provider registry.terraform.io/hashicorp/google v4.58.0
+ provider registry.terraform.io/hashicorp/google-beta v4.58.0

I used the Foundation Fabric gke-cluster module to create a simple gke cluster without autopilot feature: in the module

   enable_features = {
      autopilot = false   # this is also the default value
      # using only 2 features, irrelevant
      vertical_pod_autoscaling = true
      workload_identity = true
   }

After successfully provisioning the resources with the first terraform apply, running a subsequent terraform plan command yields the following result which forces replacement due to spot and preemptible options for the default nodepool:

      + node_config { # forces replacement
          + disk_size_gb      = (known after apply)
          + disk_type         = (known after apply)
          + guest_accelerator = (known after apply)
          + image_type        = (known after apply)
          + labels            = (known after apply)
          + local_ssd_count   = (known after apply)
          + logging_variant   = "DEFAULT"
          + machine_type      = (known after apply)
          + metadata          = (known after apply)
          + min_cpu_platform  = (known after apply)
          + oauth_scopes      = (known after apply)
          + preemptible       = false # forces replacement
          + service_account   = (known after apply)
          + spot              = false # forces replacement
          + taint             = (known after apply)
        }

In the Foundation Fabric modules\gke-cluster\main.tf I see the following block:

resource "google_container_cluster" "cluster" {
  lifecycle {
    ignore_changes = [
      node_config[0].boot_disk_kms_key,
      node_config[0].spot
    ]
  }

Adding a node_config[0].preemptible to the list doesn't help. However, rewriting to

lifecycle {
    ignore_changes = [
      node_config
    ]
  }

works perfectly and solves the problem.

I am wondering, why the node_config is created here at all. The reason is that it is created here:

  # the default nodepool is deleted here, use the gke-nodepool module instead
  # default nodepool configuration based on a shielded_nodes variable
  dynamic "node_config" {
    for_each = var.enable_features.autopilot ? [] : [""]
    content {
      dynamic "shielded_instance_config" {
      ...

So negating the condition

  dynamic "node_config" {
    for_each = !var.enable_features.autopilot ? [] : [""]
    ...

also helped. Is this a bug in the gke-cluster/main.tf or am I doing something wrong in my module configuration?

[shashlikxp]

Info: I have no nodepools enabled at that moment - just a plain control plane.

[ludoo]

Thanks for reporting this, we might be at the point where we split the gke module into a standar and an autopilot one. Julio and I are discussing this right now.

[vkaukeano-flexion]

@ludoo

I am still having this issue with the split standard gke module.
After running terraform apply, when running terraform apply for other terraform modules I receive a plan that it will destroy and create the cluster.

[ludoo]

Ok let's look into this. Thanks for reporting!

[ludoo]

@vkaukeano-flexion can you paste here the module configuration, so we can reproduce exactly?

[ludoo]

node_config has been there forever, we just added code to skip it for autopilot clusters in #1126 I have no idea why it now triggers recreation, let's try and deep dive on it

[vkaukeano-flexion]

@vkaukeano-flexion can you paste here the module configuration, so we can reproduce exactly?

@ludoo ,

Sorry, I am unable to provide configurations due to a contract. However, I am using the nodepool and service account module with the cluster-standard module.

When I set the enable secure boot flag to false it no longer destroys and recreates the cluster with every apply.

[ludoo]

One example of a similar issue from one of our colleagues

module "cluster" {
  source     = "github.com/GoogleCloudPlatform/cloud-foundation-fabric/modules/gke-cluster-standard"
  project_id = var.project_id
  name       = "cluster-${var.name}"
  location   = var.region
  vpc_config = {
    network    = google_compute_network.network.self_link
    subnetwork = google_compute_subnetwork.subnet.self_link
    secondary_range_names = {
      pods     = "pods"
      services = "services"
    }
    master_authorized_ranges = {
      rfc1918_10_8 = "10.0.0.0/8"
    }
    master_ipv4_cidr_block = "192.168.0.0/28"
  }
  enable_features = {
    dataplane_v2      = true
    workload_identity = true
    mesh_certificates = true
  }
  private_cluster_config = {
    enable_private_endpoint = true
    master_global_access    = true
  }
  labels = {
    environment = var.name
  }
}