Autoscaler workers not working when no agents?

[stubbi]

Description

Hi there,

It seems I cannot create autoscaler nodepools without creating at least one non-autoscaler node? This was working in an older version.

Kube.tf file

module "kube-hetzner" {
  source = "kube-hetzner/kube-hetzner/hcloud"
  version = "2.1.0"

  initial_k3s_channel = "v1.25"

  hcloud_token = var.hcloud_token

  cluster_name = "${var.stage}-${var.region}"
  base_domain  = var.domain

  network_region     = var.region
  load_balancer_type = var.lb_type

  lb_hostname            = local.lb_hostname
  load_balancer_location = local.locations[var.region][0]

  control_plane_nodepools = [
    for cp in range(3) :
    {
      name        = "control-plane-${var.stage}-${var.region}-${cp}",
      server_type = var.cp_type,
      location    = local.locations[var.region][cp % length(local.locations[var.region])],
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  autoscaler_nodepools = [
    for cp in range(3) :
    {
      name        = "autoscaler-${cp}"
      server_type = var.agent_type
      location    = local.locations[var.region][cp % length(local.locations[var.region])],
      min_nodes   = 0
      max_nodes   = 5
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = var.agent_type
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 0 // working when setting to > 0
    }
  ]

  ingress_controller   = "nginx"
  use_control_plane_lb = true
  cni_plugin           = "cilium"
  enable_cert_manager  = false

  create_kubeconfig    = false
  create_kustomization = false

  ssh_public_key  = file("${var.ssh_key}.pub")
  ssh_private_key = file(var.ssh_key)

  providers = {
    hcloud = hcloud
  }

}

variable "lb_type" {
  default = "lb11"
}

variable "cp_type" {
  default = "cpx11"
}

variable "agent_type" {
  default = "cpx31"
}

locals {
  lb_hostname = "lb-${var.stage}-${var.region}.${var.domain}"

  # https://docs.hetzner.com/cloud/general/locations/
  locations = {
    "eu-central" = ["fsn1", "nbg1", "hel1"],
    "us-east"    = ["ash"],
    "us-west"    = ["hil"],
  }
}

Screenshots

No response

Platform

Mac

[mysticaltech]

@stubbi You have to check the autoscaler pod logs, see what is happening.

[mysticaltech]

@stubbi Was your previews try with cilium too? Try without cilium, and as started about, the key is probably in the logs of the autoscaler pod.

[stubbi]

@mysticaltech sorry for my late reply. I do enable the cilium cni. To me it appears that the autoscaler deployment is only there when I configure at least one non-autoscaler node:

➜  infra git:(main) ✗ k -n kube-system get deployments
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
cilium-operator                   2/2     2            2           26m
cluster-autoscaler                1/1     1            1           7m35s
coredns                           1/1     1            1           26m
hcloud-cloud-controller-manager   1/1     1            1           26m
hcloud-csi-controller             1/1     1            1           26m
metrics-server                    1/1     1            1           26m

I had deployed the cluster 26min ago with only autoscaler node group and min=0, max=5. When I was looking for the autoscaler pod, there was none, nor had the deployment been there. Only after adding a non-autoscaler node it's there now and new nodes are being added on the fly.

Realistically, I can set the min workers to 1. I will test if that makes a difference. Though I believe it should work with min workers=0 and no additional fixed nodes and it used to do so.