Updating the node label node.kubernetes.io/exclude-from-external-load-balancers doesn't work as expected

[zarvd]

What happened:

When updating nodepool or node with label node.kubernetes.io/exclude-from-external-load-balancers, it doesn't update the LB Backend Pool and VMSS settings properly.

Copy the test scripts to run test scenarios.

chmod +x test.sh
./test.sh create # setup a cluster

Green scenarios

When adding a new node pool with the label, it works as expected that the VMSS and LB have no backend pool configurations. for test.sh run 01 and test.sh run 02

But updating the node pool with the label, it produces some unexpected results:

az aks nodepool update an existing nodepool with the label

Run test script

./test.sh run 03

Result:

Expand

// VMSS still have the backend pool configuration
{
  "name": "aks-nodepool3-18410684-vmss",
  "loadBalancerBackendAddressPools": [
    {
      "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/mc_jialuncai_beta_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes",
      "resourceGroup": "mc_jialuncai_beta_eastus2"
    },
    {
      "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/mc_jialuncai_beta_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool",
      "resourceGroup": "mc_jialuncai_beta_eastus2"
    }
  ]
}

// VMSS VM still have the backend pool configuration
[
  {
    "name": "aks-nodepool3-18410684-vmss_0",
    "loadBalancerBackendAddressPools": [
      {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/mc_jialuncai_beta_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes",
        "resourceGroup": "mc_jialuncai_beta_eastus2"
      },
      {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/mc_jialuncai_beta_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool",
        "resourceGroup": "mc_jialuncai_beta_eastus2"
      }
    ]
  },
  {
    "name": "aks-nodepool3-18410684-vmss_1",
    "loadBalancerBackendAddressPools": [
      {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/mc_jialuncai_beta_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes",
        "resourceGroup": "mc_jialuncai_beta_eastus2"
      },
      {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/mc_jialuncai_beta_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool",
        "resourceGroup": "mc_jialuncai_beta_eastus2"
      }
    ]
  },
  {
    "name": "aks-nodepool3-18410684-vmss_2",
    "loadBalancerBackendAddressPools": [
      {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/mc_jialuncai_beta_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes",
        "resourceGroup": "mc_jialuncai_beta_eastus2"
      },
      {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/mc_jialuncai_beta_eastus2/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool",
        "resourceGroup": "mc_jialuncai_beta_eastus2"
      }
    ]
  }
]

// LB still have the backend pool configuration
{
  "name": "kubernetes",
  "loadBalancerBackendAddresses": [
    {
      "name": "45cc328c-5aa5-4585-a336-cb8677025864",
      "networkInterfaceIPConfiguration": {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/MC_jialuncai_beta_eastus2/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool3-18410684-vmss/virtualMachines/2/networkInterfaces/aks-nodepool3-18410684-vmss/ipConfigurations/ipconfig1",
        "resourceGroup": "MC_jialuncai_beta_eastus2"
      }
    },
    {
      "name": "6a20fa1d-c828-4ebb-bbbb-bd3a6ce5f786",
      "networkInterfaceIPConfiguration": {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/MC_jialuncai_beta_eastus2/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool3-18410684-vmss/virtualMachines/1/networkInterfaces/aks-nodepool3-18410684-vmss/ipConfigurations/ipconfig1",
        "resourceGroup": "MC_jialuncai_beta_eastus2"
      }
    },
    {
      "name": "a8051c14-6d91-46c0-901e-5c97349ef236",
      "networkInterfaceIPConfiguration": {
        "id": "/subscriptions/feb5b150-60fe-4441-be73-8c02a524f55a/resourceGroups/MC_jialuncai_beta_eastus2/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool3-18410684-vmss/virtualMachines/0/networkInterfaces/aks-nodepool3-18410684-vmss/ipConfigurations/ipconfig1",
        "resourceGroup": "MC_jialuncai_beta_eastus2"
      }
    }
  ]
}

Expect: The VMSS / VMs / LB should update the backend pool configuration to exclude the nodes.

az aks nodepool update an existing nodepool with the label false

Run test script

./test.sh run 04

Result: It doesn't add backend pool configuration back to LB and VMSS
Expect: The VMSS and VMs should no longer be excluded, and the backend pool configuration should be properly set up (after LB service reconciliation).

kubectl label a node with the label

Run test script

./test.sh run 05

Result:

• It doesn't clean the VMSS backend pool if I label all the nodes
• Some of them are set with the backend pool configuration after a while

Expect:

• The VMSS should be cleaned up if all nodes are labeled?
• The backend pool configuration of LB / VMs should be consistent if no changes.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

For this issue, some clarification is needed regarding the following questions:

1. When all nodes are labeled, should the VMSS also clean up the backend pool association with the LB?
2. When some of the nodes are labeled, the VMSS is associated with the LB, but some of the VMs are not. Could other components or operations inadvertently sync the VMs back to the VMSS configuration?

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others: