HCCM networks + istio failing external load balancer healthchecks

[DoubleDi]

TL;DR

I am installing a k8s cluster using kubeadm, cry-o, flannel, istio, and hccm-networks. HCCM correctly setups the nodes, provisions the network routes, finds the load balancer, and adds the load balancer services & targets but fails to confirm 2 of 3 healthchecks for istio ingressgateway.

Expected behavior

All 3 healthchecks are green and load balancer ready to use.

Observed behavior

Can't query my cluster through DNS and by loadbalancer IP.

Minimal working example

No response

Log output

I1228 06:15:22.044830 1 controllermanager.go:329] Started "cloud-node-controller"
I1228 06:15:22.044855 1 route_controller.go:102] Starting route controller
I1228 06:15:22.044928 1 shared_informer.go:313] Waiting for caches to sync for route
I1228 06:15:22.044999 1 node_controller.go:176] Sending events to api server.
I1228 06:15:22.045071 1 node_controller.go:185] Waiting for informer caches to sync
I1228 06:15:22.076713 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController
I1228 06:15:22.076973 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1228 06:15:22.077088 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1228 06:15:22.089587 1 shared_informer.go:320] Caches are synced for service
I1228 06:15:22.090063 1 load_balancers.go:127] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="istio-ingressgateway" nodes=["wi-kube-worker-2","wi-kube-worker-1"]
I1228 06:15:22.090321 1 event.go:389] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I1228 06:15:22.145581 1 shared_informer.go:320] Caches are synced for route
I1228 06:15:22.589011 1 route_controller.go:214] action for Node "wi-kube-worker-2" with CIDR "10.244.1.0/24": "keep"
I1228 06:15:22.589085 1 route_controller.go:214] action for Node "wi-kube-master" with CIDR "10.244.0.0/24": "keep"
I1228 06:15:22.589106 1 route_controller.go:214] action for Node "wi-kube-worker-1" with CIDR "10.244.2.0/24": "keep"
I1228 06:15:23.473061 1 load_balancers.go:171] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=2189683
I1228 06:15:23.752529 1 load_balancer.go:871] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=15021 loadBalancerID=2189683
I1228 06:15:24.068170 1 load_balancer.go:871] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=2189683
I1228 06:15:24.284317 1 load_balancer.go:871] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=2189683
I1228 06:15:24.513272 1 load_balancers.go:192] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=2189683
I1228 06:15:24.781549 1 event.go:389] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"

Additional information

1. installing hccm-networks by this guide
   https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/docs/deploy_with_networks.md
2. installing flannel by this guide from readme
   https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/README.md
3. installing istio by this guide
   https://istio.io/latest/docs/setup/install/istioctl/

istioctl install --set "components.egressGateways[0].name=istio-egressgateway" \
                   --set "components.egressGateways[0].enabled=true"

4. annotations on istio-ingressgateway LoadBalancer

    load-balancer.hetzner.cloud/disable-private-ingress: 'true'
    load-balancer.hetzner.cloud/hostname: k8s.dd-dev.com
    load-balancer.hetzner.cloud/http-redirect-https: 'false'
    load-balancer.hetzner.cloud/location: nbg1
    load-balancer.hetzner.cloud/name: kubeautolb
    load-balancer.hetzner.cloud/use-private-ip: 'true'
    load-balancer.hetzner.cloud/uses-proxyprotocol: 'true'

6. domain k8s.dd-dev.com correctly points to the loadbalancer ip in hetzner dns
7. The check for port 15021 is green, and 15021 corresponds to istio healthchecks port, which signals istio is working properly

[hajowieland]

Encountered the same problem, health checks were successful only for 1/3 of services:

• ✅15021
• ❌ 80
• ❌ 443

The root cause was that there was no Istio Gateway CustomResource. With a correct Gateway resource in place, all three healthchecks got healthy instantly.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
  namespace: istio-ingress
spec:
  selector:
    istio: gateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: your-tls-secret
    hosts:
    - "*"

You may adapt the above with the correct selector for your Istio Ingressgateways. Check for example with this simple command (if your Gateway pods are running in istio-ingress Namespace as recommended by Istio docs):

kubectl get pods -n istio-ingress -l istio=gateway

[DoubleDi]

Thank you @hajowieland for explaining! It worked after the configuration for istio was setup. For me I actually thought the installation was wrong, and did not install any gateways.

A small comment in the docs could be helpful.