Unable to connect to CoreDNS instances when running local tutorial #1830
Description
When running the tutorial at https://www.k8gb.io/docs/local.html, it works for me up until the point where you run dig for each instance. I get an error saying it can't connect:
dig -p 5053 +tcp @localhost localtargets-roundrobin.cloud.example.com && \
dig -p 5054 +tcp @localhost localtargets-roundrobin.cloud.example.com
;; communications error to 127.0.0.1#5053: connection reset
;; communications error to 127.0.0.1#5053: connection reset
;; communications error to 127.0.0.1#5053: connection reset
; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> -p 5053 +tcp @localhost localtargets-roundrobin.cloud.example.com
; (1 server found)
;; global options: +cmd
;; no servers could be reached
I am however able to run the dig command slightly above that in the tutorial where I connect to the EdgeDNS cluster:
$ dig @localhost -p 1053 roundrobin.cloud.example.com +short +tcp
172.19.0.8
172.19.0.7
172.19.0.4
172.19.0.5
I figured to help with troubleshooting, I should include info about the services that are setup in each cluster at this point in time:
$ k config use k3d-edgedns
Switched to context "k3d-edgedns".
$ k get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default bind NodePort 10.43.63.77 <none> 53:30053/TCP,53:30053/UDP 3h13m
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 3h14m
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 3h14m
$ k config use k3d-test-gslb1
Switched to context "k3d-test-gslb1".
$ k get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 3h15m
istio-ingress istio-ingressgateway LoadBalancer 10.43.171.174 172.19.0.4,172.19.0.5 8080:31999/TCP 3h14m
istio-system istiod ClusterIP 10.43.154.83 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 3h14m
k8gb k8gb-coredns ClusterIP 10.43.64.73 <none> 53/UDP 3h15m
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 3h15m
test-gslb-istio frontend-podinfo ClusterIP 10.43.156.0 <none> 9898/TCP,9999/TCP 3h14m
test-gslb-istio unhealthy-app ClusterIP 10.43.93.229 <none> 80/TCP 3h14m
test-gslb frontend-podinfo ClusterIP 10.43.218.93 <none> 9898/TCP,9999/TCP 3h14m
test-gslb unhealthy-app ClusterIP 10.43.130.87 <none> 80/TCP 3h14m
$ k config use k3d-test-gslb2
Switched to context "k3d-test-gslb2".
$ k get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 3h16m
istio-ingress istio-ingressgateway LoadBalancer 10.43.131.244 172.19.0.7,172.19.0.8 8080:32125/TCP 3h14m
istio-system istiod ClusterIP 10.43.183.147 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 3h14m
k8gb k8gb-coredns ClusterIP 10.43.232.163 <none> 53/UDP 3h14m
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 3h15m
test-gslb-istio frontend-podinfo ClusterIP 10.43.111.176 <none> 9898/TCP,9999/TCP 3h14m
test-gslb-istio unhealthy-app ClusterIP 10.43.242.38 <none> 80/TCP 3h14m
test-gslb frontend-podinfo ClusterIP 10.43.173.240 <none> 9898/TCP,9999/TCP 3h14m
test-gslb unhealthy-app ClusterIP 10.43.165.178 <none> 80/TCP 3h14m
My understanding so far is that k3d is being used to run a few k3s clusters in Docker containers. I usually use kind for this. This is my first time using k3d. I understand with kind, if you're using Ingress, there is some setup involved to forward ports to your local machine. I'm wondering if that's where this might be breaking down. Perhaps this is required with the way we're using k3d in this tutorial and there's something wrong with that configuration. That's just my guess so far though.
More info about my setup:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
$ kubectl version
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.30.2+k3s1
WARNING: version difference between client (1.32) and server (1.30) exceeds the supported minor version skew of +/-1
$ helm version
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}
$ docker version
Client: Docker Engine - Community
Version: 27.5.1
API version: 1.47
Go version: go1.22.11
Git commit: 9f9e405
Built: Wed Jan 22 13:41:48 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.5.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.11
Git commit: 4c9b3b0
Built: Wed Jan 22 13:41:48 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.25
GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc:
Version: 1.2.4
GitCommit: v1.2.4-0-g6c52b3f
docker-init:
Version: 0.19.0
GitCommit: de40ad0
$ k3d version
k3d version v5.8.1
k3s version v1.31.4-k3s1 (default)
$ golangci-lint version
golangci-lint has version 1.62.0 built with go1.23.2 from 22b58c9b on 2024-11-10T19:09:02Z