Kubernetes weave-net pods raise CrashLoopBackOff error

[beacon-hash]

What you expected to happen?

Expected weave-net pods to be up and running on all worker nodes.

What happened?

The pods are giving CrashLoopBackOff error.

How to reproduce it?

I don't know how to reproduce the error.

Anything else we need to know?

I'm just trying to install Kubernetes manually service by service and everything worked fine as expected except weave-net. I'm using L4 load balancer in front of two instances of kube-apiservers and every service on worker nodes are pointing to the load balancer.

Versions:

$ weave version
I didn't find any binaries of this name "weave" in the downloaded package.

$ docker version
Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:47:44 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:46:10 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ uname -a
Linux worker-2 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Logs:

$ docker logs weave

E0411 02:39:40.230114       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:39:45.280217       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:40:13.077259       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:40:24.553533       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:40:41.115840       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:40:47.769260       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:41:10.816831       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:41:19.182499       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:41:46.219019       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:42:04.164261       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:42:11.320436       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:42:34.151626       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:42:36.377320       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:42:57.506367       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:43:14.482410       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:43:24.752246       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:43:28.755258       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:44:03.943241       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:44:07.155627       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:44:25.989563       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:44:36.150941       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:44:58.244105       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:45:01.142493       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:45:32.826854       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:45:50.024455       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:45:55.450961       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:46:20.517218       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:46:21.763890       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:46:32.183033       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:47:04.785586       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:47:07.304498       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:47:09.863042       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:47:42.048370       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:48:00.545652       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:48:01.731171       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:48:29.966778       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:48:38.149587       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:48:57.688809       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:49:15.184400       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:49:35.575247       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:49:53.341321       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:50:05.852036       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:50:14.734131       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:50:31.567045       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:50:58.393622       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:51:06.503496       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:51:11.449004       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:51:36.752626       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:52:04.055440       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:52:08.358374       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:52:31.143616       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:52:52.173220       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:53:04.851167       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:53:07.259431       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate is valid for 10.96.0.1, 10.0.0.50, 10.0.0.11, 10.0.0.12, 127.0.0.1, not 10.0.0.1
E0411 02:53:40.453200       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:338: Failed to list *v1.NetworkPolicy: Get "https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:53:41.279945       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:336: Failed to list *v1.Namespace: Get "https://10.0.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
E0411 02:54:04.556199       1 reflector.go:178] github.com/weaveworks/weave/prog/weave-npc/main.go:337: Failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority

journalctl -u kubelet.service --no-pager


Apr 11 02:54:27 worker-2 kubelet[26892]: I0411 02:54:27.559769   26892 scope.go:110] "RemoveContainer" containerID="d0fe45cbb25b47f6a1cfc9f14bbe2dfe9120311819549396e345c0a9c438a7d7"
Apr 11 02:54:28 worker-2 kubelet[26892]: I0411 02:54:28.565096   26892 generic.go:296] "Generic (PLEG): container finished" podID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c" exitCode=1
Apr 11 02:54:28 worker-2 kubelet[26892]: I0411 02:54:28.565357   26892 kubelet.go:2101] "SyncLoop (PLEG): event for pod" pod="kube-system/weave-net-9bms8" event=&{ID:eb1fdcb4-b6e6-45ef-805a-2b2410635edf Type:ContainerDied Data:31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c}
Apr 11 02:54:28 worker-2 kubelet[26892]: I0411 02:54:28.565393   26892 scope.go:110] "RemoveContainer" containerID="d0fe45cbb25b47f6a1cfc9f14bbe2dfe9120311819549396e345c0a9c438a7d7"
Apr 11 02:54:28 worker-2 kubelet[26892]: I0411 02:54:28.565685   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:54:28 worker-2 kubelet[26892]: E0411 02:54:28.565935   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:54:35 worker-2 kubelet[26892]: I0411 02:54:35.600035   26892 kubelet.go:2173] "SyncLoop (probe)" probe="readiness" status="" pod="kube-system/weave-net-9bms8"
Apr 11 02:54:35 worker-2 kubelet[26892]: I0411 02:54:35.600303   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:54:35 worker-2 kubelet[26892]: E0411 02:54:35.600502   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:54:50 worker-2 kubelet[26892]: I0411 02:54:50.564056   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:54:50 worker-2 kubelet[26892]: E0411 02:54:50.564357   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:55:03 worker-2 kubelet[26892]: I0411 02:55:03.559484   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:55:03 worker-2 kubelet[26892]: E0411 02:55:03.559725   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:55:15 worker-2 kubelet[26892]: I0411 02:55:15.559327   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:55:15 worker-2 kubelet[26892]: E0411 02:55:15.559580   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:55:30 worker-2 kubelet[26892]: I0411 02:55:30.560985   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:55:30 worker-2 kubelet[26892]: E0411 02:55:30.561226   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:55:42 worker-2 kubelet[26892]: I0411 02:55:42.561472   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:55:42 worker-2 kubelet[26892]: E0411 02:55:42.561752   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:55:57 worker-2 kubelet[26892]: I0411 02:55:57.560460   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:55:57 worker-2 kubelet[26892]: E0411 02:55:57.560880   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:56:09 worker-2 kubelet[26892]: I0411 02:56:09.560783   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:56:09 worker-2 kubelet[26892]: E0411 02:56:09.561244   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:56:20 worker-2 kubelet[26892]: I0411 02:56:20.561807   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:56:20 worker-2 kubelet[26892]: E0411 02:56:20.562239   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:56:33 worker-2 kubelet[26892]: I0411 02:56:33.560303   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:56:33 worker-2 kubelet[26892]: E0411 02:56:33.560913   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf
Apr 11 02:56:45 worker-2 kubelet[26892]: I0411 02:56:45.559579   26892 scope.go:110] "RemoveContainer" containerID="31061c99e1f8980fab91138a5e378e9289856e78cf936baab57f58f2ce94f31c"
Apr 11 02:56:45 worker-2 kubelet[26892]: E0411 02:56:45.559981   26892 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"weave\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=weave pod=weave-net-9bms8_kube-system(eb1fdcb4-b6e6-45ef-805a-2b2410635edf)\"" pod="kube-system/weave-net-9bms8" podUID=eb1fdcb4-b6e6-45ef-805a-2b2410635edf

[hswong3i]

For kubernetes 1.23.x I face some similar issue with weave v2.8.1.

Please give a try as #3939 (comment), with a patched version which working for newly provisioned cluster (work for both upgrade a new cluster from kubernetes 1.21.x -> 1.22.x -> 1.23.x, or a direct new install from 1.23.x).