Skip to content
This repository was archived by the owner on Jun 20, 2024. It is now read-only.

Weave-net CNI can not work on containerd=1.6.4 #3942

Open
fuzhibo opened this issue May 16, 2022 · 6 comments
Open

Weave-net CNI can not work on containerd=1.6.4 #3942

fuzhibo opened this issue May 16, 2022 · 6 comments

Comments

@fuzhibo
Copy link

fuzhibo commented May 16, 2022

What you expected to happen?

Using Weave-net CNI for kubernetes=v1.20.1

What happened?

Found weave-net CNI can not work on kubernetes=v1.20.1 and containerd.io=1.6.4, the veth of containers can not be created.

How to reproduce it?

Upgrade containerd.io to 1.6.4

Anything else we need to know?

When I downgrades containerd.io=1.5.11,everything works.

Versions:

$ weave version
weave 2.8.1
$ docker version
```console
Client: Docker Engine - Community
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Thu May 12 09:17:28 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.16
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.10
  Git commit:       f756502
  Built:            Thu May 12 09:15:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ uname -a

Linux k8s-master 4.15.0-177-generic #186-Ubuntu SMP Thu Apr 14 20:23:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:00:47Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
## Logs:

$ docker logs weave

or, if using Kubernetes:

$ kubectl logs -n kube-system weave

<!-- (If output is long, please consider a Gist.) -->
<!-- Anything interesting or unusual output by the below, potentially relevant, commands?
$ journalctl -u docker.service --no-pager
$ journalctl -u kubelet --no-pager
May 12 14:59:51 k8s-master containerd[2087]: time="2022-05-12T14:59:51.590299653+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:fluentd-k26wt,Uid:cbe02001-06b7-4b8c-872e-7d559b61395c,Namespace:kube-system,Attempt:0,}"

May 12 14:59:51 k8s-master kernel: [11607.056092] weave: port 2(vethwepl73cc41b) entered blocking state

May 12 14:59:51 k8s-master kernel: [11607.056096] weave: port 2(vethwepl73cc41b) entered disabled state

May 12 14:59:51 k8s-master kernel: [11607.056207] device vethwepl73cc41b entered promiscuous mode

May 12 14:59:51 k8s-master systemd-udevd[22597]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.

May 12 14:59:51 k8s-master systemd-udevd[22598]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.

May 12 14:59:51 k8s-master networkd-dispatcher[1866]: WARNING:Unknown index 5167 seen, reloading interface list

May 12 14:59:51 k8s-master systemd-udevd[22598]: Could not generate persistent MAC address for vethwepl73cc41b: No such file or directory

May 12 14:59:51 k8s-master kernel: [11607.085723] eth0: renamed from vethwepg73cc41b

May 12 14:59:51 k8s-master systemd-udevd[22597]: link_config: could not get ethtool features for vethwepg73cc41b

May 12 14:59:51 k8s-master systemd-udevd[22597]: Could not set offload features of vethwepg73cc41b: No such device

May 12 14:59:51 k8s-master networkd-dispatcher[1866]: ERROR:Unknown interface index 5167 seen even after reload

May 12 14:59:51 k8s-master libvirtd[2076]: 2022-05-12 06:59:51.985+0000: 2831: error : virFileReadAll:1420 : Failed to open file '/sys/class/net/vethwepg73cc41b/operstate': No such file or directory

May 12 14:59:51 k8s-master libvirtd[2076]: 2022-05-12 06:59:51.985+0000: 2831: error : virNetDevGetLinkInfo:2530 : unable to read: /sys/class/net/vethwepg73cc41b/operstate: No such file or directory

May 12 14:59:52 k8s-master systemd-networkd[1618]: vethwepl73cc41b: Link UP

May 12 14:59:52 k8s-master kernel: [11607.219944] IPv6: ADDRCONF(NETDEV_UP): vethwepl73cc41b: link is not ready

May 12 14:59:52 k8s-master systemd-networkd[1618]: vethwepl73cc41b: Gained carrier

May 12 14:59:52 k8s-master kernel: [11607.226183] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

May 12 14:59:52 k8s-master kernel: [11607.226199] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

May 12 14:59:52 k8s-master kernel: [11607.226237] IPv6: ADDRCONF(NETDEV_CHANGE): vethwepl73cc41b: link becomes ready

May 12 14:59:52 k8s-master kernel: [11607.226293] weave: port 2(vethwepl73cc41b) entered blocking state

May 12 14:59:52 k8s-master kernel: [11607.226295] weave: port 2(vethwepl73cc41b) entered forwarding state

May 12 14:59:52 k8s-master systemd-networkd[1618]: vethwepl73cc41b: Link DOWN

May 12 14:59:52 k8s-master systemd-networkd[1618]: vethwepl73cc41b: Lost carrier

May 12 14:59:52 k8s-master kernel: [11607.299911] weave: port 2(vethwepl73cc41b) entered disabled state

May 12 14:59:52 k8s-master kernel: [11607.303493] device vethwepl73cc41b left promiscuous mode

May 12 14:59:52 k8s-master kernel: [11607.303495] weave: port 2(vethwepl73cc41b) entered disabled state

May 12 14:59:52 k8s-master containerd[2087]: time="2022-05-12T14:59:52.223538455+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:fluentd-k26wt,Uid:cbe02001-06b7-4b8c-872e-7d559b61395c,Namespace:kube-system,Attempt:0,} failed, error" error="failed to setup network for sandbox \"73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993\": failed to find network info for sandbox \"73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993\""

May 12 14:59:52 k8s-master kubelet[2063]: E0512 14:59:52.224570    2063 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to setup network for sandbox "73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993": failed to find network info for sandbox "73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993"

May 12 14:59:52 k8s-master kubelet[2063]: E0512 14:59:52.224657    2063 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "fluentd-k26wt_kube-system(cbe02001-06b7-4b8c-872e-7d559b61395c)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993": failed to find network info for sandbox "73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993"

May 12 14:59:52 k8s-master kubelet[2063]: E0512 14:59:52.224683    2063 kuberuntime_manager.go:755] createPodSandbox for pod "fluentd-k26wt_kube-system(cbe02001-06b7-4b8c-872e-7d559b61395c)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993": failed to find network info for sandbox "73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993"

May 12 14:59:52 k8s-master kubelet[2063]: E0512 14:59:52.224767    2063 pod_workers.go:191] Error syncing pod cbe02001-06b7-4b8c-872e-7d559b61395c ("fluentd-k26wt_kube-system(cbe02001-06b7-4b8c-872e-7d559b61395c)"), skipping: failed to "CreatePodSandbox" for "fluentd-k26wt_kube-system(cbe02001-06b7-4b8c-872e-7d559b61395c)" with CreatePodSandboxError: "CreatePodSandbox for pod \"fluentd-k26wt_kube-system(cbe02001-06b7-4b8c-872e-7d559b61395c)\" failed: rpc error: code = Unknown desc = failed to setup network for sandbox \"73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993\": failed to find network info for sandbox \"73cc41b331bf40599cbd3fa813203c8e6d6478c753f298606421b1ae7a15e993\""

May 12 15:00:01 k8s-master kernel: [11616.961093] sh (22763): drop_caches: 3

May 12 15:00:01 k8s-master kernel: [11616.962742] sh (22760): drop_caches: 3

May 12 15:00:01 k8s-master kernel: [11616.964162] sh (22759): drop_caches: 3
$ kubectl get events
-->

## Network:
<!-- If your problem has anything to do with one network endpoint not being able to contact another, please run the following commands -->

$ ip route
$ ip -4 -o addr
$ sudo iptables-save

@busyboy77
Copy link

Up for this Issue.

@jpetazzo
Copy link

For more details, see:

I'm not sure if it's weave's fault or containerd's fault; if it's weave's I guess they'll want to fix it; if it's containerd's they'll want to advocate and explain why 😅

@boskoop
Copy link

boskoop commented May 30, 2022

Can confirm as well on kubernetes v1.24.1/weave 2.8.1 where downgrading to containerd.io=1.5.11-1 solved the issue too.

@busyboy77
Copy link

and I am sure until they find who's fault was it, people will forget weave altogether on Kubernetes, my guess after seeing that weave was last updated about an year ago

@kingdonb
Copy link
Contributor

kingdonb commented Jun 6, 2022

If there is someone who has the chops and wants to see Weave net maintained, send your PRs @ me and I will help you to try and get them merged.

My understanding from containerd/containerd#6921 (comment) is that this all works again, thanks to a change from upstream which has resolved the backwards-incompatible changes in CNI.

Which means of course, people can install weave net again (and they might be in danger with no maintainers actively pushing out releases.)

(Edit: the discussion in #3939 is a good place to start if you haven't seen it yet.)

@rajch
Copy link
Contributor

rajch commented Jun 7, 2022

Meanwhile, I can confirm that weave net works as-is on containerd 1.6.6, with Kubernetes 1.24, 1.23 and 1.22.

skitt added a commit to skitt/shipyard that referenced this issue Jun 29, 2022
kind 0.13 and 0.14 add support for Kubernetes 1.24, but the non-1.24
Kubernetes images used with these versions currently embed containerd
1.6.4, which breaks Weave (see
weaveworks/weave#3942 for details).

To allow testing Kubernetes 1.24, install kind 0.14, while preserving
kind 0.12; both binaries are now named explicitly with their version
number (kind-0.12 and kind-0.14). When Kubernetes 1.24 is requested,
deploy it with kind 0.14, otherwise rely on 0.12.

See https://github.com/kubernetes-sigs/kind/releases/tag/v0.13.0 and
https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0 for
details.

Signed-off-by: Stephen Kitt <[email protected]>
skitt added a commit to skitt/shipyard that referenced this issue Jun 30, 2022
kind 0.13 and 0.14 add support for Kubernetes 1.24, but the non-1.24
Kubernetes images used with these versions currently embed containerd
1.6.4, which breaks Weave (see
weaveworks/weave#3942 for details).

To allow testing Kubernetes 1.24, install kind 0.14, while preserving
kind 0.12; both binaries are now named explicitly with their version
number (kind-0.12 and kind-0.14). When Kubernetes 1.24 is requested,
deploy it with kind 0.14, otherwise rely on 0.12.

See https://github.com/kubernetes-sigs/kind/releases/tag/v0.13.0 and
https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0 for
details.

Signed-off-by: Stephen Kitt <[email protected]>
skitt added a commit to skitt/shipyard that referenced this issue Jun 30, 2022
kind 0.13 and 0.14 add support for Kubernetes 1.24, but the non-1.24
Kubernetes images used with these versions currently embed containerd
1.6.4, which breaks Weave (see
weaveworks/weave#3942 for details).

To allow testing Kubernetes 1.24, install kind 0.14, while preserving
kind 0.12; both binaries are now named explicitly with their version
number (kind-0.12 and kind-0.14). When Kubernetes 1.24 is requested,
deploy it with kind 0.14, otherwise rely on 0.12.

See https://github.com/kubernetes-sigs/kind/releases/tag/v0.13.0 and
https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0 for
details.

Signed-off-by: Stephen Kitt <[email protected]>
skitt added a commit to skitt/shipyard that referenced this issue Jun 30, 2022
kind 0.13 and 0.14 add support for Kubernetes 1.24, but the non-1.24
Kubernetes images used with these versions currently embed containerd
1.6.4, which breaks Weave (see
weaveworks/weave#3942 for details).

To allow testing Kubernetes 1.24, install kind 0.14, while preserving
kind 0.12; both binaries are now named explicitly with their version
number (kind-0.12 and kind-0.14). When Kubernetes 1.24 is requested,
deploy it with kind 0.14, otherwise rely on 0.12.

See https://github.com/kubernetes-sigs/kind/releases/tag/v0.13.0 and
https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0 for
details.

Signed-off-by: Stephen Kitt <[email protected]>
skitt added a commit to skitt/shipyard that referenced this issue Jul 6, 2022
kind 0.13 and 0.14 add support for Kubernetes 1.24, but the non-1.24
Kubernetes images used with these versions currently embed containerd
1.6.4, which breaks Weave (see
weaveworks/weave#3942 for details).

To allow testing Kubernetes 1.24, install kind 0.14, while preserving
kind 0.12; kind 0.12 is installed as "kind", kind 0.14 as "kind-0.14"
(we need a "kind" binary because other parts of our CI expect it).
When Kubernetes 1.24 is requested, deploy it with kind 0.14, otherwise
rely on 0.12.

See https://github.com/kubernetes-sigs/kind/releases/tag/v0.13.0 and
https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0 for
details.

This will break OVN on K8s 1.24 since the OVN deployment script drives
kind directly.

Signed-off-by: Stephen Kitt <[email protected]>
skitt added a commit to skitt/shipyard that referenced this issue Jul 6, 2022
kind 0.13 and 0.14 add support for Kubernetes 1.24, but the non-1.24
Kubernetes images used with these versions currently embed containerd
1.6.4, which breaks Weave (see
weaveworks/weave#3942 for details).

To allow testing Kubernetes 1.24, install kind 0.14, while preserving
kind 0.12; kind 0.12 is installed as "kind", kind 0.14 as "kind-0.14"
(we need a "kind" binary because other parts of our CI expect it).
When Kubernetes 1.24 is requested, deploy it with kind 0.14, otherwise
rely on 0.12.

See https://github.com/kubernetes-sigs/kind/releases/tag/v0.13.0 and
https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0 for
details.

This will break OVN on K8s 1.24 since the OVN deployment script drives
kind directly.

Signed-off-by: Stephen Kitt <[email protected]>
skitt added a commit to submariner-io/shipyard that referenced this issue Jul 6, 2022
kind 0.13 and 0.14 add support for Kubernetes 1.24, but the non-1.24
Kubernetes images used with these versions currently embed containerd
1.6.4, which breaks Weave (see
weaveworks/weave#3942 for details).

To allow testing Kubernetes 1.24, install kind 0.14, while preserving
kind 0.12; kind 0.12 is installed as "kind", kind 0.14 as "kind-0.14"
(we need a "kind" binary because other parts of our CI expect it).
When Kubernetes 1.24 is requested, deploy it with kind 0.14, otherwise
rely on 0.12.

See https://github.com/kubernetes-sigs/kind/releases/tag/v0.13.0 and
https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0 for
details.

This will break OVN on K8s 1.24 since the OVN deployment script drives
kind directly.

Signed-off-by: Stephen Kitt <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants