Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] 集群节点重启，nat-gw pod eip丢失 （非必现） #4987

Open
kldancer opened this issue Feb 12, 2025 · 6 comments
Open

[BUG] 集群节点重启，nat-gw pod eip丢失 （非必现） #4987

kldancer opened this issue Feb 12, 2025 · 6 comments
Labels
bug Something isn't working eip

Comments

@kldancer
Copy link
Contributor

Kube-OVN Version

v1.12.22

Kubernetes Version

v1.31.1

Operation-system/Kernel Version

5.10.0-136.12.0.86.4.hl202.x86_64

Description

存在该问题：
重启集群节点后，vpc-nat-gw pod存在概率未正常重建（仅重启容器），gw pod的 creationTimestamp未变，导致redo eip的逻辑未成功触发。从而导致该nat-gw pod绑定的eip丢失。
请问该如何规避或解决

pod 事件、kubelet日志、multus-cni、kube-ovn-cni日志如下：

[root@node-106 ~]#  kubectl get pod -A -owide | grep nat-gw
kube-system                    vpc-nat-gw-eip-42pkfhov-0                                 1/1     Running     0              13h     192.168.251.254   node-107   <none>           <none>
kube-system                    vpc-nat-gw-eip-5kt5yrkn-0                                 1/1     Running     0              13h     192.168.250.254   node-107   <none>           <none>
kube-system                    vpc-nat-gw-eip-84w1f247-0                                 1/1     Running     0              13h     192.168.230.254   node-107   <none>           <none>
kube-system                    vpc-nat-gw-eip-fzps6w0z-0                                 1/1     Running     0              13h     19.19.19.254      node-107   <none>           <none>
kube-system                    vpc-nat-gw-eip-iiq12yc9-0                                 1/1     Running     0              13h     192.168.240.254   node-107   <none>           <none>
kube-system                    vpc-nat-gw-eip-uyuzv87b-0                                 1/1     Running     0              13h     192.168.200.254   node-107   <none>           <none>
kube-system                    vpc-nat-gw-eip-x1kho6vq-0                                 1/1     Running     0              13h     192.168.220.254   node-107   <none>           <none>
kube-system                    vpc-nat-gw-eip-z10eqvi9-0                                 1/1     Running     1 (13h ago)    13h     192.168.210.254   node-107   <none>           <none>
kube-system                    vpc-nat-gw-eip-zu0xjsv0-0                                 1/1     Running     0              13h     192.168.100.254   node-107   <none>           <none>

[root@node-107 ~]# kubectl describe pod -n kube-system                    vpc-nat-gw-eip-z10eqvi9-0  
Name:             vpc-nat-gw-eip-z10eqvi9-0
Namespace:        kube-system
Priority:         0
Service Account:  default
Node:             node-107/192.166.0.107
        Tue, 11 Feb 2025 19:48:15 +0800
Labels:           app=vpc-nat-gw-eip-z10eqvi9
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=vpc-nat-gw-eip-z10eqvi9-57bb998f9c
                  ovn.kubernetes.io/vpc-nat-gw=true
                  statefulset.kubernetes.io/pod-name=vpc-nat-gw-eip-z10eqvi9-0
Annotations:      k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "kube-ovn",
                        "interface": "eth0",
                        "ips": [
                            "192.168.210.254"
                        ],
                        "mac": "62:8b:55:12:26:c9",
                        "default": true,
                        "dns": {},
                        "gateway": [
                            "192.168.210.1"
                        ]
                    },{
                        "name": "kube-system/net-96m3optj",
                        "interface": "net1",
                        "ips": [
                            "172.12.0.16"
                        ],
                        "mac": "12:e8:aa:42:c9:43",
                        "dns": {}
                    }]
                  k8s.v1.cni.cncf.io/networks: kube-system/net-96m3optj
                  net-96m3optj.kube-system.kubernetes.io/allocated: true
                  net-96m3optj.kube-system.kubernetes.io/cidr: 172.12.0.0/24
                  net-96m3optj.kube-system.kubernetes.io/gateway: 172.12.0.1
                  net-96m3optj.kube-system.kubernetes.io/ip_address: 172.12.0.16
                  ovn.kubernetes.io/allocated: true
                  ovn.kubernetes.io/cidr: 192.168.210.0/24
                  ovn.kubernetes.io/gateway: 192.168.210.1
                  ovn.kubernetes.io/ip_address: 192.168.210.254
                  ovn.kubernetes.io/logical_router: vpc-g69pfq9u
                  ovn.kubernetes.io/logical_switch: subnet-1udcchv9
                  ovn.kubernetes.io/mac_address: 62:8b:55:12:26:c9
                  ovn.kubernetes.io/pod_nic_type: veth-pair
                  ovn.kubernetes.io/routed: true
                  ovn.kubernetes.io/vpc_cidrs: ["192.168.210.0/24"]
                  ovn.kubernetes.io/vpc_nat_gw: eip-z10eqvi9
                  ovn.kubernetes.io/vpc_nat_gw_init: true
Status:           Running
IP:               192.168.210.254
IPs:
  IP:           192.168.210.254
Controlled By:  StatefulSet/vpc-nat-gw-eip-z10eqvi9
Init Containers:
  vpc-nat-gw-init:
    Container ID:  containerd://5f48920be70a03fa88f5f13c91d454c3087f882a4806bda4a315bbf8666d4540
    Image:         docker.io/kubeovn/vpc-nat-gateway:v1.12.22
    Image ID:      sha256:4e296ee5aa5905c39841e873290cc7897c3ccdb059aad2dbeaf244c0f0548aff
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
    Args:
      -c
      bash /kube-ovn/nat-gateway.sh init 10.96.0.0/12,192.168.210.1
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 11 Feb 2025 20:03:23 +0800
      Finished:     Tue, 11 Feb 2025 20:03:23 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rxzgm (ro)
Containers:
  vpc-nat-gw:
    Container ID:  containerd://b5d886e2feb3955f3b363a4ea6a711c9e5c981a822c6de6913d48457d8ac27ea
    Image:         docker.io/kubeovn/vpc-nat-gateway:v1.12.22
    Image ID:      sha256:4e296ee5aa5905c39841e873290cc7897c3ccdb059aad2dbeaf244c0f0548aff
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
    Args:
      -c
      while true; do sleep 10000; done
    State:          Running
      Started:      Tue, 11 Feb 2025 20:03:23 +0800
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Tue, 11 Feb 2025 19:49:47 +0800
      Finished:     Tue, 11 Feb 2025 20:00:03 +0800
    Ready:          True
    Restart Count:  1
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rxzgm (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-rxzgm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              nodeType=controller
Tolerations:                 node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 5s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 5s
Events:
  Type     Reason                  Age                    From                  Message
  ----     ------                  ----                   ----                  -------
  Normal   Scheduled               19m                    default-scheduler     Successfully assigned kube-system/vpc-nat-gw-eip-z10eqvi9-0 to node-107
  Normal   AddedInterface          19m                    multus                Add eth0 [192.168.210.254/24] from kube-ovn
  Normal   AddedInterface          19m                    multus                Add net1 [172.12.0.16/24] from kube-system/net-96m3optj
  Normal   Pulled                  18m                    kubelet               Container image "docker.io/kubeovn/vpc-nat-gateway:v1.12.22" already present on machine
  Normal   Created                 18m                    kubelet               Created container vpc-nat-gw-init
  Normal   Started                 18m                    kubelet               Started container vpc-nat-gw-init
  Normal   Pulled                  18m                    kubelet               Container image "docker.io/kubeovn/vpc-nat-gateway:v1.12.22" already present on machine
  Normal   Created                 18m                    kubelet               Created container vpc-nat-gw
  Normal   Started                 18m                    kubelet               Started container vpc-nat-gw
  Warning  NodeNotReady            7m28s                  node-controller       Node is not ready
  Warning  FailedCreatePodSandBox  7m20s                  kubelet               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "992a0881ecf83dc1dcd71de5ed135d63c8bb06ae41d62a1b138c1ffee69f1c9b": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/multus.sock: connect: no such file or directory
  Warning  FailedCreatePodSandBox  7m6s                   kubelet               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4102a18946ec67561023a0fa1a5dfe6bea3adddc52568962de885ebe30dc084c": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/multus.sock: connect: no such file or directory
  Normal   SandboxChanged          6m51s (x3 over 7m20s)  kubelet               Pod sandbox changed, it will be killed and re-created.
  Normal   AddedInterface          5m21s                  multus                Add eth0 [192.168.210.254/24] from kube-ovn
  Normal   AddedInterface          4m28s                  multus                Add net1 [172.12.0.16/24] from kube-system/net-96m3optj
  Normal   Pulled                  4m25s                  kubelet               Container image "docker.io/kubeovn/vpc-nat-gateway:v1.12.22" already present on machine
  Normal   Created                 4m25s                  kubelet               Created container vpc-nat-gw-init
  Normal   Started                 4m25s                  kubelet               Started container vpc-nat-gw-init
  Normal   Pulled                  4m25s                  kubelet               Container image "docker.io/kubeovn/vpc-nat-gateway:v1.12.22" already present on machine
  Normal   Created                 4m25s                  kubelet               Created container vpc-nat-gw
  Normal   Started                 4m25s                  kubelet               Started container vpc-nat-gw
  Warning  FailedAddingInterface   2m21s (x4 over 4m27s)  pod-networks-updates  pod [kube-system/vpc-nat-gw-eip-z10eqvi9-0]: failed adding interface  to network: net-96m3optj

[root@node-107 ~]# journalctl -u kubelet.service | grep  z10eqvi9 
Feb 11 20:00:28 node-107 kubelet[5322]: E0211 20:00:28.496275    5322 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"992a0881ecf83dc1dcd71de5ed135d63c8bb06ae41d62a1b138c1ffee69f1c9b\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": dial unix /run/multus/multus.sock: connect: no such file or directory" pod="kube-system/vpc-nat-gw-eip-z10eqvi9-0"
Feb 11 20:00:28 node-107 kubelet[5322]: E0211 20:00:28.496296    5322 kuberuntime_manager.go:1170] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"992a0881ecf83dc1dcd71de5ed135d63c8bb06ae41d62a1b138c1ffee69f1c9b\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": dial unix /run/multus/multus.sock: connect: no such file or directory" pod="kube-system/vpc-nat-gw-eip-z10eqvi9-0"
Feb 11 20:00:28 node-107 kubelet[5322]: E0211 20:00:28.496332    5322 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"vpc-nat-gw-eip-z10eqvi9-0_kube-system(055d716e-2557-4639-9bfc-232f39b1eb67)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"vpc-nat-gw-eip-z10eqvi9-0_kube-system(055d716e-2557-4639-9bfc-232f39b1eb67)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"992a0881ecf83dc1dcd71de5ed135d63c8bb06ae41d62a1b138c1ffee69f1c9b\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): failed to send CNI request: Post \\\"http://dummy/cni\\\": dial unix /run/multus/multus.sock: connect: no such file or directory\"" pod="kube-system/vpc-nat-gw-eip-z10eqvi9-0" podUID="055d716e-2557-4639-9bfc-232f39b1eb67"
Feb 11 20:00:42 node-107 kubelet[5322]: E0211 20:00:42.797207    5322 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"4102a18946ec67561023a0fa1a5dfe6bea3adddc52568962de885ebe30dc084c\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": dial unix /run/multus/multus.sock: connect: no such file or directory" pod="kube-system/vpc-nat-gw-eip-z10eqvi9-0"
Feb 11 20:00:42 node-107 kubelet[5322]: E0211 20:00:42.797231    5322 kuberuntime_manager.go:1170] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"4102a18946ec67561023a0fa1a5dfe6bea3adddc52568962de885ebe30dc084c\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": dial unix /run/multus/multus.sock: connect: no such file or directory" pod="kube-system/vpc-nat-gw-eip-z10eqvi9-0"
Feb 11 20:00:42 node-107 kubelet[5322]: E0211 20:00:42.797271    5322 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"vpc-nat-gw-eip-z10eqvi9-0_kube-system(055d716e-2557-4639-9bfc-232f39b1eb67)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"vpc-nat-gw-eip-z10eqvi9-0_kube-system(055d716e-2557-4639-9bfc-232f39b1eb67)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"4102a18946ec67561023a0fa1a5dfe6bea3adddc52568962de885ebe30dc084c\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): failed to send CNI request: Post \\\"http://dummy/cni\\\": dial unix /run/multus/multus.sock: connect: no such file or directory\"" pod="kube-system/vpc-nat-gw-eip-z10eqvi9-0" podUID="055d716e-2557-4639-9bfc-232f39b1eb67"
[root@node-107 ~]# kubectl logs -f -n kube-system                    kube-multus-ds-f9zc9  | grep z10eqvi9
Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (init), delete-host-config (init)
DEL starting CNI request &{ContainerID:c74059ca69648e3fe178af80dbff267352aced35d0a6bdb17c7f8dd6d1e2938e Netns: IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_INFRA_CONTAINER_ID=c74059ca69648e3fe178af80dbff267352aced35d0a6bdb17c7f8dd6d1e2938e;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 49 45 107 117 98 101 45 111 118 110 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}
2025-02-11T20:01:12+08:00 [verbose] Del: kube-system:vpc-nat-gw-eip-z10eqvi9-0:055d716e-2557-4639-9bfc-232f39b1eb67:net-96m3optj:net1 {"cniVersion":"0.3.0","ipam":{"provider":"net-96m3optj.kube-system","server_socket":"/run/openvswitch/kube-ovn-daemon.sock","type":"kube-ovn"},"master":"ens2f1","mode":"bridge","name":"net-96m3optj","type":"macvlan"}
2025-02-11T20:01:12+08:00 [verbose] Del: kube-system:vpc-nat-gw-eip-z10eqvi9-0:055d716e-2557-4639-9bfc-232f39b1eb67:kube-ovn:eth0 {
2025-02-11T20:01:12+08:00 [verbose] DEL finished CNI request {ContainerID:c74059ca69648e3fe178af80dbff267352aced35d0a6bdb17c7f8dd6d1e2938e Netns: IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_INFRA_CONTAINER_ID=c74059ca69648e3fe178af80dbff267352aced35d0a6bdb17c7f8dd6d1e2938e;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 49 45 107 117 98 101 45 111 118 110 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}, result: "", err: DelegateDel: error invoking DelegateDel - "macvlan": error in getting result from DelNetwork: RPC failed; Post "http://dummy/api/v1/del": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory / DelegateDel: error invoking ConflistDel - "kube-ovn": conflistDel: error in getting result from DelNetworkList: plugin type="kube-ovn" failed (delete): RPC failed; Post "http://dummy/api/v1/del": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory
2025-02-11T20:01:12+08:00 [verbose] ADD starting CNI request &{ContainerID:8374b046516b9539f0a1f66bcd3326d5ea4866a42ccc694c43bafa1d66f62c5d Netns:/var/run/netns/cni-9dfd9cdf-20c6-fec5-afce-ea7420bdd65d IfName:eth0 Args:K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_INFRA_CONTAINER_ID=8374b046516b9539f0a1f66bcd3326d5ea4866a42ccc694c43bafa1d66f62c5d Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 49 45 107 117 98 101 45 111 118 110 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}
2025-02-11T20:02:27+08:00 [verbose] Add: kube-system:vpc-nat-gw-eip-z10eqvi9-0:055d716e-2557-4639-9bfc-232f39b1eb67:kube-ovn(kube-ovn):eth0 {"cniVersion":"0.3.1","interfaces":[{"name":"eth0","mac":"62:8b:55:12:26:c9","sandbox":"/var/run/netns/cni-9dfd9cdf-20c6-fec5-afce-ea7420bdd65d"}],"ips":[{"version":"4","interface":0,"address":"192.168.210.254/24","gateway":"192.168.210.1"}],"routes":[{"dst":"0.0.0.0/0","gw":"192.168.210.1"},{"dst":"192.168.210.0/24"}],"dns":{}}
I0211 20:02:27.528581   24460 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"vpc-nat-gw-eip-z10eqvi9-0", UID:"055d716e-2557-4639-9bfc-232f39b1eb67", APIVersion:"v1", ResourceVersion:"23121231", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [192.168.210.254/24] from kube-ovn
2025-02-11T20:03:20+08:00 [verbose] Add: kube-system:vpc-nat-gw-eip-z10eqvi9-0:055d716e-2557-4639-9bfc-232f39b1eb67:kube-system/net-96m3optj(net-96m3optj):net1 {"cniVersion":"0.3.0","interfaces":[{"name":"net1","mac":"12:e8:aa:42:c9:43","sandbox":"/var/run/netns/cni-9dfd9cdf-20c6-fec5-afce-ea7420bdd65d"}],"ips":[{"version":"4","interface":0,"address":"172.12.0.16/24"}],"dns":{}}
2025-02-11T20:03:20+08:00 [verbose] 2025-02-11T20:03:20+08:00 [verbose] I0211 20:03:20.042594   24460 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"vpc-nat-gw-eip-z10eqvi9-0", UID:"055d716e-2557-4639-9bfc-232f39b1eb67", APIVersion:"v1", ResourceVersion:"23121231", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add net1 [172.12.0.16/24] from kube-system/net-96m3optj
2025-02-11T20:03:20+08:00 [verbose] ADD starting delegate request &{ContainerID:fccd46cf945e9e031b397ac26a00499cae842cb925830cc51d3f06adea317d27 Netns:/proc/185521/ns/net IfName: Args:K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 105 112 97 109 34 58 123 34 112 114 111 118 105 100 101 114 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 46 107 117 98 101 45 115 121 115 116 101 109 34 44 34 115 101 114 118 101 114 95 115 111 99 107 101 116 34 58 34 47 114 117 110 47 111 112 101 110 118 115 119 105 116 99 104 47 107 117 98 101 45 111 118 110 45 100 97 101 109 111 110 46 115 111 99 107 34 44 34 116 121 112 101 34 58 34 107 117 98 101 45 111 118 110 34 125 44 34 109 97 115 116 101 114 34 58 34 101 110 115 50 102 49 34 44 34 109 111 100 101 34 58 34 98 114 105 100 103 101 34 44 34 110 97 109 101 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 34 44 34 116 121 112 101 34 58 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}
2025-02-11T20:03:21+08:00 [verbose] ADD finished Delegate request {ContainerID:fccd46cf945e9e031b397ac26a00499cae842cb925830cc51d3f06adea317d27 Netns:/proc/185521/ns/net IfName: Args:K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 105 112 97 109 34 58 123 34 112 114 111 118 105 100 101 114 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 46 107 117 98 101 45 115 121 115 116 101 109 34 44 34 115 101 114 118 101 114 95 115 111 99 107 101 116 34 58 34 47 114 117 110 47 111 112 101 110 118 115 119 105 116 99 104 47 107 117 98 101 45 111 118 110 45 100 97 101 109 111 110 46 115 111 99 107 34 44 34 116 121 112 101 34 58 34 107 117 98 101 45 111 118 110 34 125 44 34 109 97 115 116 101 114 34 58 34 101 110 115 50 102 49 34 44 34 109 111 100 101 34 58 34 98 114 105 100 103 101 34 44 34 110 97 109 101 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 34 44 34 116 121 112 101 34 58 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}, result: "", err: error configuring pod [kube-system/vpc-nat-gw-eip-z10eqvi9-0] networking: DelegateAdd: cannot set "macvlan" interface name to "": validateIfName: no net namespace /proc/185521/ns/net found: failed to Statfs "/proc/185521/ns/net": no such file or directory
2025-02-11T20:03:22+08:00 [verbose] ADD finished CNI request {ContainerID:8374b046516b9539f0a1f66bcd3326d5ea4866a42ccc694c43bafa1d66f62c5d Netns:/var/run/netns/cni-9dfd9cdf-20c6-fec5-afce-ea7420bdd65d IfName:eth0 Args:K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_INFRA_CONTAINER_ID=8374b046516b9539f0a1f66bcd3326d5ea4866a42ccc694c43bafa1d66f62c5d Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 49 45 107 117 98 101 45 111 118 110 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}, result: "{\"Result\":{\"cniVersion\":\"1.0.0\",\"interfaces\":[{\"name\":\"eth0\",\"mac\":\"62:8b:55:12:26:c9\",\"sandbox\":\"/var/run/netns/cni-9dfd9cdf-20c6-fec5-afce-ea7420bdd65d\"}],\"ips\":[{\"interface\":0,\"address\":\"192.168.210.254/24\",\"gateway\":\"192.168.210.1\"}],\"routes\":[{\"dst\":\"0.0.0.0/0\",\"gw\":\"192.168.210.1\"},{\"dst\":\"192.168.210.0/24\"}],\"dns\":{}}}", err: <nil>
2025-02-11T20:03:24+08:00 [verbose] ADD starting delegate request &{ContainerID:b5d886e2feb3955f3b363a4ea6a711c9e5c981a822c6de6913d48457d8ac27ea Netns:/proc/66343/ns/net IfName: Args:K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 105 112 97 109 34 58 123 34 112 114 111 118 105 100 101 114 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 46 107 117 98 101 45 115 121 115 116 101 109 34 44 34 115 101 114 118 101 114 95 115 111 99 107 101 116 34 58 34 47 114 117 110 47 111 112 101 110 118 115 119 105 116 99 104 47 107 117 98 101 45 111 118 110 45 100 97 101 109 111 110 46 115 111 99 107 34 44 34 116 121 112 101 34 58 34 107 117 98 101 45 111 118 110 34 125 44 34 109 97 115 116 101 114 34 58 34 101 110 115 50 102 49 34 44 34 109 111 100 101 34 58 34 98 114 105 100 103 101 34 44 34 110 97 109 101 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 34 44 34 116 121 112 101 34 58 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}
ADD finished Delegate request {ContainerID:b5d886e2feb3955f3b363a4ea6a711c9e5c981a822c6de6913d48457d8ac27ea Netns:/proc/66343/ns/net IfName: Args:K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 105 112 97 109 34 58 123 34 112 114 111 118 105 100 101 114 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 46 107 117 98 101 45 115 121 115 116 101 109 34 44 34 115 101 114 118 101 114 95 115 111 99 107 101 116 34 58 34 47 114 117 110 47 111 112 101 110 118 115 119 105 116 99 104 47 107 117 98 101 45 111 118 110 45 100 97 101 109 111 110 46 115 111 99 107 34 44 34 116 121 112 101 34 58 34 107 117 98 101 45 111 118 110 34 125 44 34 109 97 115 116 101 114 34 58 34 101 110 115 50 102 49 34 44 34 109 111 100 101 34 58 34 98 114 105 100 103 101 34 44 34 110 97 109 101 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 34 44 34 116 121 112 101 34 58 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}, result: "", err: error configuring pod [kube-system/vpc-nat-gw-eip-z10eqvi9-0] networking: interface name is empty
2025-02-11T20:04:31+08:00 [verbose] ADD starting delegate request &{ContainerID:b5d886e2feb3955f3b363a4ea6a711c9e5c981a822c6de6913d48457d8ac27ea Netns:/proc/66343/ns/net IfName: Args:K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 105 112 97 109 34 58 123 34 112 114 111 118 105 100 101 114 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 46 107 117 98 101 45 115 121 115 116 101 109 34 44 34 115 101 114 118 101 114 95 115 111 99 107 101 116 34 58 34 47 114 117 110 47 111 112 101 110 118 115 119 105 116 99 104 47 107 117 98 101 45 111 118 110 45 100 97 101 109 111 110 46 115 111 99 107 34 44 34 116 121 112 101 34 58 34 107 117 98 101 45 111 118 110 34 125 44 34 109 97 115 116 101 114 34 58 34 101 110 115 50 102 49 34 44 34 109 111 100 101 34 58 34 98 114 105 100 103 101 34 44 34 110 97 109 101 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 34 44 34 116 121 112 101 34 58 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}
2025-02-11T20:04:47+08:00 [verbose] ADD finished Delegate request {ContainerID:b5d886e2feb3955f3b363a4ea6a711c9e5c981a822c6de6913d48457d8ac27ea Netns:/proc/66343/ns/net IfName: Args:K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 105 112 97 109 34 58 123 34 112 114 111 118 105 100 101 114 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 46 107 117 98 101 45 115 121 115 116 101 109 34 44 34 115 101 114 118 101 114 95 115 111 99 107 101 116 34 58 34 47 114 117 110 47 111 112 101 110 118 115 119 105 116 99 104 47 107 117 98 101 45 111 118 110 45 100 97 101 109 111 110 46 115 111 99 107 34 44 34 116 121 112 101 34 58 34 107 117 98 101 45 111 118 110 34 125 44 34 109 97 115 116 101 114 34 58 34 101 110 115 50 102 49 34 44 34 109 111 100 101 34 58 34 98 114 105 100 103 101 34 44 34 110 97 109 101 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 34 44 34 116 121 112 101 34 58 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}, result: "", err: error configuring pod [kube-system/vpc-nat-gw-eip-z10eqvi9-0] networking: interface name is empty
2025-02-11T20:05:01+08:00 [verbose] ADD starting delegate request &{ContainerID:b5d886e2feb3955f3b363a4ea6a711c9e5c981a822c6de6913d48457d8ac27ea Netns:/proc/66343/ns/net IfName: Args:K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 105 112 97 109 34 58 123 34 112 114 111 118 105 100 101 114 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 46 107 117 98 101 45 115 121 115 116 101 109 34 44 34 115 101 114 118 101 114 95 115 111 99 107 101 116 34 58 34 47 114 117 110 47 111 112 101 110 118 115 119 105 116 99 104 47 107 117 98 101 45 111 118 110 45 100 97 101 109 111 110 46 115 111 99 107 34 44 34 116 121 112 101 34 58 34 107 117 98 101 45 111 118 110 34 125 44 34 109 97 115 116 101 114 34 58 34 101 110 115 50 102 49 34 44 34 109 111 100 101 34 58 34 98 114 105 100 103 101 34 44 34 110 97 109 101 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 34 44 34 116 121 112 101 34 58 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}
2025-02-11T20:05:27+08:00 [verbose] ADD finished Delegate request {ContainerID:b5d886e2feb3955f3b363a4ea6a711c9e5c981a822c6de6913d48457d8ac27ea Netns:/proc/66343/ns/net IfName: Args:K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=vpc-nat-gw-eip-z10eqvi9-0;K8S_POD_UID=055d716e-2557-4639-9bfc-232f39b1eb67 Path: StdinData:[123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 48 34 44 34 105 112 97 109 34 58 123 34 112 114 111 118 105 100 101 114 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 46 107 117 98 101 45 115 121 115 116 101 109 34 44 34 115 101 114 118 101 114 95 115 111 99 107 101 116 34 58 34 47 114 117 110 47 111 112 101 110 118 115 119 105 116 99 104 47 107 117 98 101 45 111 118 110 45 100 97 101 109 111 110 46 115 111 99 107 34 44 34 116 121 112 101 34 58 34 107 117 98 101 45 111 118 110 34 125 44 34 109 97 115 116 101 114 34 58 34 101 110 115 50 102 49 34 44 34 109 111 100 101 34 58 34 98 114 105 100 103 101 34 44 34 110 97 109 101 34 58 34 110 101 116 45 57 54 109 51 111 112 116 106 34 44 34 116 121 112 101 34 58 34 109 97 99 118 108 97 110 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 10 125 10]}, result: "", err: error configuring pod [kube-system/vpc-nat-gw-eip-z10eqvi9-0] networking: interface name is empty
^C

[root@node-107 ~]# kubectl logs -f -n kube-system                    kube-ovn-cni-b26dr  | grep z10eqvi9
Defaulted container "cni-server" out of: cni-server, install-cni (init)
I0211 20:01:53.759867   36156 handler.go:82] add port request: {kube-ovn vpc-nat-gw-eip-z10eqvi9-0 kube-system 8374b046516b9539f0a1f66bcd3326d5ea4866a42ccc694c43bafa1d66f62c5d /var/run/netns/cni-9dfd9cdf-20c6-fec5-afce-ea7420bdd65d eth0 ovn [] {[]  [] []}    }
I0211 20:03:19.764849   36156 handler.go:82] add port request: {macvlan vpc-nat-gw-eip-z10eqvi9-0 kube-system 8374b046516b9539f0a1f66bcd3326d5ea4866a42ccc694c43bafa1d66f62c5d /var/run/netns/cni-9dfd9cdf-20c6-fec5-afce-ea7420bdd65d net1 net-96m3optj.kube-system [] {[]  [] []}    }

[root@node-105 ~]#  kubectl logs -f -n kube-system                    kube-multus-ds-f9zc9 
Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (init), delete-host-config (init)
2025-02-11T20:00:46+08:00 [verbose] server configured with chroot: /hostroot
2025-02-11T20:00:46+08:00 [verbose] Generated MultusCNI config: {"capabilities":{"portMappings":true},"cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","clusterNetwork":"/host/etc/cni/net.d/01-kube-ovn.conflist","type":"multus-shim","socketDir":"/host/run/multus/"}
2025-02-11T20:00:46+08:00 [verbose] started to watch file /host/etc/cni/net.d/01-kube-ovn.conflist

Steps To Reproduce

  1. 创建多个nat-gw，绑定一些eip
  2. 重启集群节点

Current Behavior

nat-gw pod容器异常重启（pod未重建），已有的eip能重新绑定

Expected Behavior

nat-gw pod容器异常重启（pod未重建），已绑定的eip丢失
@kldancer kldancer added the bug Something isn't working label Feb 12, 2025
@dosubot dosubot bot added the eip label Feb 12, 2025
@dosubot DosuBot
Copy link

dosubot bot commented Feb 12, 2025

I found a similar open issue that might be relevant to your problem. The issue discusses EIP connectivity problems after restarting the kube-ovn network plugin. The suggested solutions include:

  1. Prevent serving traffic before routes and iptables rules are configured or flush conntrack entries without SNAT/DNAT after routes and iptables rules are configured.
  2. Disable ARP on the network interface before and after attaching the EIP.
  3. Use the release-1.12-mc branch or the master branch of kube-ovn, as many bugs related to ovn eip dnat snat fip were fixed in these versions [1].

To address the specific issue where the vpc-nat-gw pod's EIP is lost after a cluster node restart, ensure that the redo EIP logic is triggered. This can be achieved by handling the update of the VPC EIP properly. The function handleUpdateIptablesEip is responsible for refreshing existing EIPs and re-applying them if necessary. Make sure this function is called appropriately after a node restart to maintain the EIP association [2][3].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other | Bug Report

@zbb88888
Copy link
Collaborator 

  Warning  NodeNotReady            7m28s                  node-controller       Node is not ready
  Warning  FailedCreatePodSandBox  7m20s                  kubelet               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "992a0881ecf83dc1dcd71de5ed135d63c8bb06ae41d62a1b138c1ffee69f1c9b": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/multus.sock: connect: no such file or directory
  Warning  FailedCreatePodSandBox  7m6s                   kubelet               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4102a18946ec67561023a0fa1a5dfe6bea3adddc52568962de885ebe30dc084c": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/multus.sock: connect: no such file or directory
  Normal   SandboxChanged          6m51s (x3 over 7m20s)  kubelet               Pod sandbox changed, it will be killed and re-created.
  Normal   AddedInterface          5m21s                  multus                Add eth0 [192.168.210.254/24] from kube-ovn
  Normal   AddedInterface          4m28s                  multus                Add net1 [172.12.0.16/24] from kube-system/net-96m3optj
  Normal   Pulled                  4m25s                  kubelet               Container image "docker.io/kubeovn/vpc-nat-gateway:v1.12.22" already present on machine
  Normal   Created                 4m25s                  kubelet               Created container vpc-nat-gw-init
  Normal   Started                 4m25s                  kubelet               Started container vpc-nat-gw-init
  Normal   Pulled                  4m25s                  kubelet               Container image "docker.io/kubeovn/vpc-nat-gateway:v1.12.22" already present on machine
  Normal   Created                 4m25s                  kubelet               Created container vpc-nat-gw
  Normal   Started                 4m25s                  kubelet               Started container vpc-nat-gw
  Warning  FailedAddingInterface   2m21s (x4 over 4m27s)  pod-networks-updates  pod [kube-system/vpc-nat-gw-eip-z10eqvi9-0]: failed adding interface  to network: net-96m3optj

这里面的的 NodeNotReady ， /run/multus/multus.sock: connect:， failed adding interface to network: net-96m3optj

可能是 node 的网络配置有问题，node 没问题的话， pod 是可以建出来的。

如果 pod 没有ready，那eip的redo应该是失败的，会持续重试的。

@kldancer
Copy link
Contributor Author

kldancer commented Mar 7, 2025

Warning NodeNotReady 7m28s node-controller Node is not ready
Warning FailedCreatePodSandBox 7m20s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "992a0881ecf83dc1dcd71de5ed135d63c8bb06ae41d62a1b138c1ffee69f1c9b": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/multus.sock: connect: no such file or directory
Warning FailedCreatePodSandBox 7m6s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4102a18946ec67561023a0fa1a5dfe6bea3adddc52568962de885ebe30dc084c": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/multus.sock: connect: no such file or directory
Normal SandboxChanged 6m51s (x3 over 7m20s) kubelet Pod sandbox changed, it will be killed and re-created.
Normal AddedInterface 5m21s multus Add eth0 [192.168.210.254/24] from kube-ovn
Normal AddedInterface 4m28s multus Add net1 [172.12.0.16/24] from kube-system/net-96m3optj
Normal Pulled 4m25s kubelet Container image "docker.io/kubeovn/vpc-nat-gateway:v1.12.22" already present on machine
Normal Created 4m25s kubelet Created container vpc-nat-gw-init
Normal Started 4m25s kubelet Started container vpc-nat-gw-init
Normal Pulled 4m25s kubelet Container image "docker.io/kubeovn/vpc-nat-gateway:v1.12.22" already present on machine
Normal Created 4m25s kubelet Created container vpc-nat-gw
Normal Started 4m25s kubelet Started container vpc-nat-gw
Warning FailedAddingInterface 2m21s (x4 over 4m27s) pod-networks-updates pod [kube-system/vpc-nat-gw-eip-z10eqvi9-0]: failed adding interface to network: net-96m3optj
这里面的的 NodeNotReady ， /run/multus/multus.sock: connect:， failed adding interface to network: net-96m3optj

可能是 node 的网络配置有问题，node 没问题的话， pod 是可以建出来的。

如果 pod 没有ready，那eip的redo应该是失败的，会持续重试的。

集群所有节点重启，某一个nat-gw pod可能未能正常重建（仅仅是容器重启了，pod的creationTimestamp未变），观察到该nat-gw的pod的配置（曾添加的eip、iptables规则、路由）均丢失了，且无法恢复（其他绑定的相关资源的redo逻辑无法触发），实际上手动rm -f 一个正常nat-gw pod的“/pause”容器和“vpc-nat-gw”容器，就能够模拟这种情况。

正常环境上pod的容器可能因为各种不太可控的原因被杀掉重启，都会导致nat-gw pod的配置丢失，以这次的情况为例，用户可能不想因为仅仅集群节点重启，其中某个EIP可能就会异常不通，需要后台手动重建nat-gw pod 才能恢复😢。看是否能够使用容器最新的启动时间 pod.status.containerStatuses.state.running.startedAt来判断是否执行其他绑定资源的redo逻辑呢？

@zbb88888
Copy link
Collaborator

zbb88888 commented Mar 9, 2025

@kldancer 大佬，你可以试一下，一开始重启的逻辑只是基于 pod的creationTimestamp

@kldancer
Copy link
Contributor Author

@kldancer 大佬，你可以试一下，一开始重启的逻辑只是基于 pod的creationTimestamp

换了个思路，还是基于 pod的creationTimestamp。

  1. 在handleAddOrUpdatePod方法里，判断vpc natgw pod 如果 pod.Status.ContainerStatuses[] 中vpc-nat-gw容器的RestartCount >0，就给addOrUpdateVpcNatGatewayQueue入队，示意应该更新VpcNatGw pod。
  2. 在handleAddOrUpdateVpcNatGw方法里，尝试获取VpcNatGw pod，拿到RestartCount，如果RestartCount >0 说明StatefulSet应该重建pod， 设计一个注释来控制vpcnatgw StatefulSet是否应该重启 ，如果原本natgw的StatefulSet没有注释且RestartCount>0，则给StatefulSet.Spec.Template.Annotations添加上注释。

最终达到nat-gw pod容器发生重启后，能触发nat-gw StatefulSet的重建。
我在本地试了一下这种改动，能够达到该目的，看这种逻辑是否可行呢？

@zbb88888
Copy link
Collaborator

@kldancer 大佬，我感觉你的提议更合理，可以帮忙提个PR么？

This was referenced Mar 11, 2025
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working eip
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zbb88888 @kldancer