contiv-vpp can not start up when kube-proxy with ipvs mode

[LipingMao]

Issue:
In the default contiv-vpp configure, contiv-etcd will start up with nodeport, and other conponent will connect to 127.0.0.1:$ETCDPORT to access etcd. But when kube-proxy enable ipvs, it does not support localhost:Nodeport now [1].
So if you deploy contiv-vpp with ipvs the default yaml in [2] will not work, such as, It will wait for 127.0.0.1:$ETCDPORT , if the pod not on the same host of contiv-etcd, it will failed here.(this is normal case if you have 3 kube-master)

  initContainers:
    # This init container waits until etcd is started
    - name: wait-foretcd
      env:
        - name: ETCDPORT
          value: "32379"
      image: busybox:1.29.3
      imagePullPolicy: IfNotPresent

command: ['sh', '-c', 'until nc -w 2 127.0.0.1:$ETCDPORT; do echo waiting for etcd; sleep 2; done;']

[1] kubernetes/kubernetes#67730
[2] https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp.yaml

[rastislavs]

Hi,
thanks for letting us know. We will try to implement a workaround/some other solution instead of localhost:<nodeport>.

[LipingMao]

@rastislavszabo thanks for the info. And another thing is HA for contiv-etcd, deafult yaml is using statefulset with one etcd, there would be HA risk in the case with multi k8s master. Do we have plan to make it in full HA and with TLS support to etcd?

[rastislavs]

TLS support for ETCD is documented here: https://github.com/contiv/vpp/blob/master/docs/setup/ETCD_SECURITY.md

As for the multi-master, we plan to add support for it (not supported yet). In that case, we would most likely rely on ETCD cluster configured by the administrator instead of the one that we provide via statefulset.

[LipingMao]

@rastislavszabo thanks for the info, nice to know.

[rastislavs]

Hi,
the localhost nodeport issue should be fixed by #1434 - could you please verify?

[LipingMao]

@rastislavszabo I would try it today, and update here if there is any issue, appreciate your help!

[LipingMao]

@rastislavszabo
It works for me by using the updated yaml, in my 3 controller + ipvs test env:

[root@ams-dev-ctrl001 ~]# kubectl get pods -o wide --all-namespaces | grep contiv
kube-system contiv-crd-g2m9t 1/1 Running 0 43m 10.253.111.168 ams-dev-ctrl002
kube-system contiv-crd-t6xpw 1/1 Running 0 43m 10.253.111.160 ams-dev-ctrl003
kube-system contiv-crd-zv47q 1/1 Running 0 43m 10.253.111.165 ams-dev-ctrl001
kube-system contiv-etcd-0 1/1 Running 0 43m 10.253.111.165 ams-dev-ctrl001
kube-system contiv-ksr-7clqz 1/1 Running 0 43m 10.253.111.168 ams-dev-ctrl002
kube-system contiv-ksr-cnqpk 1/1 Running 0 43m 10.253.111.165 ams-dev-ctrl001
kube-system contiv-ksr-q7blv 1/1 Running 0 43m 10.253.111.160 ams-dev-ctrl003
kube-system contiv-vswitch-g4rbr 1/1 Running 0 43m 10.253.111.165 ams-dev-ctrl001
kube-system contiv-vswitch-ngdvs 1/1 Running 0 43m 10.253.111.162 ams-dev-work002
kube-system contiv-vswitch-prfzp 1/1 Running 0 43m 10.253.111.148 ams-dev-work001
kube-system contiv-vswitch-scjd6 1/1 Running 0 43m 10.253.111.160 ams-dev-ctrl003
kube-system contiv-vswitch-v47rn 1/1 Running 0 43m 10.253.111.168 ams-dev-ctrl002
[root@ams-dev-ctrl001 ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.17.0.1:32379 rr
-> ams-dev-ctrl001:12379 Masq 1 0 0
TCP 172.30.3.2:32379 rr
-> ams-dev-ctrl001:12379 Masq 1 0 0
TCP 10.96.0.1:https rr
-> ams-dev-ctrl003:sun-sr-https Masq 1 0 0
-> ams-dev-ctrl001:sun-sr-https Masq 1 2 0
-> ams-dev-ctrl002:sun-sr-https Masq 1 0 0
TCP 10.96.0.3:domain rr
-> 10.1.1.4:domain Masq 1 0 0

Thanks it for your helps, but before moving to forward, we would use external etcd cluster to instead of one stateful set. contiv-etcd is still single point of failure.