Pod unable to reach itself through a service (unless --cni=true is set)

[arrawatia]

Minikube version (use minikube version):
minikube version: v0.16.0 and k8s version v1.6.4
(But I tried v0.17.1 and v0.19.1 too)
Environment:

• OS (e.g. from /etc/os-release):
• VM Driver :
  Virtualbox
• ISO version :
  minikube-v1.0.6.iso
• Install tools:
  curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.16.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
• Others:

What happened:
If a pod has a service which points to the pod, the pod cannot reach itself through the service IP. Other pods can reach the service and the pod itself can reach other services. This means all components (especially clustered & distributed systems) which expect to talk to themselves for leader election fail to startup properly.

What you expected to happen:
I expect the pod to be able to reach itself.

How to reproduce it (as minimally and precisely as possible):
It happens with all our services and pods but I can reproduce it with kube-system pods too.

Get service IP : kubectl describe svc kube-dns --namespace kube-system | grep IP:. I get 10.0.0.10
Get endpoint IP: kubectl describe svc kube-dns --namespace kube-system | grep Endpoints. I get 172.17.0.3

Exec into the pod:
kubectl --namespace kube-system exec -it kube-dns-v20-54536 sh

Run the following :
Using the service IP hangs

Name: kubernetes-dashboard.kube-system.svc.cluster.local
Address 1: 10.0.0.212 kubernetes-dashboard.kube-system.svc.cluster.local
/ # nslookup kubernetes-dashboard.kube-system.svc.cluster.local 10.0.0.10
Server: 10.0.0.10
^C

Using the endoint IP works

/ # nslookup kubernetes-dashboard.kube-system.svc.cluster.local 172.17.0.3
Server: 172.17.0.3
Address 1: 172.17.0.3 kube-dns-v20-54536

Name: kubernetes-dashboard.kube-system.svc.cluster.local
Address 1: 10.0.0.212 kubernetes-dashboard.kube-system.svc.cluster.local

Accessing a different service IP works. Using the kubernetes-dashboard IP from the last command
/ # telnet 10.0.0.212 80
get
HTTP/1.1 400 Bad Request
Content-Type: text/plain
Connection: close

400 Bad RequestConnection closed by foreign host

Anything else do we need to know:
minikube v0.17.1 works with K8S 1.5.3
I tried the following and it worked. So, I suspect it has something to do with upgrading minikube to v0.17.1 and K8S to v1.6.4

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.17.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
rm -rf ~/.minikube
minikube start --kubernetes-version 1.5.3 --cpus 4 --memory 6096 --v=8 --logtostderr

[ursuad]

It seems it's related to this: kubernetes/kubernetes#19930 and this kubernetes/kubernetes#20475

[ursuad]

For me, this helped to fix it kubernetes/kubernetes#20475 (comment)

So you can do:

minikube ssh
sudo ip link set docker0 promisc on

Maybe this fix can be merged directly in minikube so people won;t need to do custom things?

[arrawatia]

@ursuad It worked for me too (minikube 0.19.1 and k8s v1.6.4). Thanks a lot for your help. :)

[ursuad]

@arrawatia No problem, but maybe you should leave this issue open, so we'll have a longer term fix merged in minkube.
This is a bug, and my fix is just a workaround.

[rajasaur]

Not sure if there is a page to capture minikube gotchas. @ursuad's suggestion should be there

[yonatane]

Use Headless Service as a workaround (clusterIP: None). StatefulSets that talk to themselves like Kafka use headless services anyway.
This might still be considered a bug though.

[ursuad]

@kubernetes/sig-minikube

[r2d4]

I'll take a look at this

[medyagh]

@r2d4 I am pretty sure this is same as this #1690

[medyagh]

@arrawatia can you try with minikube with k8s 1.5.1 I am almost certain it will work in 1.5.1
I have that problem ! #1690