Help diagnosing local testing networking issues with minikube

[mrocklin]

I'm no longer able to get my local environment to run tests. Worker can connect to the scheduler in the local process, but the host process can't connect down to the workers running in minikube. Historically I think I've resolved this with ip route, however the instructions in the README are no longer doing me much good.

mrocklin@carbon:~/workspace/dask-kubernetes$ sudo ip route add 172.17.0.0/16 via $(minikube ip)
[sudo] password for mrocklin: 
RTNETLINK answers: File exists
mrocklin@carbon:~/workspace/dask-kubernetes$ sudo cat /etc/docker/daemon.json
{
   "bip": "172.19.1.1/16"
}

This is stopping me from making progress on this library.

cc @yuvipanda in case he has suggestions

[jacobtomlinson]

@mrocklin do you still want this or has it gone stale now?

[mrocklin]

I'll reraise next time I run into this.

[limx0]

+1 also having this issue.

[yuvipanda]

@limx0 can you tell me what is the output of ip route show and minikube ip?

[limx0]

@yuvipanda I'm on OSX and using actually using docker-for-desktop now after having a bunch of issues getting minikube working consistently on my mac. Is there an equivalent for docker kubes?

[yuvipanda]

Ah, I've never used the kubernetes for docker setup, so can't help unfortunately.

[limx0]

Hey @yuvipanda, I got my minikube working finally, and still having the same issue.

After starting a single worker cluster with the instructions at https://kubernetes.dask.org/en/latest/,I can successfully submit a task

but when I try to gather I get the following:

distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://172.17.0.5:41291'], len-02801d9fe445b87768d77b61e9186579
NoneType: None
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'len-02801d9fe445b87768d77b61e9186579': ('tcp://172.17.0.5:41291',)}

As requested, my minikube ip

~$ minikube ip
192.168.99.100

and osx ip route show

~$  ip route show
default via 10.0.0.1 dev en0
10.0.0.0/24 dev en0  scope link
10.0.0.1/32 dev en0  scope link
10.0.0.8/32 dev en0  scope link
127.0.0.0/8 via 127.0.0.1 dev lo0
169.254.0.0/16 dev en0  scope link
192.168.99.0/24 dev vboxnet  scope link
224.0.0.0/4 dev en0  scope link
255.255.255.255/32 dev en0  scope link

[yuvipanda]

Unfortunately i don't know enough about OS X networking to help :( sorry!

[jjerphan]

I got the same problem ; see the last comment of #127.