Flannel unable to find interface

[benmoss]

Here's my Flannel output:

PS C:\k>  C:/k//flannel-daemon.ps1
I0710 17:33:34.043295    4956 main.go:486] Determining IP address of default interface
I0710 17:33:34.258096    4956 main.go:499] Using interface with name vEthernet (Ethernet) and address 10.142.0.3
I0710 17:33:34.258096    4956 main.go:516] Defaulting external address to interface address (10.142.0.3)
I0710 17:33:34.290317    4956 kube.go:131] Waiting 10m0s for node controller to sync
I0710 17:33:34.291291    4956 kube.go:294] Starting kube subnet manager
I0710 17:33:35.293173    4956 kube.go:138] Node controller sync successful
I0710 17:33:35.293173    4956 main.go:240] Created subnet manager: Kubernetes Subnet Manager - windows01
I0710 17:33:35.302920    4956 main.go:243] Installing signal handlers
I0710 17:33:35.304874    4956 main.go:358] Found network config - Backend type: host-gw
I0710 17:33:35.327332    4956 hostgw_windows.go:116] Found existing HNS network [&{Id:32427549-8c0e-456d-8da7-ecda292a16a1 Name:cbr0 Type:l2bridge NetworkAdapterName: SourceMac: Policies:[] MacPools:[{StartMacAddress:00-15-5D-3A-80-00 EndMacAddress:00-15-5D-3A-8F-FF}] Subnets:[{AddressPrefix:10.233.67.0/24 GatewayAddress:10.233.67.1 Policies:[]}] DNSSuffix: DNSServerList: DNSServerCompartment:0 ManagementIP:10.142.0.3 AutomaticDNS:false}]
I0710 17:33:35.341976    4956 hostgw_windows.go:166] Found existing HNS bridge endpoint [&{Id:4b984054-39e8-4037-ad1e-2e23ce7f55d4 Name:cbr0_ep VirtualNetwork:32427549-8c0e-456d-8da7-ecda292a16a1 VirtualNetworkName:cbr0 Policies:[[123 32 34 84 121 112 101 34 32 58 32 34 76 50 68 114 105 118 101 114 34 32 125]] MacAddress:00-15-5D-3A-84-87 IPAddress:10.233.67.2 DNSSuffix: DNSServerList: GatewayAddress:10.233.67.1 EnableInternalDNS:false DisableICC:false PrefixLength:24 IsRemoteEndpoint:false}]
I0710 17:33:35.352718    4956 hostgw_windows.go:197] Attached bridge endpoint [cbr0_ep] to host
I0710 17:33:35.556816    4956 hostgw_windows.go:211] Enabled forwarding on [vEthernet (Ethernet)] index [8]
E0710 17:33:35.687650    4956 main.go:285] Error registering network: unable to find interface for IP Addess [10.233.67.2], error: Interface not found: 10.233.67.2
I0710 17:33:35.687650    4956 main.go:338] Stopping shutdownHandler...

and my netsh output:

PS C:\k> netsh interface ipv4 show addresses

Configuration for interface "vEthernet (Ethernet)"
    DHCP enabled:                         Yes
    IP Address:                           10.142.0.3
    Subnet Prefix:                        10.142.0.0/20 (mask 255.255.240.0)
    Default Gateway:                      10.142.0.1
    Gateway Metric:                       0
    InterfaceMetric:                      15

Configuration for interface "vEthernet (cbr0_ep)"
    DHCP enabled:                         No
    IP Address:                           10.233.0.1
    Subnet Prefix:                        10.0.0.0/8 (mask 255.0.0.0)
    IP Address:                           10.233.0.3
    Subnet Prefix:                        10.0.0.0/8 (mask 255.0.0.0)
    IP Address:                           10.233.16.214
    Subnet Prefix:                        10.0.0.0/8 (mask 255.0.0.0)
    IP Address:                           10.233.20.39
    Subnet Prefix:                        10.0.0.0/8 (mask 255.0.0.0)
    IP Address:                           10.233.67.2
    Subnet Prefix:                        10.233.67.0/24 (mask 255.255.255.0)
    Default Gateway:                      10.233.67.1
    Gateway Metric:                       256
    InterfaceMetric:                      15

Configuration for interface "Loopback Pseudo-Interface 1"
    DHCP enabled:                         No
    IP Address:                           127.0.0.1
    Subnet Prefix:                        127.0.0.0/8 (mask 255.0.0.0)
    InterfaceMetric:                      75

Configuration for interface "vEthernet (nat)"
    DHCP enabled:                         Yes
    IP Address:                           172.22.192.1
    Subnet Prefix:                        172.22.192.0/20 (mask 255.255.240.0)
    InterfaceMetric:                      5000

It seems all fine to me, or at least as far as I understand it, but from looking at the code it looks like https://github.com/ptylenda/flannel/blob/5237d22358e8bb89cfb3bd7716567aa798c25dee/vendor/github.com/rakelkar/gonetsh/netsh/netsh.go#L155 is going to make it only ever find the first IP for the interface.

[ptylenda]

@benmoss , yes it looks like the issue with the highlighted part of code. The exact change is here (by me): ptylenda/flannel@5237d22#diff-8326fda6ab4295b649c013b52359913c but it was a temporary workaround for problems with resolving the interface address when using kube-proxy in userspace. My original pull request is here: rakelkar/flannel#8 - treat it more like a "indication" of a problem, not a resolution of the bug.

And now it looks that something has changed in the way how netsh enumerates the interfaces, and there is a chance, that the original flannel code works properly. I would suggest trying the code from rakelkar: flannel-io/flannel#921, bear in mind you have it to compile manually, package it into tar.gz and then provide it over here: https://github.com/ptylenda/kubernetes-for-windows/blob/master/ansible/roles/windows/flannel/defaults/main.yml#L2

I know it is a long way to achieve the result, but unfortunately there has been very little offical commits around flannel in terms of Windows support over last 6 months :(

There is also another option, you may try getting rid of flannel totally and replacing it with wincni (similar to mentioned here in official docs: https://kubernetes.io/docs/getting-started-guides/windows/ - look for wincni.exe). I remember MS was recently open sourcing this CNI plugin and put it over here https://github.com/Microsoft/SDN/tree/master/Kubernetes/wincni

[benmoss]

@ptylenda switching back to rakelkar/flannel#8 did indeed get Flannel to start successfully, but now requests to my NodePort service aren't responding. I can still reach the pods from on the Windows machine. So complicated 😿

[pablodav]

@benmoss @ptylenda , the unique time when I got that error about interface not found was when I didn't get correct firewall ports openned.

Fixed in these lines:

https://github.com/ptylenda/kubernetes-for-windows/blob/master/ansible/roles/windows/flannel/tasks/main.yml#L10