In 1.19, iptables reported error '[unsupported revision]' for DNAT rules

[Shuanglu]

What happened:
I'm using kubeadm to build a cluster with 1.19 version. The kube-proxy is running fine but there is error with the DNAT rules like below.
-A KUBE-SEP-FRUH4FMRRHR5RCGV -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT [unsupported revision]
I increased the kube-proxy log level to 9 but didn't find error about appending the DNAT rules
What you expected to happen:
DNAT in iptables works fine
How to reproduce it (as minimally and precisely as possible):

1. follow the tutorial to build cluster with kubeadm
2. check the iptables after cluster provisioning.
   Anything else we need to know?:
3. root@kubeadm-m1:/home/testshuang/calico# cat /proc/sys/net/ipv4/ip_forward
   1
4. If I manually add a rule with DNAT, it works fine

Environment:

• Kubernetes version (use kubectl version): 1.19.0
• Cloud provider or hardware configuration: Azure
• OS (e.g: cat /etc/os-release):
  NAME="Ubuntu"
  VERSION="18.04.5 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.5 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic
• Kernel (e.g. uname -a):
  Linux kubeadm-m1 5.4.0-1025-azure Change 'this' varable to more sensible abbreviation #25~18.04.1-Ubuntu SMP Sat Sep 5 15:28:57 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
  kubeadm
• Network plugin and version (if this is a network-related bug):
  calico
• Others:
  iptables: iptables-save v1.6.1

[Shuanglu]

/sig network

[athenabot]

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

[jayunit100]

I think this is an ubuntu 18.04 issue, robbertkl/docker-ipv6nat#47 .... something buggy specifically w/ that version of iptables in ubuntu :) . can you upgrade and see if that fixes it ?

[jayunit100]

(assuming this can be closed but ill leave it open for now)

[Shuanglu]

/sig network

I think this is an ubuntu 18.04 issue, robbertkl/docker-ipv6nat#47 .... something buggy specifically w/ that version of iptables in ubuntu :) . can you upgrade and see if that fixes it ?

thanks a lot. it's fixed by upgrading the iptables. I'll close this issue.

[stefanlasiewski]

Just FTR: This is happening to me as well with Ubuntu 18.04 and Kubernetes 16.13.

# iptables-save > iptables-save
# iptables-restore < iptables-save
Bad argument `[unsupported'
Error occurred at line: 483
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
# 

Unfortunately, I can't just upgrade from Ubuntu 18 -> 20 to fix one bug.

[Shuanglu]

Unfortunately, I can't just upgrade from Ubuntu 18 -> 20 to fix one bug.

you may add the source list of ubuntu 20.04 focal to your repo list and upgrade the iptable i think

[stefanlasiewski]

Right, and that workaround might be okay for some people. Mixing OS versions isn't a good idea for a production system, especially for something as critical as iptables. We want to stick with something solid and tested.

If I find the cause on my system, I'll post here FTR.

Seems that Docker itself also has this problem: moby/moby#40428

[oxr463]

@stefanlasiewski perhaps one could file a ticket on launchpad to backport the newer iptables version since 18.04 is an LTS release?

[stefanlasiewski]

@oxr463 Yes perhaps. I'll see if a ticket is out there already.

I think the bigger issue is that Kubernetes needs to maintain solid support for Ubuntu 18.04 LTS which I'm sure is still one the top 2 OS's used for servers. It's going to take a year for 20.04 LTS to hit that milestone.

[stefanlasiewski]

I tracked down a related bug for regular Ubuntu 18.04 with the HWE Kernel (Kernel v5 instead of Kernel v4) and Kubernetes 1.17, and filed it here:

#95409

[sacheth003]

Hey @stefanlasiewski, I'm running into a similar issue and it's not feasible to simply upgrade to Ubuntu 20 to get around it. I couldn't fully make out whether there was a resolution from the bug that you filed. Was there a way to get around it, or was the resolution that the error wasn't "actually an error"?

[stefanlasiewski]

@sacheth003

It's a bug within Ubuntu regarding the HWE kernel. When I uninstalled the HWE kernel, the problem went away.

I filed a bug report at https://bugs.launchpad.net/ubuntu/+source/linux-meta-hwe-5.4/+bug/1899690 . If you have the same problem, please add yourself as an "Affected User" to help raise the score.

It's also not feasible for us to simply upgrade to Ubuntu 20 to address one bug.

[stefanlasiewski]

Also, this affects all versions that I've tried: Kubernetes 1.16, 1.17 & 1.18.