test_decap execution fails at "warmup" test on S6100

[gechiang]

Description
When executing test case decap/test_decap.py it is always failing while executing "warmup" under ptftests/vxlan-decap.py.
If the exception handling under warmup is bypassed and set ret as success for this method, then the rest of the decap test testcases can run successfully to completion.
Not sure what is causing the "warmup" to hit exception...

Steps to reproduce the issue:

1. Manually run vxlan/test_vxlan_decap.py on platform S6100

Describe the results you received:
Observe the test failed with result with something as shown here:

        if (res.is_failed or 'exception' in res) and not module_ignore_errors:
>           raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res)
E           RunAnsibleModuleFail: run module shell failed, Ansible Results =>
E           {
E               "changed": true,
E               "cmd": "ptf --test-dir ptftests vxlan-decap.Vxlan --platform-dir ptftests --qlen=10000 --platform remote -t 'vxlan_enabled=True;count=10;config_file='\"'\"'/tmp/vxlan_decap.json'\"'\"';sonic_admin_user=u'\"'\"'admin'\"'\"';dut_host=u'\"'\"'10.3.147.243'\"'\"';sonic_admin_password=u'\"'\"'password'\"'\"'' --relax --debug info --log-file /tmp/vxlan-decap.Vxlan.Enabled.2021-01-20-05:35:40.log",
E               "delta": "0:01:45.018798",
E               "end": "2021-01-20 06:05:36.721712",
E               "failed": true,
E               "invocation": {
E                   "module_args": {
E                       "_raw_params": "ptf --test-dir ptftests vxlan-decap.Vxlan --platform-dir ptftests --qlen=10000 --platform remote -t 'vxlan_enabled=True;count=10;config_file='\"'\"'/tmp/vxlan_decap.json'\"'\"';sonic_admin_user=u'\"'\"'admin'\"'\"';dut_host=u'\"'\"'10.3.147.243'\"'\"';sonic_admin_password=u'\"'\"'password'\"'\"'' --relax --debug info --log-file /tmp/vxlan-decap.Vxlan.Enabled.2021-01-20-05:35:40.log",
E                       "_uses_shell": true,
E                       "argv": null,
E                       "chdir": "/root",
E                       "creates": null,
E                       "executable": null,
E                       "removes": null,
E                       "stdin": null,
E                       "stdin_add_newline": true,
E                       "strip_empty_ends": true,
E                       "warn": true
E                   }
E               },
E               "msg": "non-zero return code",
E               "rc": 1,
E               "start": "2021-01-20 06:03:51.702914",
E               "stderr": "WARNING: No route found for IPv6 destination :: (no default route?)\n/usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.\n  from cryptography.hazmat.backends import default_backend\nvxlan-decap.Vxlan ... FAIL\n\n======================================================================\nFAIL: vxlan-decap.Vxlan\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"ptftests/vxlan-decap.py\", line 391, in runTest\n    self.warmup()\n  File \"ptftests/vxlan-decap.py\", line 328, in warmup\n    raise AssertionError(\"Warmup failed\")\nAssertionError: Warmup failed\n\n----------------------------------------------------------------------\nRan 1 test in 102.951s\n\nFAILED (failures=1)",
E               "stderr_lines": [
E                   "WARNING: No route found for IPv6 destination :: (no default route?)",
E                   "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.",
E                   "  from cryptography.hazmat.backends import default_backend",
E                   "vxlan-decap.Vxlan ... FAIL",
E                   "",
E                   "======================================================================",
E                   "FAIL: vxlan-decap.Vxlan",
E                   "----------------------------------------------------------------------",
E                   "Traceback (most recent call last):",
E                   "  File \"ptftests/vxlan-decap.py\", line 391, in runTest",
E                   "    self.warmup()",
E                   "  File \"ptftests/vxlan-decap.py\", line 328, in warmup",
E                   "    raise AssertionError(\"Warmup failed\")",
E                   "AssertionError: Warmup failed",
E                   "",
E                   "----------------------------------------------------------------------",
E                   "Ran 1 test in 102.951s",
E                   "",
E                   "FAILED (failures=1)"
E               ],
E               "stdout": "",
E               "stdout_lines": []
E           }

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

Any recent master image should reproduce with same result. But a private BRCM SAI fix for syncd crash is required

admin@str-s6100-acs-1:~$ show vers

SONiC Software Version: SONiC.master.0-dirty-20210122.032139
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: 3c16f803
Build date: Fri Jan 22 06:48:43 UTC 2021
Built by: xxx@xxx-vm-01

[anshuv-mfst]

Issue triage meeting 2/3:
@volodymyrsamotiy could you please provide input, thanks.

[volodymyrsamotiy]

Not sure if it is the same but recently we observed similar issue, please find details below, maybe it will help if it is the same.

In issue that we observed "vxlan-decap" test itself and VxLAN funtionality are working without any issues.
Only "warmup" test case failure is seen due to other issues related to "show fdb" and "show arp" CLI commands.
It is not always reproducible and observed only in specific cases.

It happens because in "vxlan-decap" test there is "wait_dut()" function which checks output of "show fdb" and "show arp".
https://github.com/Azure/sonic-mgmt/blob/master/ansible/roles/test/files/ptftests/vxlan-decap.py#L312

So, if output is invalid or displays an error or exception then "vxlan-decap" will fail.
In our case both these commands show the following error: "Failed to get Vlan id for bvid oid:0x26000000000013".
As a result "vxlan-decap" test failure is observed.

They are known issues and there are already PRs with fixes but one is not merged yet.

Issues:

• show mac display wrong information for tagged vlan  sonic-utilities#894 show mac display wrong information for tagged vlan
• Command "show arp" returns error: "int() argument must be a string, a bytes-like object or a number, not 'NoneType'" #6367 Command "show arp" returns error: "int() argument must be a string, a bytes-like object or a number, not 'NoneType'"

PRs:

• [show] Fix 'show mac' output, when FDB entry with Vlan 1 is present sonic-utilities#1368 [show] Fix 'show mac' output, when FDB entry with Vlan 1 is present
• [show] Fix show arp in case with FDB entries, linked to default VLAN sonic-utilities#1357 [show] Fix show arp in case with FDB entries, linked to default VLAN

[gechiang]

Saw this again.
Found the root cause.
Will open a PR to fix this...
Also the test case mentioned when I opened this was incorrect. It was the "vxlan/test_vxlan_decap.py" test case that encountered this issue.

[gechiang]

Fix checked in.
Closing this issue.