Description
Description
The test case test_hash will extract route from APPL_DB by redis-dump -d 0 -k 'ROUTE*' -y.
Recently, test_hash was consistently failing with exception
E "Traceback (most recent call last):",
E " File \"ptftests/hash_test.py\", line 264, in runTest",
E " self.check_hash(hash_key)",
E " File \"ptftests/hash_test.py\", line 83, in check_hash",
E " (matched_index, _) = self.check_ip_route(hash_key, in_port, dst_ip, exp_port_list)",
E " File \"ptftests/hash_test.py\", line 91, in check_ip_route",
E " (matched_index, received) = self.check_ipv4_route(hash_key, in_port, dst_port_list)",
E " File \"ptftests/hash_test.py\", line 161, in check_ipv4_route",
E " return verify_packet_any_port(self, masked_exp_pkt, dst_port_list)",
E " File \"/usr/lib/python2.7/dist-packages/ptf/testutils.py\", line 2538, in verify_packet_any_port",
E " % (ports, device_number, result.format()))",
E "AssertionError: Did not receive expected packet on any of ports [] for device 0.",
The exception was because of incorrect default route, which results in empty exp_port_list.
"ROUTE_TABLE:0.0.0.0/0": {
"expireat": 1610968205.649774,
"ttl": -0.001,
"type": "hash",
"value": {
"ifname": "eth0",
"nexthop": "10.3.146.1"
}
}
The most bizarre thing is the route in kernel is correct.
root@str2-7260cx3-acs-9:~# show ip route 0.0.0.0
Routing entry for 0.0.0.0/0
Known via "bgp", distance 20, metric 0, best
Last update 02:06:27 ago
* 10.0.0.33, via PortChannel0001, weight 1
* 10.0.0.35, via PortChannel0002, weight 1
* 10.0.0.37, via PortChannel0003, weight 1
* 10.0.0.39, via PortChannel0004, weight 1
Still not sure if this issue is caused by previously running test cases or triggered by some operations.
Steps to reproduce the issue:
The issue is not easy to repro when running manually. But was consistently failing on regression test.
- Run
test_hash
Describe the results you received:
Test case should pass.
Describe the results you expected:
Test case failed.
Additional information you deem important (e.g. issue happens only occasionally):
**Output of `show version`:**
SONiC Software Version: SONiC.HEAD.260-1dcab4d1
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: 1dcab4d1
Build date: Wed Jan 13 14:30:11 UTC 2021
Built by: johnar@jenkins-worker-22
Platform: x86_64-arista_7260cx3_64
HwSKU: Arista-7260CX3-D108C8
ASIC: broadcom
ASIC Count: 1
Serial Number: JPE20271584
Uptime: 12:05:09 up 19 min, 1 user, load average: 3.11, 2.55, 1.92
Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm HEAD.260-1dcab4d1 ca05b2873d11 640MB
docker-syncd-brcm latest ca05b2873d11 640MB
docker-snmp HEAD.260-1dcab4d1 31464fc4ec71 434MB
docker-snmp latest 31464fc4ec71 434MB
docker-teamd HEAD.260-1dcab4d1 636c9ea31de4 462MB
docker-teamd latest 636c9ea31de4 462MB
docker-nat HEAD.260-1dcab4d1 c5b3d6107182 465MB
docker-nat latest c5b3d6107182 465MB
docker-router-advertiser HEAD.260-1dcab4d1 96bb9ae5db0f 394MB
docker-router-advertiser latest 96bb9ae5db0f 394MB
docker-platform-monitor HEAD.260-1dcab4d1 cd0aa4b805a8 599MB
docker-platform-monitor latest cd0aa4b805a8 599MB
docker-lldp HEAD.260-1dcab4d1 f32e1e39f80a 434MB
docker-lldp latest f32e1e39f80a 434MB
docker-dhcp-relay HEAD.260-1dcab4d1 121828b00a2f 400MB
docker-dhcp-relay latest 121828b00a2f 400MB
docker-sonic-mgmt-framework HEAD.260-1dcab4d1 35d87d2d0014 607MB
docker-sonic-mgmt-framework latest 35d87d2d0014 607MB
docker-orchagent HEAD.260-1dcab4d1 b7ac345616a5 480MB
docker-orchagent latest b7ac345616a5 480MB
docker-macsec HEAD.260-1dcab4d1 dd50a44dd043 465MB
docker-macsec latest dd50a44dd043 465MB
docker-sonic-telemetry HEAD.260-1dcab4d1 01d17b325f2a 468MB
docker-sonic-telemetry latest 01d17b325f2a 468MB
docker-fpm-frr HEAD.260-1dcab4d1 7e22fd0f3300 479MB
docker-fpm-frr latest 7e22fd0f3300 479MB
docker-sflow HEAD.260-1dcab4d1 5f969b191610 463MB
docker-sflow latest 5f969b191610 463MB
docker-database HEAD.260-1dcab4d1 250a04c60781 393MB
docker-database latest 250a04c60781 393MB
**Attach debug file `sudo generate_dump`:**
```
(paste your output here)
```