Description
The test case test_hash will extract route from APPL_DB by redis-dump -d 0 -k 'ROUTE*' -y.
Recently, test_hash was consistently failing with exception
E "Traceback (most recent call last):",
E " File \"ptftests/hash_test.py\", line 264, in runTest",
E " self.check_hash(hash_key)",
E " File \"ptftests/hash_test.py\", line 83, in check_hash",
E " (matched_index, _) = self.check_ip_route(hash_key, in_port, dst_ip, exp_port_list)",
E " File \"ptftests/hash_test.py\", line 91, in check_ip_route",
E " (matched_index, received) = self.check_ipv4_route(hash_key, in_port, dst_port_list)",
E " File \"ptftests/hash_test.py\", line 161, in check_ipv4_route",
E " return verify_packet_any_port(self, masked_exp_pkt, dst_port_list)",
E " File \"/usr/lib/python2.7/dist-packages/ptf/testutils.py\", line 2538, in verify_packet_any_port",
E " % (ports, device_number, result.format()))",
E "AssertionError: Did not receive expected packet on any of ports [] for device 0.",
The exception was because of incorrect default route, which results in empty exp_port_list.
"ROUTE_TABLE:0.0.0.0/0": {
"expireat": 1610968205.649774,
"ttl": -0.001,
"type": "hash",
"value": {
"ifname": "eth0",
"nexthop": "10.3.146.1"
}
}
The most bizarre thing is the route in kernel is correct.
root@str2-7260cx3-acs-9:~# show ip route 0.0.0.0
Routing entry for 0.0.0.0/0
Known via "bgp", distance 20, metric 0, best
Last update 02:06:27 ago
* 10.0.0.33, via PortChannel0001, weight 1
* 10.0.0.35, via PortChannel0002, weight 1
* 10.0.0.37, via PortChannel0003, weight 1
* 10.0.0.39, via PortChannel0004, weight 1
Still not sure if this issue is caused by previously running test cases or triggered by some operations.
Steps to reproduce the issue:
The issue is not easy to repro when running manually. But was consistently failing on regression test.
- Run
test_hash
Describe the results you received:
Test case should pass.
Describe the results you expected:
Test case failed.
Additional information you deem important (e.g. issue happens only occasionally):
**Output of `show version`:**
SONiC Software Version: SONiC.HEAD.260-1dcab4d1
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: 1dcab4d1
Build date: Wed Jan 13 14:30:11 UTC 2021
Built by: johnar@jenkins-worker-22
Platform: x86_64-arista_7260cx3_64
HwSKU: Arista-7260CX3-D108C8
ASIC: broadcom
ASIC Count: 1
Serial Number: JPE20271584
Uptime: 12:05:09 up 19 min, 1 user, load average: 3.11, 2.55, 1.92
Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm HEAD.260-1dcab4d1 ca05b2873d11 640MB
docker-syncd-brcm latest ca05b2873d11 640MB
docker-snmp HEAD.260-1dcab4d1 31464fc4ec71 434MB
docker-snmp latest 31464fc4ec71 434MB
docker-teamd HEAD.260-1dcab4d1 636c9ea31de4 462MB
docker-teamd latest 636c9ea31de4 462MB
docker-nat HEAD.260-1dcab4d1 c5b3d6107182 465MB
docker-nat latest c5b3d6107182 465MB
docker-router-advertiser HEAD.260-1dcab4d1 96bb9ae5db0f 394MB
docker-router-advertiser latest 96bb9ae5db0f 394MB
docker-platform-monitor HEAD.260-1dcab4d1 cd0aa4b805a8 599MB
docker-platform-monitor latest cd0aa4b805a8 599MB
docker-lldp HEAD.260-1dcab4d1 f32e1e39f80a 434MB
docker-lldp latest f32e1e39f80a 434MB
docker-dhcp-relay HEAD.260-1dcab4d1 121828b00a2f 400MB
docker-dhcp-relay latest 121828b00a2f 400MB
docker-sonic-mgmt-framework HEAD.260-1dcab4d1 35d87d2d0014 607MB
docker-sonic-mgmt-framework latest 35d87d2d0014 607MB
docker-orchagent HEAD.260-1dcab4d1 b7ac345616a5 480MB
docker-orchagent latest b7ac345616a5 480MB
docker-macsec HEAD.260-1dcab4d1 dd50a44dd043 465MB
docker-macsec latest dd50a44dd043 465MB
docker-sonic-telemetry HEAD.260-1dcab4d1 01d17b325f2a 468MB
docker-sonic-telemetry latest 01d17b325f2a 468MB
docker-fpm-frr HEAD.260-1dcab4d1 7e22fd0f3300 479MB
docker-fpm-frr latest 7e22fd0f3300 479MB
docker-sflow HEAD.260-1dcab4d1 5f969b191610 463MB
docker-sflow latest 5f969b191610 463MB
docker-database HEAD.260-1dcab4d1 250a04c60781 393MB
docker-database latest 250a04c60781 393MB
**Attach debug file `sudo generate_dump`:**
```
(paste your output here)
```
Description
The test case
test_hashwill extract route fromAPPL_DBbyredis-dump -d 0 -k 'ROUTE*' -y.Recently,
test_hashwas consistently failing with exceptionThe exception was because of incorrect default route, which results in empty
exp_port_list.The most bizarre thing is the route in kernel is correct.
Still not sure if this issue is caused by previously running test cases or triggered by some operations.
Steps to reproduce the issue:
The issue is not easy to repro when running manually. But was consistently failing on regression test.
test_hashDescribe the results you received:
Test case should pass.
Describe the results you expected:
Test case failed.
Additional information you deem important (e.g. issue happens only occasionally):