Skip to content

Directed broadcast test fails , packets are dropped at egress #3224

@ArunSaravananBalachandran

Description

Description

Directed broadcast test fails for Trident 3 platforms. The packets are dropped at egress.

Steps to reproduce the issue:

  1. Run https://github.com/Azure/sonic-mgmt/blob/master/ansible/roles/test/tasks/dir_bcast.yml

Upon sending the traffic through script , the packet drops are seen at egress

Describe the results you received:

Script logs:

TASK [test : PTF Test - dir_bcast test] ****************************************
task path: /var/johnar/sonic-mgmt/ansible/roles/test/tasks/ptf_runner.yml:42
Friday 26 July 2019  16:24:25 +0000 (0:00:00.131)       0:00:18.018 *********** 
ok: [sonic-s5232-64] => {
    "msg": "ptf --test-dir ptftests dir_bcast_test.BcastTest   --platform-dir ptftests  --platform remote  -t \"testbed_type='t0';router_mac='3c:2c:30:6d:80:00';vlan_info='/root/vlan_info.txt'\"  --relax --debug info --log-file /tmp/dir_bcast.BcastTest.2019-07-26-16:24:25.log 2>&1"
}

TASK [test : command] **********************************************************
task path: /var/johnar/sonic-mgmt/ansible/roles/test/tasks/ptf_runner.yml:45
Friday 26 July 2019  16:24:26 +0000 (0:00:00.181)       0:00:18.199 *********** 
<10.16.210.140> ESTABLISH SSH CONNECTION FOR USER: root
<10.16.210.140> SSH: EXEC sshpass -d15 ssh -C -q -o ControlMaster=auto -o ControlPersist=120s -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r 10.16.210.140 'LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python'
changed: [sonic-s5232-64 -> 10.16.210.140] => {"changed": true, "cmd": "ptf --test-dir ptftests dir_bcast_test.BcastTest   --platform-dir ptftests  --platform remote  -t \"testbed_type='t0';router_mac='3c:2c:30:6d:80:00';vlan_info='/root/vlan_info.txt'\"  --relax --debug info --log-file /tmp/dir_bcast.BcastTest.2019-07-26-16:24:26.log 2>&1", "delta": "0:00:02.951175", "end": "2019-07-26 16:24:29.218739", "failed": false, "failed_when_result": false, "invocation": {"module_args": {"_raw_params": "ptf --test-dir ptftests dir_bcast_test.BcastTest   --platform-dir ptftests  --platform remote  -t \"testbed_type='t0';router_mac='3c:2c:30:6d:80:00';vlan_info='/root/vlan_info.txt'\"  --relax --debug info --log-file /tmp/dir_bcast.BcastTest.2019-07-26-16:24:26.log 2>&1", "_uses_shell": true, "chdir": "/root", "creates": null, "executable": null, "removes": null, "warn": true}, "module_name": "command"}, "rc": 1, "start": "2019-07-26 16:24:26.267564", "stderr": "", "stdout": "WARNING: No route found for IPv6 destination :: (no default route?)\ndir_bcast_test.BcastTest ... FAIL\n\n======================================================================\nFAIL: dir_bcast_test.BcastTest\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"ptftests/dir_bcast_test.py\", line 177, in runTest\n    self.check_all_dir_bcast()\n  File \"ptftests/dir_bcast_test.py\", line 86, in check_all_dir_bcast\n    self.check_ip_dir_bcast(bcast_ip, dst_port_list)\n  File \"ptftests/dir_bcast_test.py\", line 123, in check_ip_dir_bcast\n    assert (pkt_count == len(dst_port_list))\nAssertionError\n\n----------------------------------------------------------------------\nRan 1 test in 1.926s\n\nFAILED (failures=1)", "stdout_lines": ["WARNING: No route found for IPv6 destination :: (no default route?)", "dir_bcast_test.BcastTest ... FAIL", "", "======================================================================", "FAIL: dir_bcast_test.BcastTest", "----------------------------------------------------------------------", "Traceback (most recent call last):", "  File \"ptftests/dir_bcast_test.py\", line 177, in runTest", "    self.check_all_dir_bcast()", "  File \"ptftests/dir_bcast_test.py\", line 86, in check_all_dir_bcast", "    self.check_ip_dir_bcast(bcast_ip, dst_port_list)", "  File \"ptftests/dir_bcast_test.py\", line 123, in check_ip_dir_bcast", "    assert (pkt_count == len(dst_port_list))", "AssertionError", "", "----------------------------------------------------------------------", "Ran 1 test in 1.926s", "", "FAILED (failures=1)"], "warnings": []}

TASK [test : debug] ************************************************************
task path: /var/johnar/sonic-mgmt/ansible/roles/test/tasks/ptf_runner.yml:52
Friday 26 July 2019  16:24:29 +0000 (0:00:03.221)       0:00:21.421 *********** 
ok: [sonic-s5232-64] => {
    "out.stdout_lines": [
        "WARNING: No route found for IPv6 destination :: (no default route?)", 
        "dir_bcast_test.BcastTest ... FAIL", 
        "", 
        "======================================================================", 
        "FAIL: dir_bcast_test.BcastTest", 
        "----------------------------------------------------------------------", 
        "Traceback (most recent call last):", 
        "  File \"ptftests/dir_bcast_test.py\", line 177, in runTest", 
        "    self.check_all_dir_bcast()", 
        "  File \"ptftests/dir_bcast_test.py\", line 86, in check_all_dir_bcast", 
        "    self.check_ip_dir_bcast(bcast_ip, dst_port_list)", 
        "  File \"ptftests/dir_bcast_test.py\", line 123, in check_ip_dir_bcast", 
        "    assert (pkt_count == len(dst_port_list))", 
        "AssertionError", 
        "", 
        "----------------------------------------------------------------------", 
        "Ran 1 test in 1.926s", 
        "", 
        "FAILED (failures=1)"
    ]
}

DUT logs:

DUT's counters showing TX drops after running the script

root@sonic-s5232-64:~# show interfaces counters
      IFACE    STATE    RX_OK    RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK    TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
-----------  -------  -------  --------  ---------  --------  --------  --------  -------  --------  ---------  --------  --------  --------
  Ethernet0        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
  Ethernet4        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
  Ethernet8        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet12        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet16        U       72  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet20        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet24        U       72  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet28        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet32        U       72  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet36        U       72  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet40        U       72  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet44        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet48        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet52        U       72  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet56        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet60        U       72  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet64        U       72  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet68        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet72        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet76        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet80        U       72  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet84        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0         1         0
 Ethernet88        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet92        U       72  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
 Ethernet96        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0         1         0
Ethernet100        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet104        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet108        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet112        U      455  0.45 B/s      0.00%         0       246         0      381  0.40 B/s      0.00%         0         0         0
Ethernet116        U      382  0.52 B/s      0.00%         0       246         0      343  0.52 B/s      0.00%         0         0         0
Ethernet120        U      383  0.52 B/s      0.00%         0       247         0      344  0.52 B/s      0.00%         0         0         0
Ethernet124        U      327  0.31 B/s      0.00%         0       247         0      254  0.25 B/s      0.00%         0         0         0
Ethernet128        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet129        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
root@sonic-s5232-64:~# 

After sending 400 packets through the script , following are the values of the DUT's counters

root@sonic-s5232-64:~# show interfaces counters
      IFACE    STATE    RX_OK    RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK    TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
-----------  -------  -------  --------  ---------  --------  --------  --------  -------  --------  ---------  --------  --------  --------
  Ethernet0        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
  Ethernet4        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
  Ethernet8        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet12        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet16        U       72  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet20        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet24        U       72  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet28        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet32        U       72  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet36        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet40        U      472  0.47 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet44        U       72  0.07 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet48        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet52        U       72  0.07 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet56        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet60        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet64        U       72  0.07 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet68        U       72  0.07 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet72        U       72  0.07 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet76        U       71  0.06 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet80        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet84        U       71  0.06 B/s      0.00%         0        67         0        6  0.01 B/s      0.00%         0       400         0
 Ethernet88        U       72  0.07 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet92        U       72  0.07 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
 Ethernet96        U       72  0.07 B/s      0.00%         0        67         0        7  0.01 B/s      0.00%         0       400         0
Ethernet100        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet104        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet108        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet112        U      453  0.45 B/s      0.00%         0       244         0      380  0.40 B/s      0.00%         0         0         0
Ethernet116        U      387  0.53 B/s      0.00%         0       244         0      362  0.56 B/s      0.00%         0         0         0
Ethernet120        U      390  0.54 B/s      0.00%         0       247         0      365  0.56 B/s      0.00%         0         0         0
Ethernet124        U      326  0.31 B/s      0.00%         0       247         0      255  0.26 B/s      0.00%         0         0         0
Ethernet128        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet129        X        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
root@sonic-s5232-64:~#

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions