Skip to content

Commit b1beed0

Browse files
vmittal-msftsanmalho-git
vmittal-msft
and
sanmalho-git
authored
Enhance qos tests to support single-asic, multi-asic, and multi-dut testing (#8059)
* Enhance qos tests to support single-asic, multi-asic, and multi-dut testing The existing QoS (test_qos_sai.py) is written to accomodate a single asic on a single Dut. But, we require the same tests to be executed against a T2 chassis (with single/multi-asic linecards) and multi-asic pizza boxes. All the test cases create a list of src and dst ports. For the different modes, here is the distribution of the src and dst ports: - single_asic: The src and dst ports are on the same asic on the same linecard. - single_dut_multi_asic: On a multi-asic DUT/linecard, the src port is on an asic, while the dst ports are on another asic on the same DUT/linecard - multi_dut: The src port is on an asic on one of the DUT/linecards, and the dst port is on another asic on another DUT/linecard. This is currently only required for T2 topology Approach to accomplish this is the following: - All the tests have to parameterized for the 3 modes defined above. - This is done using the 'select_src_and_dst_dut_and_asic' fixture that is parameterized for 'single_asic', 'single_dut_multi_asic', 'multi_dut' Based on the mode, it sets the src_dut_index, dst_dut_index, src_asic_index and dst_asic_index - Added fixture 'get_src_dst_asic_and_duts' that returns dictionary of the src_dut_index, dst_dut_index, src_asic_index, and dst_asic_index, and the src_dut and dst_dut (instances of MultiAsicSonicHost), src_asic and dst_asic (instances of Asic), and also a list of all DUTs and all Asics - dutConfig is modified such that testPortIds and testPortIps are collecting from all the duts and asics involved and stored in a dictionary with key being the dutIndex and value being a dictionary per asic index. - __buildTestPorts then sets the src and dst ports based on the src_dut_index, dst_dut_index, src_asic_index and dst_asic_index - All the other fixtures and tests, we use 'get_src_dst_asci_and_duts' fixture instead of enum_rand_one_frontend_hostname and enum_frontend_index. - The code instead the fixtures and tests is modified to the actions on the correct src/dst dut or asic. For example: - swap_syncd fixture would swap syncd docker on all DUT's (both src and dst) instead of just one DUT as before. - stopServices - do it all_duts (src and dst duts) - Similarly, changes to saitests involved dealing with multiple DUTs (and thus multiple sai clients) and modifying other data structure like 'interface_to_front_mapping' in sai_base_test.py and port_list, sai_port_list, front_port_list in switch.py to deal with multiple duts (modified to be dictionary with key being 'src' and 'dst') - tests in sai_qos_tests.py pass src_dut_index, src_asic_index, dst_dut_index and dst_asic_index in the testParams. - The saitests classes then use this to do the actions on the right client and ports. Assumptions: - For multi-dut, we are assuming that hwsku for all the cards are same. * Fixes to QoS tests for mellanox and cisco-8000 platforms * Fix json.loads exception in dut_qos_maps if corresponding data is not present in the output of sonic-cfggen * Fix to allow tests to run one a single DUT in the testbed that has multiple DUTs defined * Fixed missing 'target' parameter in sai_thrift_read_queue_occupancy calls for cisco-8000 * Fixes for T0 topology tests * Fixes for Mellanox platforms --------- Co-authored-by: sanmalho <sandeep.malhotra@nokia.com>
1 parent dc66487 commit b1beed0

13 files changed

Lines changed: 1377 additions & 876 deletions

File tree

‎tests/common/devices/sonic.py‎

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1873,22 +1873,6 @@ def is_service_running(self, service_name, docker_name):
18731873

18741874
return "RUNNING" in service_status
18751875

1876-
def remove_ssh_tunnel_sai_rpc(self):
1877-
"""
1878-
Removes any ssh tunnels if present created for syncd RPC communication
1879-
1880-
Returns:
1881-
None
1882-
"""
1883-
try:
1884-
pid_list = self.shell(
1885-
'pgrep -f "ssh -o StrictHostKeyChecking=no -fN -L \*:9092"'
1886-
)["stdout_lines"]
1887-
except RunAnsibleModuleFail:
1888-
return
1889-
for pid in pid_list:
1890-
self.shell("kill {}".format(pid), module_ignore_errors=True)
1891-
18921876
def get_up_ip_ports(self):
18931877
"""
18941878
Get a list for all up ip interfaces

‎tests/common/devices/sonic_asic.py‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ class SonicAsic(object):
1818

1919
_MULTI_ASIC_SERVICE_NAME = "{}@{}" # service name, asic_id
2020
_MULTI_ASIC_DOCKER_NAME = "{}{}" # docker name, asic_id
21+
_RPC_PORT_FOR_SSH_TUNNEL = 9092
2122

2223
def __init__(self, sonichost, asic_index):
2324
""" Initializing a ASIC on a SONiC host.
@@ -325,6 +326,9 @@ def bgp_drop_rule(self, ip_version, state="present"):
325326

326327
logger.debug(output)
327328

329+
def get_rpc_port_ssh_tunnel(self):
330+
return self._RPC_PORT_FOR_SSH_TUNNEL + self.asic_index
331+
328332
def remove_ssh_tunnel_sai_rpc(self):
329333
"""
330334
Removes any ssh tunnels if present created for syncd RPC communication
@@ -334,7 +338,15 @@ def remove_ssh_tunnel_sai_rpc(self):
334338
"""
335339
if not self.sonichost.is_multi_asic:
336340
return
337-
return self.sonichost.remove_ssh_tunnel_sai_rpc()
341+
342+
try:
343+
pid_list = self.sonichost.shell(
344+
r'pgrep -f "ssh -o StrictHostKeyChecking=no -fN -L \*:{}"'.format(self.get_rpc_port_ssh_tunnel())
345+
)["stdout_lines"]
346+
except RunAnsibleModuleFail:
347+
return
348+
for pid in pid_list:
349+
self.shell("kill {}".format(pid), module_ignore_errors=True)
338350

339351
def create_ssh_tunnel_sai_rpc(self):
340352
"""
@@ -363,9 +375,9 @@ def create_ssh_tunnel_sai_rpc(self):
363375

364376
self.sonichost.shell(
365377
("ssh -o StrictHostKeyChecking=no -fN"
366-
" -L *:9092:{}:9092 localhost"
367-
).format(ns_docker_if_ipv4)
368-
)
378+
" -L *:{}:{}:{} localhost").format(self.get_rpc_port_ssh_tunnel(), ns_docker_if_ipv4,
379+
self._RPC_PORT_FOR_SSH_TUNNEL ))
380+
369381

370382
def command(self, cmdstr):
371383
"""

‎tests/common/fixtures/duthost_utils.py‎

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -434,8 +434,8 @@ def utils_create_test_vlans(duthost, cfg_facts, vlan_ports_list, vlan_intfs_dict
434434
duthost.shell_cmds(cmds=cmds)
435435

436436

437-
@pytest.fixture(scope='module')
438-
def dut_qos_maps(rand_selected_front_end_dut):
437+
@pytest.fixture(scope='class')
438+
def dut_qos_maps(get_src_dst_asic_and_duts):
439439
"""
440440
A module level fixture to get QoS map from DUT host.
441441
Return a dict
@@ -452,32 +452,32 @@ def dut_qos_maps(rand_selected_front_end_dut):
452452
or an empty dict if failed to parse the output
453453
"""
454454
maps = {}
455+
dut = get_src_dst_asic_and_duts['src_dut']
455456
try:
456-
if rand_selected_front_end_dut.is_multi_asic:
457+
if dut.is_multi_asic:
457458
sonic_cfggen_cmd = "sonic-cfggen -n asic0 -d --var-json"
458459
else:
459460
sonic_cfggen_cmd = "sonic-cfggen -d --var-json"
460461

461462
# port_qos_map
462-
port_qos_map_data = rand_selected_front_end_dut.shell("{} 'PORT_QOS_MAP'".format(sonic_cfggen_cmd))['stdout']
463-
maps['port_qos_map'] = json.loads(port_qos_map_data) if port_qos_map_data else None
463+
port_qos_map = dut.shell("{} 'PORT_QOS_MAP'".format(sonic_cfggen_cmd))['stdout']
464+
maps['port_qos_map'] = json.loads(port_qos_map) if port_qos_map else None
465+
464466
# dscp_to_tc_map
465-
dscp_to_tc_map_data = rand_selected_front_end_dut.shell(
466-
"{} 'DSCP_TO_TC_MAP'".format(sonic_cfggen_cmd))['stdout']
467-
maps['dscp_to_tc_map'] = json.loads(dscp_to_tc_map_data) if dscp_to_tc_map_data else None
467+
dscp_to_tc_map = dut.shell("{} 'DSCP_TO_TC_MAP'".format(sonic_cfggen_cmd))['stdout']
468+
maps['dscp_to_tc_map'] = json.loads(dscp_to_tc_map) if dscp_to_tc_map else None
469+
468470
# tc_to_queue_map
469-
tc_to_queue_map_data = rand_selected_front_end_dut.shell(
470-
"{} 'TC_TO_QUEUE_MAP'".format(sonic_cfggen_cmd))['stdout']
471-
maps['tc_to_queue_map'] = json.loads(tc_to_queue_map_data) if tc_to_queue_map_data else None
471+
tc_to_queue_map = dut.shell("{} 'TC_TO_QUEUE_MAP'".format(sonic_cfggen_cmd))['stdout']
472+
maps['tc_to_queue_map'] = json.loads(tc_to_queue_map) if tc_to_queue_map else None
473+
472474
# tc_to_priority_group_map
473-
tc_to_priority_group_map_data = rand_selected_front_end_dut.shell(
474-
"{} 'TC_TO_PRIORITY_GROUP_MAP'".format(sonic_cfggen_cmd))['stdout']
475-
maps['tc_to_priority_group_map'] = json.loads(
476-
tc_to_priority_group_map_data) if tc_to_priority_group_map_data else None
475+
tc_to_priority_group_map = dut.shell("{} 'TC_TO_PRIORITY_GROUP_MAP'".format(sonic_cfggen_cmd))['stdout']
476+
maps['tc_to_priority_group_map'] = json.loads(tc_to_priority_group_map) if tc_to_priority_group_map else None
477+
477478
# tc_to_dscp_map
478-
tc_to_dscp_map_data = rand_selected_front_end_dut.shell(
479-
"{} 'TC_TO_DSCP_MAP'".format(sonic_cfggen_cmd))['stdout']
480-
maps['tc_to_dscp_map'] = json.loads(tc_to_dscp_map_data) if tc_to_dscp_map_data else None
479+
tc_to_dscp_map = dut.shell("{} 'TC_TO_DSCP_MAP'".format(sonic_cfggen_cmd))['stdout']
480+
maps['tc_to_dscp_map'] = json.loads(tc_to_dscp_map) if tc_to_dscp_map else None
481481
except Exception as e:
482482
logger.error("Got exception: " + repr(e))
483483
return maps

‎tests/common/fixtures/ptfhost_utils.py‎

Lines changed: 32 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -508,24 +508,34 @@ def ptf_test_port_map_active_active(ptfhost, tbinfo, duthosts, mux_server_url, d
508508
# Loop ptf_map of each DUT. Each ptf_map maps from ptf port index to dut port index
509509
disabled_ptf_ports = disabled_ptf_ports.union(set(ptf_map.keys()))
510510

511-
router_macs = [duthost.facts['router_mac'] for duthost in duthosts]
511+
router_macs = []
512+
all_dut_names = [duthost.hostname for duthost in duthosts]
513+
for a_dut_name in tbinfo['duts']:
514+
if a_dut_name in all_dut_names:
515+
duthost = duthosts[a_dut_name]
516+
router_macs.append(duthost.facts['router_mac'])
517+
else:
518+
router_macs.append(None)
512519

513520
logger.info('active_dut_map={}'.format(active_dut_map))
514521
logger.info('disabled_ptf_ports={}'.format(disabled_ptf_ports))
515522
logger.info('router_macs={}'.format(router_macs))
516523

517-
asic_idx = 0
518524
ports_map = {}
519525
for ptf_port, dut_intf_map in tbinfo['topo']['ptf_dut_intf_map'].items():
520526
if str(ptf_port) in disabled_ptf_ports:
521527
# Skip PTF ports that are connected to disabled VLAN interfaces
522528
continue
529+
asic_idx = 0
530+
dut_port = None
523531

524532
if len(dut_intf_map.keys()) == 2:
525533
# PTF port is mapped to two DUTs -> dualtor topology and the PTF port is a vlan port
526534
# Packet sent from this ptf port will only be accepted by the active side DUT
527-
# DualToR DUTs use same special Vlan interface MAC address
535+
# DualToR DUTs use same special Vlan interface MAC addres
528536
target_dut_indexes = list(map(int, active_dut_map[ptf_port]))
537+
target_dut_port = int(list(dut_intf_map.values())[0])
538+
target_hostname = duthosts[target_dut_indexes[0]].hostname
529539
ports_map[ptf_port] = {
530540
'target_dut': target_dut_indexes,
531541
'target_dest_mac': tbinfo['topo']['properties']['topology']['DUT']['vlan_configs']['one_vlan_a']
@@ -535,30 +545,39 @@ def ptf_test_port_map_active_active(ptfhost, tbinfo, duthosts, mux_server_url, d
535545
}
536546
else:
537547
# PTF port is mapped to single DUT
548+
dut_index_for_pft_port = int(dut_intf_map.keys()[0])
549+
if router_macs[dut_index_for_pft_port] is None:
550+
continue
538551
target_dut_index = int(list(dut_intf_map.keys())[0])
539552
target_dut_port = int(list(dut_intf_map.values())[0])
540553
router_mac = router_macs[target_dut_index]
541-
dut_port = None
542-
if len(duts_minigraph_facts[duthosts[target_dut_index].hostname]) > 1:
543-
for list_idx, mg_facts_tuple in enumerate(duts_minigraph_facts[duthosts[target_dut_index].hostname]):
554+
target_hostname = tbinfo['duts'][target_dut_index]
555+
556+
if len(duts_minigraph_facts[target_hostname]) > 1:
557+
# Dealing with multi-asic target dut
558+
for list_idx, mg_facts_tuple in enumerate(duts_minigraph_facts[target_hostname]):
544559
idx, mg_facts = mg_facts_tuple
545560
if target_dut_port in list(mg_facts['minigraph_port_indices'].values()):
546-
router_mac = duts_running_config_facts[duthosts[target_dut_index].hostname][list_idx][1]\
561+
router_mac = duts_running_config_facts[target_hostname][list_idx][1]\
547562
['DEVICE_METADATA']['localhost']['mac'].lower()
548563
asic_idx = idx
549-
for a_dut_port, a_dut_port_index in mg_facts['minigraph_port_indices'].items():
550-
if a_dut_port_index == target_dut_port and "Ethernet-Rec" not in a_dut_port and \
551-
"Ethernet-IB" not in a_dut_port and "Ethernet-BP" not in a_dut_port:
552-
dut_port = a_dut_port
553-
break
564+
554565
ports_map[ptf_port] = {
555566
'target_dut': [target_dut_index],
556567
'target_dest_mac': router_mac,
557568
'target_src_mac': [router_mac],
558-
'dut_port': dut_port,
559569
'asic_idx': asic_idx
560570
}
561571

572+
_, asic_mg_facts = duts_minigraph_facts[target_hostname][asic_idx]
573+
for a_dut_port, a_dut_port_index in asic_mg_facts['minigraph_port_indices'].items():
574+
if a_dut_port_index == target_dut_port and "Ethernet-Rec" not in a_dut_port and \
575+
"Ethernet-IB" not in a_dut_port and "Ethernet-BP" not in a_dut_port:
576+
dut_port = a_dut_port
577+
break
578+
579+
ports_map[ptf_port]['dut_port'] = dut_port
580+
562581
logger.debug('ptf_test_port_map={}'.format(json.dumps(ports_map, indent=2)))
563582

564583
ptfhost.copy(content=json.dumps(ports_map), dest=PTF_TEST_PORT_MAP)

‎tests/conftest.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
from tests.common.fixtures.duthost_utils import backup_and_restore_config_db_session
2929
from tests.common.fixtures.ptfhost_utils import ptf_portmap_file # lgtm[py/unused-import]
3030
from tests.common.fixtures.ptfhost_utils import run_icmp_responder_session # lgtm[py/unused-import]
31+
from tests.common.fixtures.ptfhost_utils import ptf_test_port_map_active_active # noqa F401
3132

3233
from tests.common.helpers.constants import (
3334
ASIC_PARAM_TYPE_ALL, ASIC_PARAM_TYPE_FRONTEND, DEFAULT_ASIC_ID, ASICS_PRESENT
@@ -1599,6 +1600,59 @@ def duts_running_config_facts(duthosts):
15991600
return cfg_facts
16001601

16011602
@pytest.fixture(scope='class')
1603+
def dut_test_params_qos(duthosts, tbinfo, ptfhost, get_src_dst_asic_and_duts, lower_tor_host, creds,
1604+
mux_server_url, mux_status_from_nic_simulator, duts_running_config_facts, duts_minigraph_facts):
1605+
if 'dualtor' in tbinfo['topo']['name']:
1606+
all_duts = [lower_tor_host]
1607+
else:
1608+
all_duts = get_src_dst_asic_and_duts['all_duts']
1609+
1610+
src_asic = get_src_dst_asic_and_duts['src_asic']
1611+
dst_asic = get_src_dst_asic_and_duts['dst_asic']
1612+
1613+
src_dut = get_src_dst_asic_and_duts['src_dut']
1614+
src_dut_ip = src_dut.host.options['inventory_manager'].get_host(src_dut.hostname).vars['ansible_host']
1615+
src_server = "{}:{}".format(src_dut_ip, src_asic.get_rpc_port_ssh_tunnel())
1616+
1617+
duthost = all_duts[0]
1618+
mgFacts = duthost.get_extended_minigraph_facts(tbinfo)
1619+
topo = tbinfo["topo"]["name"]
1620+
1621+
rtn_dict = {
1622+
"topo": topo,
1623+
"hwsku": mgFacts["minigraph_hwsku"],
1624+
"basicParams": {
1625+
"router_mac": duthost.facts["router_mac"],
1626+
"src_server" : src_server,
1627+
"port_map_file": ptf_test_port_map_active_active(
1628+
ptfhost, tbinfo, duthosts, mux_server_url,
1629+
duts_running_config_facts, duts_minigraph_facts,
1630+
mux_status_from_nic_simulator()),
1631+
"sonic_asic_type": duthost.facts['asic_type'],
1632+
"sonic_version": duthost.os_version,
1633+
"src_dut_index": get_src_dst_asic_and_duts['src_dut_index'],
1634+
"src_asic_index": get_src_dst_asic_and_duts['src_asic_index'],
1635+
"dst_dut_index": get_src_dst_asic_and_duts['dst_dut_index'],
1636+
"dst_asic_index": get_src_dst_asic_and_duts['dst_asic_index'],
1637+
"dut_username": creds['sonicadmin_user'],
1638+
"dut_password": creds['sonicadmin_password']
1639+
},
1640+
1641+
}
1642+
1643+
# Add dst server info if src and dst asic are different
1644+
if src_asic != dst_asic:
1645+
dst_dut = get_src_dst_asic_and_duts['dst_dut']
1646+
dst_dut_ip = dst_dut.host.options['inventory_manager'].get_host(dst_dut.hostname).vars['ansible_host']
1647+
rtn_dict["basicParams"]["dst_server"] = "{}:{}".format(dst_dut_ip, dst_asic.get_rpc_port_ssh_tunnel())
1648+
1649+
if 'platform_asic' in duthost.facts:
1650+
rtn_dict['basicParams']["platform_asic"] = duthost.facts['platform_asic']
1651+
1652+
yield rtn_dict
1653+
1654+
1655+
@ pytest.fixture(scope='class')
16021656
def dut_test_params(duthosts, enum_rand_one_per_hwsku_frontend_hostname, tbinfo,
16031657
ptf_portmap_file, lower_tor_host, creds): # noqa F811
16041658
"""

‎tests/qos/files/mellanox/qos_param_generator.py‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
import math
22

33
class QosParamMellanox(object):
4-
def __init__(self, qos_params, asic_type, speed_cable_len, dutConfig, ingressLosslessProfile, ingressLossyProfile, egressLosslessProfile, egressLossyProfile, sharedHeadroomPoolSize, dualTor):
4+
def __init__(self, qos_params, asic_type, speed_cable_len, dutConfig, ingressLosslessProfile,
5+
ingressLossyProfile, egressLosslessProfile, egressLossyProfile, sharedHeadroomPoolSize,
6+
dualTor, src_dut_index, src_asic_index, dst_asic_index, dst_dut_index):
57
self.asic_param_dic = {
68
'spc1': {
79
'cell_size': 96,
@@ -44,7 +46,10 @@ def __init__(self, qos_params, asic_type, speed_cable_len, dutConfig, ingressLos
4446
self.sharedHeadroomPoolSize = None
4547
self.dutConfig = dutConfig
4648
self.dualTor = dualTor
47-
49+
self.src_dut_index = src_dut_index
50+
self.src_asic_index = src_asic_index
51+
self.dst_dut_index = dst_dut_index
52+
self.dst_asic_index = dst_asic_index
4853
return
4954

5055
def run(self):
@@ -88,18 +93,19 @@ def collect_qos_configurations(self):
8893
pkts_num_trig_egr_drp = egress_lossy_size + 1
8994

9095
if self.sharedHeadroomPoolSize:
91-
testPortIds = self.dutConfig['testPortIds']
96+
src_testPortIds = self.dutConfig['testPortIds'][self.src_dut_index][self.src_asic_index]
97+
dst_testPortIds = self.dutConfig['testPortIds'][self.dst_dut_index][self.dst_asic_index]
9298
ingress_ports_num_shp = 8
9399
pkts_num_trig_pfc_shp = []
94100
ingress_ports_list_shp = []
95101
occupancy_per_port = ingress_lossless_size
96-
self.qos_parameters['dst_port_id'] = testPortIds[0]
102+
self.qos_parameters['dst_port_id'] = dst_testPortIds[0]
97103
pgs_per_port = 2 if not self.dualTor else 4
98104
for i in range(1, ingress_ports_num_shp):
99105
for j in range(pgs_per_port):
100106
pkts_num_trig_pfc_shp.append(occupancy_per_port + xon + hysteresis)
101107
occupancy_per_port /= 2
102-
ingress_ports_list_shp.append(testPortIds[i])
108+
ingress_ports_list_shp.append(src_testPortIds[i])
103109
self.qos_parameters['pkts_num_trig_pfc_shp'] = pkts_num_trig_pfc_shp
104110
self.qos_parameters['src_port_ids'] = ingress_ports_list_shp
105111
self.qos_parameters['pkts_num_hdrm_full'] = xoff - 2
@@ -209,4 +215,4 @@ def calculate_parameters(self):
209215
self.qos_params_mlnx['ecn_{}'.format(i+1)]['cell_size'] = self.cell_size
210216

211217
self.qos_params_mlnx['shared-headroom-pool'] = self.sharedHeadroomPoolSize
212-
self.qos_params_mlnx['pkts_num_private_headrooom'] = self.asic_param_dic[self.asic_type]['private_headroom']
218+
self.qos_params_mlnx['pkts_num_private_headrooom'] = self.asic_param_dic[self.asic_type]['private_headroom']

‎tests/qos/qos_helpers.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ def eos_to_linux_intf(eos_intf_name, hwsku=None):
3535
"""
3636
if hwsku == "MLNX-OS":
3737
linux_intf_name = eos_intf_name.replace("ernet 1/", "sl1p").replace("/", "sp")
38+
elif "Nokia" in hwsku:
39+
linux_intf_name = eos_intf_name
3840
else:
3941
linux_intf_name = eos_intf_name.replace('Ethernet', 'et').replace('/', '_')
4042
return linux_intf_name

0 commit comments

Comments
 (0)