[test_crm] Fix test_acl_counter and test_crm_nexthop_group in test_crm by bingwang-ms · Pull Request #2272 · sonic-net/sonic-mgmt

bingwang-ms · 2020-09-25T12:15:05Z

Description of PR

Summary:
Fixes # (issue)
This PR is to fix test_acl_counter in test_crm script.

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Approach

What is the motivation for this PR?

This PR is to fix test_acl_counter and test_crm_nexthop_group in test_crm script.
Because the acl counter is relatively large in some DUT, and even 1% of the acl counter will be larger than the avaliable acl entry.
We can check that with crm show resources acl table
On Arista 7260

Table ID         Resource Name      Used Count    Available Count
---------------  ---------------  ------------  -----------------
0x7000000001561  acl_entry                   0                512
0x7000000001561  acl_counter                 0              54272

On Celestica-DX010

Table ID         Resource Name      Used Count    Available Count
---------------  ---------------  ------------  -----------------
0x70000000005ff  acl_entry                   2                510
0x70000000005ff  acl_counter                 2              29690

We can tell that the avaliable acl_counter on Arista-7260 is relatively large, and even 1% of that figure will be larger that the total available acl_entry. Some exception will happen if we attempt to insert more acl rules than the available count.
This PR adds a limit to the acl entry, and if the acl_counter is too large, the check for percentage will be skipped.
Similar issue exists in test_crm_nexthop_group. On Celestica-DX010, the total available nexthop group member is much larger than nexthop group object

~$ crm show  resources  nexthop group member 

Resource Name           Used Count    Available Count
--------------------  ------------  -----------------
nexthop_group_member             8              16376

~$ crm show  resources  nexthop group object 

Resource Name      Used Count    Available Count
---------------  ------------  -----------------
nexthop_group               2                126

If we attempt to insert 1% of 16736, that is 167 nexthop members, SAI will return errors and cause exception in orchagent.

How did you do it?

Adds a limit to the acl entry and nexthop group member/object.

How did you verify/test it?

Verified on both Arista 7260 and Celestica-DX010.
Result of test_acl_counter on Arista 7260

py.test --inventory ../ansible/str,../ansible/veos --host-pattern str-7260cx3-acs-2 --module-path ../ansible --testbed vms7-t0-7260-2 --testbed_file ../ansible/testbed.csv --junit-xml=tr.xml --log-cli-level warn --collect_techsupport=False --topology=t0,any,util crm/test_crm.py::test_acl_entry crm/test_crm.py::test_acl_counter
========================================================================================= test session starts =========================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
metadata: {'Python': '2.7.12', 'Platform': 'Linux-4.10.0-42-generic-x86_64-with-Ubuntu-16.04-xenial', 'Packages': {'py': '1.9.0', 'pytest': '4.6.5', 'pluggy': '0.13.1'}, 'Plugins': {u'repeat': u'0.8.0', u'html': u'1.22.1', u'xdist': u'1.28.0', u'ansible': u'2.2.2', u'forked': u'1.3.0', u'metadata': u'1.10.0'}}
ansible: 2.8.12
rootdir: /data/Networking-acs-sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, forked-1.3.0, xdist-1.28.0, html-1.22.1, repeat-0.8.0, metadata-1.10.0
collected 2 items                                                                                                                                                                                     

crm/test_crm.py::test_acl_entry PASSED                                                                                                                                                          [ 50%]
crm/test_crm.py::test_acl_counter 
-------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------
11:57:52 WARNING test_crm.py:verify_thresholds:219: CRM used entries is < 1 or >= 100 percent
11:57:52 WARNING test_crm.py:verify_thresholds:219: CRM used entries is < 1 or >= 100 percent
PASSED                                                                                                                                                                                          [100%]

------------------------------------------------------------------ generated xml file: /data/Networking-acs-sonic-mgmt/tests/tr.xml -------------------------------------------------------------------
===================================================================================== 2 passed in 293.04 seconds ======================================================================================

Result of test_acl_counter on Celestica-DX010:

py.test --inventory ../ansible/str,../ansible/veos --host-pattern str-dx010-4 --module-path ../ansible --testbed vms7-t0-dx010-4 --testbed_file ../ansible/testbed.csv --junit-xml=tr.xml --log-cli-level warn --collect_techsupport=False --topology=t0,any,util crm/test_crm.py::test_acl_entry crm/test_crm.py::test_acl_counter
========================================================================================= test session starts =========================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
metadata: {'Python': '2.7.12', 'Platform': 'Linux-4.10.0-42-generic-x86_64-with-Ubuntu-16.04-xenial', 'Packages': {'py': '1.9.0', 'pytest': '4.6.5', 'pluggy': '0.13.1'}, 'Plugins': {u'repeat': u'0.8.0', u'html': u'1.22.1', u'xdist': u'1.28.0', u'ansible': u'2.2.2', u'forked': u'1.3.0', u'metadata': u'1.10.0'}}
ansible: 2.8.12
rootdir: /data/Networking-acs-sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, forked-1.3.0, xdist-1.28.0, html-1.22.1, repeat-0.8.0, metadata-1.10.0
collected 2 items                                                                                                                                                                                     

crm/test_crm.py::test_acl_entry PASSED                                                                                                                                                          [ 50%]
crm/test_crm.py::test_acl_counter ^@PASSED                                                                                                                                                        [100%]

------------------------------------------------------------------ generated xml file: /data/Networking-acs-sonic-mgmt/tests/tr.xml -------------------------------------------------------------------
===================================================================================== 2 passed in 381.97 seconds ======================================================================================

Result of test_crm_nexthop_group on Celestica-DX010:

py.test --inventory ../ansible/str,../ansible/veos --host-pattern str-dx010-4 --module-path ../ansible --testbed vms7-t0-dx010-4 --testbed_file ../ansible/testbed.csv --junit-xml=tr.xml --log-cli-level warn --collect_techsupport=False --topology=t0,any,util crm/test_crm.py::test_crm_nexthop_group
========================================================================================= test session starts =========================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
metadata: {'Python': '2.7.12', 'Platform': 'Linux-4.10.0-42-generic-x86_64-with-Ubuntu-16.04-xenial', 'Packages': {'py': '1.9.0', 'pytest': '4.6.5', 'pluggy': '0.13.1'}, 'Plugins': {u'repeat': u'0.8.0', u'html': u'1.22.1', u'xdist': u'1.28.0', u'ansible': u'2.2.2', u'forked': u'1.3.0', u'metadata': u'1.10.0'}}
ansible: 2.8.12
rootdir: /data/Networking-acs-sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, forked-1.3.0, xdist-1.28.0, html-1.22.1, repeat-0.8.0, metadata-1.10.0
collected 2 items                                                                                                                                                                                     

crm/test_crm.py::test_crm_nexthop_group[False-2.2.2.0/24] PASSED                                                                                                                                [ 50%]
crm/test_crm.py::test_crm_nexthop_group[True-2.2.2.0/24] PASSED                                                                                                                                 [100%]^@

------------------------------------------------------------------ generated xml file: /data/Networking-acs-sonic-mgmt/tests/tr.xml -------------------------------------------------------------------
===================================================================================== 2 passed in 318.60 seconds 
======================================================================================

Any platform specific information?

No.

Supported testbed topology if it's a new test case?

No.

Documentation

No.

Because the acl counter is relatively large in some DUT, and even 1% of the acl counter will be larger than the avaliable acl entry. This PR adds a limit to the acl entry that are applied to DUT.

lgtm-com · 2020-09-25T12:47:08Z

This pull request fixes 1 alert when merging 8ac823c into 30180c9 - view on LGTM.com

fixed alerts:

1 for Unused import

yxieca · 2020-09-25T16:12:33Z

tests/crm/test_crm.py

-            if used_percent < 1:
-                logger.warning("CRM used entries is < 1 percent")
+            if used_percent < 1 or used_percent >= 100:
+                logger.warning("CRM used entries is < 1 or >= 100 percent")


Can you add used_percent in the debug output?

Also, the wording is a bit confusing, is this intended use percentage or used percentage?

If this is intended use percentage, can we change the test to test a smaller number instead of skipping?

Thanks. I've updated the log and added the used_percentage.
The test script is to configure the crm threshold type to percentage, and then insert a number of acl rules to trigger acm WARNING log. Therefore, the least used percentage is 1% to do the test. If we are limited to the available acl entry number, and can't add enough acl rules, the used percentage of acl group counter will be below 1%, then the verification has to be skipped.

lgtm-com · 2020-09-27T09:02:08Z

This pull request fixes 1 alert when merging 7e0a246 into d190bb2 - view on LGTM.com

fixed alerts:

1 for Unused import

bingwang-ms · 2020-09-27T11:15:41Z

retest vsimage please

The test case will attempt to insert more nexthop members than the total available number on some devices. As a result, SAI will return error. This commit add a check and get the min value between avaiable nexthop_group_member and nexthop_group_object before inserting new nexthops.

bingwang-ms · 2020-09-28T06:56:25Z

An update is post to fix similar issue in test_crm_nexthop_group

lgtm-com · 2020-09-28T07:05:53Z

This pull request fixes 1 alert when merging d3b2770 into d190bb2 - view on LGTM.com

fixed alerts:

1 for Unused import

bingwang-ms · 2020-09-29T01:21:39Z

@yvolynets-mlnx Could you help to review? Thanks!

yvolynets-mlnx

Looks good

bingwang-ms · 2020-09-30T08:19:36Z

Looks good

Thanks. I've merged.

… advance submodule head (sonic-net#11518) sairedis: * 38c0bb1 2022-07-21 | [sairedis] Fix reopen recoding file (sonic-net#1087) (HEAD -> 202205, github/202205) [Kamil Cudnik] platform-daemon: * 17587b6 2022-07-22 | [ycabled] add secure channel support for grpc dualtor active-active connectivity (sonic-net#275) (HEAD -> 202205, github/202205) [vdahiya12] linkmgrd: * c911ec7 2022-07-21 | Avoid unnecessary error logs from `handleGetServerMacAddressNotification` (sonic-net#96) (HEAD -> 202205) [Jing Zhang] * bbae81d 2022-07-18 | Add support for reconciliation after warm restart (sonic-net#76) [Jing Zhang] utilities: * bcc1206 2022-07-20 | Change db_migrator major version on master branch from version 2 to 3 (sonic-net#2272) (HEAD -> 202205) [Vaibhav Hemant Dixit] * ad40697 2022-07-21 | Fix test for pfcwd_sw_enable in db_migrator_test (sonic-net#2253) [bingwang-ms] * 886f612 2022-07-22 | Revert "show commands for SYSTEM READY (sonic-net#1851) (sonic-net#2261)" (sonic-net#2274) (github/202205) [Ying Xie] * a6404b7 2022-07-17 | show commands for SYSTEM READY (sonic-net#1851) (sonic-net#2261) [Senthil Kumar Guruswamy] swss-common: * 509b265 2022-07-06 | Add device global table definition (sonic-net#645) (HEAD -> 202205) [tjchadaga] Signed-off-by: Ying Xie <ying.xie@microsoft.com>

Fix test_acl_counter

8ac823c

Because the acl counter is relatively large in some DUT, and even 1% of the acl counter will be larger than the avaliable acl entry. This PR adds a limit to the acl entry that are applied to DUT.

yxieca reviewed Sep 25, 2020

View reviewed changes

Update log for skip check

7e0a246

bingwang-ms changed the title ~~[test_crm] Fix test_acl_counter~~ [test_crm] Fix test_acl_counter and test_crm_nexthop_group in test_crm Sep 28, 2020

bingwang-ms requested a review from a team September 29, 2020 01:20

yxieca approved these changes Sep 30, 2020

View reviewed changes

bingwang-ms merged commit 2fe8f73 into sonic-net:master Sep 30, 2020

yvolynets-mlnx reviewed Sep 30, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[test_crm] Fix test_acl_counter and test_crm_nexthop_group in test_crm#2272

[test_crm] Fix test_acl_counter and test_crm_nexthop_group in test_crm#2272
bingwang-ms merged 3 commits intosonic-net:masterfrom
bingwang-ms:fix_crm_acl

bingwang-ms commented Sep 25, 2020 •

edited

Loading

Uh oh!

lgtm-com bot commented Sep 25, 2020

Uh oh!

yxieca Sep 25, 2020

Uh oh!

bingwang-ms Sep 27, 2020 •

edited

Loading

Uh oh!

lgtm-com bot commented Sep 27, 2020

Uh oh!

bingwang-ms commented Sep 27, 2020

Uh oh!

bingwang-ms commented Sep 28, 2020

Uh oh!

lgtm-com bot commented Sep 28, 2020

Uh oh!

bingwang-ms commented Sep 29, 2020

Uh oh!

yvolynets-mlnx left a comment

Uh oh!

bingwang-ms commented Sep 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bingwang-ms commented Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

lgtm-com bot commented Sep 25, 2020

Uh oh!

yxieca Sep 25, 2020

Choose a reason for hiding this comment

Uh oh!

bingwang-ms Sep 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lgtm-com bot commented Sep 27, 2020

Uh oh!

bingwang-ms commented Sep 27, 2020

Uh oh!

bingwang-ms commented Sep 28, 2020

Uh oh!

lgtm-com bot commented Sep 28, 2020

Uh oh!

bingwang-ms commented Sep 29, 2020

Uh oh!

yvolynets-mlnx left a comment

Choose a reason for hiding this comment

Uh oh!

bingwang-ms commented Sep 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bingwang-ms commented Sep 25, 2020 •

edited

Loading

bingwang-ms Sep 27, 2020 •

edited

Loading