[Snappi] Validate PFC frames should not be counted as RX_OK, TX_OK, RX_DRP, TX_DRP by developfast · Pull Request #9775 · sonic-net/sonic-mgmt

developfast · 2023-08-30T21:58:47Z

Description of PR

Summary: While receiving or sending out PFC frames, the PFC counter should be the only counter increasing on the device. The other counters are not expected to increase. The regular TX and RX counters are also polled to ensure they do not increment by the PFC count.
Fixes (issue) #6638

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012
202205

Approach

What is the motivation for this PR?

PFC validation and behavior on the DUT

How did you do it?

Send PFC pause frames from TGEN to DUT, check the RX_OK, RX_DRP and PFC rx counter. Except PFC rx counter, other counters should not increase.
From DUT, send out PFC pause frames to neighbor, check the TX_OK, TX_DRP and PFC tx counter. Except PFC tx counter, other counter should not increase.
Traffic counters will increment minimally for the small amount of regular test traffic sent - this is also verified by polling the DUT.

How did you verify/test it?

Tested on lab device

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2023-08-30T21:59:51Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/common/snappi_tests/traffic_generation.py Fixing tests/snappi_tests/pfc/files/helper.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/common/snappi_tests/traffic_generation.py:541:56: E261 at least two spaces before inline comment tests/common/snappi_tests/traffic_generation.py:552:56: E261 at least two spaces before inline comment tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py:80:1: E302 expected 2 blank lines, found 1 tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py:80:34: F811 redefinition of unused 'snappi_api' from line 8 ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-08-30T22:01:01Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/common/snappi_tests/traffic_generation.py Fixing tests/snappi_tests/pfc/files/helper.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/common/snappi_tests/traffic_generation.py:540:56: E261 at least two spaces before inline comment tests/common/snappi_tests/traffic_generation.py:551:56: E261 at least two spaces before inline comment tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py:80:1: E302 expected 2 blank lines, found 1 tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py:80:34: F811 redefinition of unused 'snappi_api' from line 8 ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-08-30T22:03:46Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/common/snappi_tests/traffic_generation.py Fixing tests/snappi_tests/pfc/files/helper.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/common/snappi_tests/traffic_generation.py:540:56: E261 at least two spaces before inline comment tests/common/snappi_tests/traffic_generation.py:551:56: E261 at least two spaces before inline comment tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py:91:121: E501 line too long (123 > 120 characters) tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py:128:5: E303 too many blank lines (2) ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-08-30T22:06:57Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/common/snappi_tests/traffic_generation.py Fixing tests/snappi_tests/pfc/files/helper.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py:91:121: E501 line too long (123 > 120 characters) tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py:128:5: E303 too many blank lines (2) flake8...............................................(no files to check)Skipped ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-08-30T22:08:12Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/common/snappi_tests/traffic_generation.py Fixing tests/snappi_tests/pfc/files/helper.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

tests/common/snappi_tests/common_helpers.py

…into tx_rx_check_test

mssonicbld · 2023-08-31T20:11:59Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/snappi_tests/pfc/files/helper.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

tests/common/snappi_tests/common_helpers.py

baiwei0427 · 2023-09-15T07:09:46Z

tests/common/snappi_tests/traffic_generation.py

 logger = logging.getLogger(__name__)

 SNAPPI_POLL_DELAY_SEC = 2
 CONTINUOUS_MODE = -5


What is continuous mode? What is the meaning of 5?

It's an arbitrary number that isn't used anywhere else - continuous mode means the traffic flows constantly without any fixed duration or fixed packets at some percentage rate

tests/common/snappi_tests/traffic_generation.py

baiwei0427 · 2023-09-15T07:30:00Z

tests/common/snappi_tests/traffic_generation.py

+    """
+    Verify correct frame count for tx frames on the DUT
+    (OK and DROPS) when the traffic is expected to be paused on the DUT.
+    DUT is polled after it stops receiving PFC pause frames from TGEN.


When you poll DUT, has DUT drained all the packets in the buffer?

tests/common/snappi_tests/traffic_generation.py

baiwei0427 · 2023-09-22T09:44:57Z

tests/common/snappi_tests/traffic_generation.py

+                              .format(prios[prio]))
+            else:
+                # PFC pause frames should not be transmitted when test traffic is not paused
+                pytest_assert(pfc_pause_tx_frames <= PFC_FRAME_COUNT_TOL,


have you ever seen pfc_pause_tx_frames >0 in this case?

No - this is just there as a back stopper. If you want me to lower the threshold to 2 or 3, that is also possible. It is currently set to 5.

Reset to 0

Can you please remove PFC_FRAME_COUNT_TOL?

developfast · 2023-09-26T19:50:26Z

/azp run

azure-pipelines · 2023-09-26T19:50:36Z

Azure Pipelines successfully started running 1 pipeline(s).

baiwei0427 · 2023-09-26T22:47:51Z

tests/common/snappi_tests/traffic_generation.py

+                              .format(prios[prio]))
+            else:
+                # PFC pause frames should not be transmitted when test traffic is not paused
+                pytest_assert(pfc_pause_tx_frames <= PFC_FRAME_COUNT_TOL,


Reset to 0

Can you please remove PFC_FRAME_COUNT_TOL?

tests/common/snappi_tests/traffic_generation.py

developfast · 2023-10-02T03:58:51Z

/azp run

azure-pipelines · 2023-10-02T03:59:01Z

Azure Pipelines successfully started running 1 pipeline(s).

…nter polling (#9847) Summary: Currently, all verification for PFC pause frames are done by polling the ixia device at the end of the test. We need to add functionality to poll the appropriate counters on the DUT while the DUT continues to receive PFC pause frames from the neighboring device. This PR is dependent on #9775. Fixes (issue) #9843

mssonicbld · 2023-10-02T19:06:57Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/common/snappi_tests/traffic_generation.py:662:5: F841 local variable 'test_tx_frames' is assigned to but never used tests/common/snappi_tests/traffic_generation.py:667:16: F821 undefined name 'test_traffic_pause' tests/common/snappi_tests/traffic_generation.py:678:16: F821 undefined name 'test_traffic_pause' tests/common/snappi_tests/traffic_generation.py:687:1: F811 redefinition of unused 'verify_tx_frame_count_dut' from line 616 tests/common/snappi_tests/traffic_generation.py:715:1: F811 redefinition of unused 'verify_rx_frame_count_dut' from line 644 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

developfast · 2023-10-05T00:11:12Z

/azp run

azure-pipelines · 2023-10-05T00:11:22Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2023-10-24T09:44:59Z

@developfast PR conflicts with 202012 branch

mssonicbld · 2023-10-24T09:45:01Z

@developfast PR conflicts with 202305 branch

mssonicbld · 2023-10-24T09:45:02Z

@developfast PR conflicts with 202205 branch

…nter polling (sonic-net#9847) Summary: Currently, all verification for PFC pause frames are done by polling the ixia device at the end of the test. We need to add functionality to poll the appropriate counters on the DUT while the DUT continues to receive PFC pause frames from the neighboring device. This PR is dependent on sonic-net#9775. Fixes (issue) sonic-net#9843

…nter polling (#9847) Summary: Currently, all verification for PFC pause frames are done by polling the ixia device at the end of the test. We need to add functionality to poll the appropriate counters on the DUT while the DUT continues to receive PFC pause frames from the neighboring device. This PR is dependent on #9775. Fixes (issue) #9843

developfast and others added 4 commits August 30, 2023 21:45

pfc counter checker changed

c9cc46a

tx_ok and rx_ok checker ready

1f3e4ae

fixed placement of functions

b886204

fixed comments on snappi test params

920d74e

Fix whitespace issue

acee992

Update flake checker

50ed8d2

developfast marked this pull request as ready for review August 30, 2023 22:03

Update traffic_generation.py

e032737

Update test_pfc_pause_lossless_with_snappi.py

b7091bb

neethajohn reviewed Aug 30, 2023

View reviewed changes

tests/common/snappi_tests/common_helpers.py Outdated Show resolved Hide resolved

developfast and others added 3 commits August 31, 2023 20:05

fixed conditional and added regex for counting

c9afb0a

Merge branch 'tx_rx_check_test' of github.com:developfast/sonic-mgmt …

fb709e7

…into tx_rx_check_test

removed whitespace

1a9ea54

developfast requested a review from neethajohn August 31, 2023 20:11

trimmed whitespace

fd3e33d

developfast requested a review from baiwei0427 September 5, 2023 15:59

developfast mentioned this pull request Sep 6, 2023

[Snappi] Update test PFC pause frame functionality to include DUT counter polling #9847

Merged

6 tasks

baiwei0427 requested changes Sep 15, 2023

View reviewed changes

fixed requested changes

ab5904c

developfast requested a review from baiwei0427 September 20, 2023 18:37

baiwei0427 reviewed Sep 22, 2023

View reviewed changes

Update traffic_generation.py

e946d92

developfast requested a review from baiwei0427 September 25, 2023 10:22

baiwei0427 reviewed Sep 26, 2023

View reviewed changes

resolved comments in for loop

4a08bad

Merge branch 'master' into tx_rx_check_test

306bf91

developfast added 3 commits October 3, 2023 17:20

fix merge conflict

2d18d57

Update traffic_generation.py

70c4b2c

Update traffic_generation.py

6cf3465

neethajohn approved these changes Oct 4, 2023

View reviewed changes

developfast merged commit 633892f into sonic-net:master Oct 5, 2023

developfast added Request for 202012 branch Request for 202205 branch Request for 202305 branch labels Oct 12, 2023

wangxin added Approved for 202012 Branch Approved for 202205 branch Approved for 202305 branch labels Oct 24, 2023

mssonicbld added the Cherry Pick Conflict_202012 label Oct 24, 2023

mssonicbld added Cherry Pick Conflict_202305 Cherry Pick Conflict_202205 labels Oct 24, 2023

mssonicbld mentioned this pull request Oct 24, 2023

[action] [PR:9847] [Snappi] Update test PFC pause frame functionality to include DUT counter polling #10450

Merged

6 tasks

Conversation

developfast commented Aug 30, 2023

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Aug 30, 2023

Uh oh!

mssonicbld commented Aug 30, 2023

Uh oh!

mssonicbld commented Aug 30, 2023

Uh oh!

mssonicbld commented Aug 30, 2023

Uh oh!

mssonicbld commented Aug 30, 2023

Uh oh!

Uh oh!

mssonicbld commented Aug 31, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

developfast commented Sep 26, 2023

Uh oh!

azure-pipelines bot commented Sep 26, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

developfast commented Oct 2, 2023

Uh oh!

azure-pipelines bot commented Oct 2, 2023

Uh oh!

mssonicbld commented Oct 2, 2023

Uh oh!

developfast commented Oct 5, 2023

Uh oh!

azure-pipelines bot commented Oct 5, 2023

Uh oh!

mssonicbld commented Oct 24, 2023

Uh oh!

mssonicbld commented Oct 24, 2023

Uh oh!

mssonicbld commented Oct 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants