Skip to content

[everflow] Refactor Everflow tests to support different config modes#2133

Merged
daall merged 2 commits intosonic-net:masterfrom
daall:mirror_dscp_testing
Sep 8, 2020
Merged

[everflow] Refactor Everflow tests to support different config modes#2133
daall merged 2 commits intosonic-net:masterfrom
daall:mirror_dscp_testing

Conversation

@daall
Copy link
Copy Markdown
Contributor

@daall daall commented Aug 25, 2020

  • Convert acl.json to a template
  • Add "config_mode" fixture to run tests against CLI or configlet

Signed-off-by: Danny Allen daall@microsoft.com

Description of PR

Summary: [everflow] Refactor Everflow tests to support different config modes

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Approach

What is the motivation for this PR?

There are multiple ways to configure ACLs/Everflow on a SONiC device, with two of the more prevalent ones being CLI and Configlet. We would like to be able to verify that both config methods produce the same results, however we don't want to write another set of test cases just to verify configlet.

How did you do it?

I made the following changes:

  1. ACL Rules are now generated from a .j2 template, with the data being fed in from a YAML file that contains all the desired rules + their qualifiers. This gives us 1 source of truth that will be used to generate both acl.json and configlet based ACL rules.
  2. The Everflow tests have an additional "config_mode" fixture that will run the tests against CLI, configlet (in a later PR), and any other configuration method we want to verify. To keep this PR short, currently only the existing CLI code is supported.
  3. The setup/teardown code has been consolidated so that there is one flow for v4, one flow for v6, and one flow for DSCP. The goal is to eventually have 2 (v4/v6 and dscp) or even 1.

How did you verify/test it?

Ran locally on two different platforms. No egress support:

=============================================================================================== test session starts ================================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
ansible: 2.8.7
rootdir: /var/src/Networking-acs-sonic-mgmt/tests, inifile: pytest.ini
plugins: forked-1.1.3, xdist-1.28.0, ansible-2.2.2, repeat-0.8.0
collected 40 items                                                                                                                                                                                                 

everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_basic_forwarding[cli-tor] PASSED                                                                                     [  2%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_basic_forwarding[cli-spine] PASSED                                                                                   [  5%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_neighbor_mac_change[cli-tor] PASSED                                                                                  [  7%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_neighbor_mac_change[cli-spine] PASSED                                                                                [ 10%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-tor] PASSED                                                                          [ 12%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-spine] PASSED                                                                        [ 15%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_remove_used_ecmp_next_hop[cli-tor] PASSED                                                                            [ 17%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_remove_used_ecmp_next_hop[cli-spine] PASSED                                                                          [ 20%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_dscp_with_policer[cli-tor] PASSED                                                                                    [ 22%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_dscp_with_policer[cli-spine] PASSED                                                                                  [ 25%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_basic_forwarding[cli-tor] SKIPPED                                                                                     [ 27%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_basic_forwarding[cli-spine] SKIPPED                                                                                   [ 30%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_neighbor_mac_change[cli-tor] SKIPPED                                                                                  [ 32%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_neighbor_mac_change[cli-spine] SKIPPED                                                                                [ 35%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-tor] SKIPPED                                                                          [ 37%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-spine] SKIPPED                                                                        [ 40%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_remove_used_ecmp_next_hop[cli-tor] SKIPPED                                                                            [ 42%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_remove_used_ecmp_next_hop[cli-spine] SKIPPED                                                                          [ 45%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_dscp_with_policer[cli-tor] SKIPPED                                                                                    [ 47%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_dscp_with_policer[cli-spine] SKIPPED                                                                                  [ 50%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_basic_forwarding[cli-tor] SKIPPED                                                                                     [ 52%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_basic_forwarding[cli-spine] SKIPPED                                                                                   [ 55%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_neighbor_mac_change[cli-tor] SKIPPED                                                                                  [ 57%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_neighbor_mac_change[cli-spine] SKIPPED                                                                                [ 60%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-tor] SKIPPED                                                                          [ 62%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-spine] SKIPPED                                                                        [ 65%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_remove_used_ecmp_next_hop[cli-tor] SKIPPED                                                                            [ 67%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_remove_used_ecmp_next_hop[cli-spine] SKIPPED                                                                          [ 70%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_dscp_with_policer[cli-tor] SKIPPED                                                                                    [ 72%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_dscp_with_policer[cli-spine] SKIPPED                                                                                  [ 75%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_basic_forwarding[cli-tor] SKIPPED                                                                                      [ 77%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_basic_forwarding[cli-spine] SKIPPED                                                                                    [ 80%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_neighbor_mac_change[cli-tor] SKIPPED                                                                                   [ 82%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_neighbor_mac_change[cli-spine] SKIPPED                                                                                 [ 85%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-tor] SKIPPED                                                                           [ 87%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-spine] SKIPPED                                                                         [ 90%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_remove_used_ecmp_next_hop[cli-tor] SKIPPED                                                                             [ 92%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_remove_used_ecmp_next_hop[cli-spine] SKIPPED                                                                           [ 95%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_dscp_with_policer[cli-tor] SKIPPED                                                                                     [ 97%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_dscp_with_policer[cli-spine] SKIPPED                                                                                   [100%]

===================================================================================== 10 passed, 30 skipped in 275.72 seconds ======================================================================================

Egress support:

=============================================================================================== test session starts ================================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
ansible: 2.8.7
rootdir: /var/src/Networking-acs-sonic-mgmt/tests, inifile: pytest.ini
plugins: forked-1.1.3, xdist-1.28.0, ansible-2.2.2, repeat-0.8.0
collected 40 items                                                                                                                                                                                                 

everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_basic_forwarding[cli-tor] PASSED                                                                                     [  2%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_basic_forwarding[cli-spine] PASSED                                                                                   [  5%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_neighbor_mac_change[cli-tor] PASSED                                                                                  [  7%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_neighbor_mac_change[cli-spine] PASSED                                                                                [ 10%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-tor] PASSED                                                                          [ 12%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-spine] PASSED                                                                        [ 15%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_remove_used_ecmp_next_hop[cli-tor] PASSED                                                                            [ 17%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_remove_used_ecmp_next_hop[cli-spine] PASSED                                                                          [ 20%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_dscp_with_policer[cli-tor] PASSED                                                                                    [ 22%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclIngressMirror::test_everflow_dscp_with_policer[cli-spine] PASSED                                                                                  [ 25%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_basic_forwarding[cli-tor] SKIPPED                                                                                     [ 27%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_basic_forwarding[cli-spine] SKIPPED                                                                                   [ 30%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_neighbor_mac_change[cli-tor] SKIPPED                                                                                  [ 32%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_neighbor_mac_change[cli-spine] SKIPPED                                                                                [ 35%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-tor] SKIPPED                                                                          [ 37%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-spine] SKIPPED                                                                        [ 40%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_remove_used_ecmp_next_hop[cli-tor] SKIPPED                                                                            [ 42%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_remove_used_ecmp_next_hop[cli-spine] SKIPPED                                                                          [ 45%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_dscp_with_policer[cli-tor] SKIPPED                                                                                    [ 47%]
everflow/test_everflow_testbed.py::TestEverflowV4IngressAclEgressMirror::test_everflow_dscp_with_policer[cli-spine] SKIPPED                                                                                  [ 50%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_basic_forwarding[cli-tor] SKIPPED                                                                                     [ 52%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_basic_forwarding[cli-spine] SKIPPED                                                                                   [ 55%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_neighbor_mac_change[cli-tor] SKIPPED                                                                                  [ 57%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_neighbor_mac_change[cli-spine] SKIPPED                                                                                [ 60%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-tor] SKIPPED                                                                          [ 62%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-spine] SKIPPED                                                                        [ 65%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_remove_used_ecmp_next_hop[cli-tor] SKIPPED                                                                            [ 67%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_remove_used_ecmp_next_hop[cli-spine] SKIPPED                                                                          [ 70%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_dscp_with_policer[cli-tor] SKIPPED                                                                                    [ 72%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclIngressMirror::test_everflow_dscp_with_policer[cli-spine] SKIPPED                                                                                  [ 75%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_basic_forwarding[cli-tor] PASSED                                                                                       [ 77%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_basic_forwarding[cli-spine] PASSED                                                                                     [ 80%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_neighbor_mac_change[cli-tor] PASSED                                                                                    [ 82%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_neighbor_mac_change[cli-spine] PASSED                                                                                  [ 85%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-tor] PASSED                                                                            [ 87%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_remove_unused_ecmp_next_hop[cli-spine] PASSED                                                                          [ 90%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_remove_used_ecmp_next_hop[cli-tor] PASSED                                                                              [ 92%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_remove_used_ecmp_next_hop[cli-spine] PASSED                                                                            [ 95%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_dscp_with_policer[cli-tor] PASSED                                                                                      [ 97%]
everflow/test_everflow_testbed.py::TestEverflowV4EgressAclEgressMirror::test_everflow_dscp_with_policer[cli-spine] PASSED                                                                                    [100%]

===================================================================================== 20 passed, 20 skipped in 488.13 seconds ======================================================================================

Any platform specific information?

N/A

Supported testbed topology if it's a new test case?

N/A

Documentation

- Convert acl.json to a template
- Add "config_mode" fixture to run tests against CLI or configlet

Signed-off-by: Danny Allen <daall@microsoft.com>
@lgtm-com
Copy link
Copy Markdown

lgtm-com bot commented Aug 25, 2020

This pull request introduces 2 alerts when merging f6bf8ff into 9da5e63 - view on LGTM.com

new alerts:

  • 2 for Unused import

@lgtm-com
Copy link
Copy Markdown

lgtm-com bot commented Aug 25, 2020

This pull request introduces 1 alert when merging 96cc84d into 9da5e63 - view on LGTM.com

new alerts:

  • 1 for Unused import

@daall daall merged commit 11c03ae into sonic-net:master Sep 8, 2020
@daall daall deleted the mirror_dscp_testing branch September 8, 2020 17:04
kazinator-arista pushed a commit to kazinator-arista/sonic-mgmt that referenced this pull request Mar 4, 2026
How I did it
Advance swss submodule head to include:

c3fb52b 2022-02-04 | Fix for missing lossless PG profile on certain ports (sonic-swss-common update for Vnet tables sonic-net#2133) (HEAD -> 202012, github/202012) [Ying Xie]

Signed-off-by: Ying Xie ying.xie@microsoft.com
kazinator-arista pushed a commit to kazinator-arista/sonic-mgmt that referenced this pull request Mar 4, 2026
…#12974)

utilities:
* 4b51e41 2022-12-06 | [config] Add check in config interface ip command to block if the interface is portchannel member (sonic-net#2539) (HEAD -> 202205) [Sudharsan Dhamal Gopalarathnam]
* e53b32e 2022-12-06 | [generate_dump] [Mellanox] Fix the duplicate dfw dump collection problem by adding symlinks (sonic-net#2536) [Vivek]
* 0391221 2022-12-02 | [GCU] Add RemoveCreateOnlyDependency Validator/Generator (sonic-net#2500) [jingwenxie]
* e3658e9 2022-04-13 | [scripts/fast-reboot] Shutdown remaining containers through systemd (sonic-net#2133) [Stepan Blyshchak]

swss:
* 1a4a5d9 2022-12-02 | [ACL] Support ACTION_COUNTER action in custom ACL table type (sonic-net#2550) [bingwang-ms]
* 33b0a9e 2022-12-05 | [muxorch] Adding case for maintaining current state (sonic-net#2280) [Nikola Dancejic]

sairedis:
* b29bb45 2022-12-02 | enable cisco8000 SAI bulk API feature (sonic-net#1153) (HEAD -> 202205) [Keith Lu]

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

Signed-off-by: Ying Xie <ying.xie@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants