[test_upgrade_path] Support test_upgrade_path on KVM switch on VS testbed by vaibhavhd · Pull Request #2661 · sonic-net/sonic-mgmt

vaibhavhd · 2020-12-14T23:19:23Z

Description of PR

Summary: Make test_upgrade_path testcase KVM ready.
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Approach

What is the motivation for this PR?

test_upgrade_path presently is not supported and enabled on KVM switch. Make changes to:

Support SCP'ing the base/target images from a path within sonic-mgmt container.
Support the image download from a remote URL.

How did you do it?

Added VS setup marker in test_upgrade_path.
Detect path type - Hyperlink vs filesystem path. Transfer or download the images accordingly.
Add a default mgmt gw route so that switch can download the image from a remore URL. Remove this after download.

How did you verify/test it?

Executed test_upgrade_path using both remote public URLs and local file path:

Upgrade KVM switch using URL of public Jenkins images:

./run_tests.sh -n vms-kvm-t0 -d vlab01 -f ../ansible/vtestbed.csv -i ../ansible/veos_vtb -u -c upgrade_path/test_upgrade_path.py -e "--base_image_list=https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin --target_image_list=https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin" -
l info
---

21:51:20 INFO test_upgrade_path.py:test_upgrade_path:190: Test upgrade path from https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin to https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin
21:51:20 INFO test_upgrade_path.py:test_upgrade_path:192: Installing https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin
21:51:21 INFO devices.py:get_ip_route_info:770: route raw info for 0.0.0.0/0: [u'default proto bgp src 10.1.0.32 metric 20 ', u'\tnexthop via 10.0.0.57 dev PortChannel0001 weight 1 ', u'\tnexthop via 10.0.0.59 dev PortChannel0002 weight 1 ', u'\tnexthop via 10.0.0.61 dev PortChannel0003 weight 1 ', u'\tnexthop via 10.0.0.63 dev PortChannel0004 weight 1 ']
21:51:21 INFO devices.py:get_ip_route_info:805: route parsed info for 0.0.0.0/0: {'set_src': IPv4Address(u'10.1.0.32'), 'nexthops': [(IPv4Address(u'10.0.0.57'), u'PortChannel0001'), (IPv4Address(u'10.0.0.59'), u'PortChannel0002'), (IPv4Address(u'10.0.0.61'), u'PortChannel0003'), (IPv4Address(u'10.0.0.63'), u'PortChannel0004')]}
21:51:21 INFO test_upgrade_path.py:install_sonic:159: Adding default mgmt-gateway-route to the device via 10.250.0.1
21:52:30 INFO test_upgrade_path.py:install_sonic:176: Remove default mgmt-gateway-route earlier added
21:52:30 INFO test_upgrade_path.py:test_upgrade_path:195: Cold reboot the DUT to make the base image as current
21:52:34 INFO reboot.py:reboot:121: waiting for ssh to drop
21:52:34 INFO reboot.py:execute_reboot_command:105: rebooting with command "reboot"
21:52:44 INFO reboot.py:reboot:139: waiting for ssh to startup
21:53:46 INFO reboot.py:reboot:150: ssh has started up
21:53:46 INFO reboot.py:reboot:152: waiting for switch to initialize
21:55:46 INFO reboot.py:reboot:180: cold reboot finished
21:55:51 INFO reboot.py:reboot:183: DUT up since 2020-12-14 21:53:14
21:55:52 INFO test_upgrade_path.py:test_upgrade_path:200: Upgrading to https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin
21:55:53 INFO devices.py:get_ip_route_info:770: route raw info for 0.0.0.0/0: [u'default proto bgp src 10.1.0.32 metric 20 ', u'\tnexthop via 10.0.0.57 dev PortChannel0001 weight 1 ', u'\tnexthop via 10.0.0.59 dev PortChannel0002 weight 1 ', u'\tnexthop via 10.0.0.61 dev PortChannel0003 weight 1 ', u'\tnexthop via 10.0.0.63 dev PortChannel0004 weight 1 ']
21:55:53 INFO devices.py:get_ip_route_info:805: route parsed info for 0.0.0.0/0: {'set_src': IPv4Address(u'10.1.0.32'), 'nexthops': [(IPv4Address(u'10.0.0.57'), u'PortChannel0001'), (IPv4Address(u'10.0.0.59'), u'PortChannel0002'), (IPv4Address(u'10.0.0.61'), u'PortChannel0003'), (IPv4Address(u'10.0.0.63'), u'PortChannel0004')]}
21:55:53 INFO test_upgrade_path.py:install_sonic:159: Adding default mgmt-gateway-route to the device via 10.250.0.1
21:56:59 INFO test_upgrade_path.py:install_sonic:176: Remove default mgmt-gateway-route earlier added
21:57:01 INFO ssh_utils.py:prepare_testbed_ssh_keys:17: Remove old keys from ptfhost
21:57:04 INFO ssh_utils.py:prepare_testbed_ssh_keys:26: Generate public key for ptf host
PASSED                                                                                                                                                                                                               [100%]

Upgrade KVM switch from a file path within sonic-mgmt container:

./run_tests.sh -n vms-kvm-t0 -d vlab01 -f ../ansible/vtestbed.csv -i ../ansible/veos_vtb -u -c upgrade_path/test_upgrade_path.py -e "--base_image_list=/data/sonic-vs-580.bin --target_image_list=/data/sonic-vs-586.bin" -l info
---

23:00:27 INFO test_upgrade_path.py:test_upgrade_path:191: Test upgrade path from /data/sonic-vs-580.bin to /data/sonic-vs-586.bin
23:00:27 INFO test_upgrade_path.py:test_upgrade_path:193: Installing /data/sonic-vs-580.bin
23:00:27 INFO devices.py:get_ip_route_info:770: route raw info for 0.0.0.0/0: []
23:00:27 INFO test_upgrade_path.py:install_sonic:160: Adding default mgmt-gateway-route to the device via 10.250.0.1
23:01:58 INFO test_upgrade_path.py:install_sonic:177: Remove default mgmt-gateway-route earlier added
23:01:59 INFO test_upgrade_path.py:test_upgrade_path:196: Cold reboot the DUT to make the base image as current
23:02:03 INFO reboot.py:reboot:121: waiting for ssh to drop
23:02:03 INFO reboot.py:execute_reboot_command:105: rebooting with command "reboot"
23:02:13 INFO reboot.py:reboot:139: waiting for ssh to startup
23:03:19 INFO reboot.py:reboot:150: ssh has started up
23:03:19 INFO reboot.py:reboot:152: waiting for switch to initialize
23:05:19 INFO reboot.py:reboot:180: cold reboot finished
23:05:24 INFO reboot.py:reboot:183: DUT up since 2020-12-14 23:02:48
23:05:25 INFO test_upgrade_path.py:test_upgrade_path:201: Upgrading to /data/sonic-vs-586.bin
23:05:26 INFO devices.py:get_ip_route_info:770: route raw info for 0.0.0.0/0: [u'default proto bgp src 10.1.0.32 metric 20 ', u'\tnexthop via 10.0.0.57 dev PortChannel0001 weight 1 ', u'\tnexthop via 10.0.0.59 dev PortChannel0002 weight 1 ', u'\tnexthop via 10.0.0.61 dev PortChannel0003 weight 1 ', u'\tnexthop via 10.0.0.63 dev PortChannel0004 weight 1 ']
23:05:26 INFO devices.py:get_ip_route_info:805: route parsed info for 0.0.0.0/0: {'set_src': IPv4Address(u'10.1.0.32'), 'nexthops': [(IPv4Address(u'10.0.0.57'), u'PortChannel0001'), (IPv4Address(u'10.0.0.59'), u'PortChannel0002'), (IPv4Address(u'10.0.0.61'), u'PortChannel0003'), (IPv4Address(u'10.0.0.63'), u'PortChannel0004')]}
23:05:26 INFO test_upgrade_path.py:install_sonic:160: Adding default mgmt-gateway-route to the device via 10.250.0.1
23:06:52 INFO test_upgrade_path.py:install_sonic:177: Remove default mgmt-gateway-route earlier added
23:06:54 INFO ssh_utils.py:prepare_testbed_ssh_keys:17: Remove old keys from ptfhost
23:06:57 INFO ssh_utils.py:prepare_testbed_ssh_keys:26: Generate public key for ptf host
PASSED

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

shi-su · 2020-12-15T01:24:19Z

Are there any other steps required to download images from public Jenkins? I tried to pull the code and run the test using the command provided but got the following failure. Seems that the mgmt gw route is not added properly. Not sure if there is anything I missed.

./run_tests.sh -n vms-kvm-t0 -d vlab-01 -f ../ansible/vtestbed.csv -i ../ansible/veos_vtb -u -c upgrade_path/test_upgrade_p_list=https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin --target_image_list=https://sonic-jenkins.wem/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin" -l info
=== Running tests in groups ===
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Supported in cryptography, and will be removed in a future release.
  CryptographyDeprecationWarning,
================================================================================== test session starts ============================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.9.0, pluggy-0.13.1
ansible: 2.8.12
rootdir: /data/sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, metadata-1.10.0, repeat-0.8.0, xdist-1.28.0, forked-1.3.0, html-1.22.1
collected 1 item

upgrade_path/test_upgrade_path.py::test_upgrade_path
------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------
00:48:44 INFO __init__.py:set_default:49: Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic
00:48:44 INFO __init__.py:check_test_completeness:128: Test has no defined levels. Continue without test completeness checks
00:48:52 INFO ptfhost_utils.py:change_mac_addresses:86: Change interface MAC addresses on ptfhost 'ptf-01'
00:48:53 INFO ptfhost_utils.py:copy_arp_responder_py:117: Copy arp_responder.py to ptfhost 'ptf-01'
00:48:56 INFO ptfhost_utils.py:copy_ptftests_directory:48: Copy PTF test files to PTF host 'ptf-01'
00:49:15 INFO ptfhost_utils.py:remove_ip_addresses:99: Remove existing IPs on ptfhost 'ptf-01'
00:49:16 INFO conftest.py:generate_params_dut_hostname:606: DUTs in testbed topology: ['vlab-01']
00:49:16 INFO conftest.py:creds:336: dut vlab-01 belongs to groups [u'lab', u'sonic', 'fanout']
00:49:16 INFO conftest.py:creds:348: skip empty var file ../ansible/group_vars/all/env.yml
00:49:16 INFO conftest.py:creds:348: skip empty var file ../ansible/group_vars/all/corefile_uploader.yml
00:49:17 INFO __init__.py:sanity_check:46: Start pre-test sanity check
00:49:17 INFO __init__.py:sanity_check:56: Found marker: m.name=topology, m.args=('any',), m.kwargs={}
00:49:17 INFO __init__.py:sanity_check:56: Found marker: m.name=sanity_check, m.args=(), m.kwargs={'skip_sanity': True}
00:49:17 INFO __init__.py:sanity_check:62: Process marker sanity_check in script. m.args=(), m.kwargs={'skip_sanity': True}
00:49:17 INFO __init__.py:sanity_check:90: Sanity check settings: skip_sanity=True, check_items=set(['bgp', 'processes', 'interfaces', 'monit', 'services', 'dbmemoue, recover_method=adaptive, post_check=False
00:49:17 INFO __init__.py:sanity_check:93: Skip sanity check according to command line argument or configuration of test script.
00:49:18 INFO test_upgrade_path.py:prepare_ptf:58: Preparing ptfhost
00:49:24 INFO __init__.py:loganalyzer:17: Log analyzer is disabled
------------------------------------------------------------------------------------- live log call ---------------------------------------------------------------
00:49:24 INFO test_upgrade_path.py:test_upgrade_path:191: Test upgrade path from https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/58-vs.bin to https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin
00:49:24 INFO test_upgrade_path.py:test_upgrade_path:193: Installing https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/t
00:49:25 INFO devices.py:get_ip_route_info:770: route raw info for 0.0.0.0/0: [u'default via 10.250.0.1 dev eth0 ']
00:49:25 INFO devices.py:get_ip_route_info:805: route parsed info for 0.0.0.0/0: {'set_src': None, 'nexthops': [(IPv4Address(u'10.250.0.1'), u'eth0')]}
Loading callback plugin json of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/json.pyc
Loading callback plugin json of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/json.pyc
FAILED
----------------------------------------------------------------------------------- live log teardown -------------------------------------------------------------
00:49:47 INFO ptfhost_utils.py:remove_ip_addresses:104: Remove IPs to restore ptfhost 'ptf-01'
00:49:47 INFO ptfhost_utils.py:copy_ptftests_directory:53: Delete PTF test files from PTF host 'ptf-01'
00:49:48 INFO ptfhost_utils.py:copy_arp_responder_py:122: Delete arp_responder.py from ptfhost 'ptf-01'


======================================================================================== FAILURES =================================================================
___________________________________________________________________________________ test_upgrade_path _____________________________________________________________

localhost = <tests.common.devices.Localhost object at 0x7fedb77ecad0>, duthosts = <tests.common.devices.DutHosts object at 0x7fedbdf98c10>, rand_one_dut_hostname =
ptfhost = <tests.common.devices.PTFHost object at 0x7fedb8420350>
upgrade_path_lists = ('https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin', '...enkins.westus2.clouda/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin', '')
ptf_params = {'arista_vms': [u'10.250.0.54', u'10.250.0.53', u'10.250.0.52', u'10.250.0.51'], 'default_ip_range': '192.168.100.0/18', 'dut_hostname': u'10.250.0.100:c6:f2:50', ...}
setup = None, tbinfo = {'comment': 'Tests virtual switch vm', 'conf-name': 'vms-kvm-t0', 'duts': ['vlab-01'], 'duts_map': {'vlab-01': 0}, ...}

    @pytest.mark.device_type('vs')
    def test_upgrade_path(localhost, duthosts, rand_one_dut_hostname, ptfhost, upgrade_path_lists, ptf_params, setup, tbinfo):
        duthost = duthosts[rand_one_dut_hostname]
        from_list_images, to_list_images, _ = upgrade_path_lists
        from_list = from_list_images.split(',')
        to_list = to_list_images.split(',')
        assert (from_list and to_list)
        for from_image in from_list:
            for to_image in to_list:
                logger.info("Test upgrade path from {} to {}".format(from_image, to_image))
                # Install base image
                logger.info("Installing {}".format(from_image))
>               target_version = install_sonic(duthost, from_image, tbinfo)

_          = ''
duthost    = <tests.common.devices.MultiAsicSonicHost object at 0x7fedb8e20e10>
duthosts   = <tests.common.devices.DutHosts object at 0x7fedbdf98c10>
from_image = 'https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin'
from_list  = ['https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin']
from_list_images = 'https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin'
localhost  = <tests.common.devices.Localhost object at 0x7fedb77ecad0>
ptf_params = {'arista_vms': [u'10.250.0.54', u'10.250.0.53', u'10.250.0.52', u'10.250.0.51'], 'default_ip_range': '192.168.100.0/18', 'dut_hostname': u'10.250.0.100:c6:f2:50', ...}
ptfhost    = <tests.common.devices.PTFHost object at 0x7fedb8420350>
rand_one_dut_hostname = 'vlab-01'
setup      = None
tbinfo     = {'comment': 'Tests virtual switch vm', 'conf-name': 'vms-kvm-t0', 'duts': ['vlab-01'], 'duts_map': {'vlab-01': 0}, ...}
to_image   = 'https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin'
to_list    = ['https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin']
to_list_images = 'https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin'
upgrade_path_lists = ('https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin', '...enkins.westus2.clouda/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin', '')

upgrade_path/test_upgrade_path.py:194:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
upgrade_path/test_upgrade_path.py:172: in install_sonic
    res = duthost.reduce_and_add_sonic_images(new_image_url=image_url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.common.devices.SonicHost object at 0x7fedb8e20a90>, module_args = ()
complex_args = {'new_image_url': 'https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin'}
previous_frame = <frame object at 0x2302e10>, filename = '/data/sonic-mgmt/tests/upgrade_path/test_upgrade_path.py', line_number = 172, function_name = 'install_so
lines = ['        res = duthost.reduce_and_add_sonic_images(new_image_url=image_url)\n'], index = 0, module_ignore_errors = False, module_async = False

    def _run(self, *module_args, **complex_args):

        previous_frame = inspect.currentframe().f_back
        filename, line_number, function_name, lines, index = inspect.getframeinfo(previous_frame)

        logging.debug("{}::{}#{}: [{}] AnsibleModule::{}, args={}, kwargs={}"\
            .format(filename, function_name, line_number, self.hostname,
                    self.module_name, json.dumps(module_args), json.dumps(complex_args)))

        module_ignore_errors = complex_args.pop('module_ignore_errors', False)
        module_async = complex_args.pop('module_async', False)

        if module_async:
            def run_module(module_args, complex_args):
                return self.module(*module_args, **complex_args)[self.hostname]
            pool = ThreadPool()
            result = pool.apply_async(run_module, (module_args, complex_args))
            return pool, result

        res = self.module(*module_args, **complex_args)[self.hostname]
        logging.debug("{}::{}#{}: [{}] AnsibleModule::{} Result => {}"\
            .format(filename, function_name, line_number, self.hostname, self.module_name, json.dumps(res)))

        if (res.is_failed or 'exception' in res) and not module_ignore_errors:
>           raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res)
E           RunAnsibleModuleFail: run module reduce_and_add_sonic_images failed, Ansible Results =>
E           {
E               "changed": false,
E               "failed": true,
E               "module_stderr": "OpenSSH_7.2p2 Ubuntu-4ubuntu2.10, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1:ne 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_clienid = 17142\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packeBroken pipe\r\ndebug2: Received exit status from master 1\r\n",
E               "module_stdout": "\n{\"msg\": \"Failed downloading new image: rc=6, out=, err=  % Total    % Received % Xferd  Average Speed   Time    Time     Tim                       Dload  Upload   Total   Spent    Left  Speed\\n\\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\\r  0     0     0      0 --:--:-- --:--:-- --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\\r  0     0    0     0    0     0  0:00:02 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0r  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0\\r 0     0      0      0 --:--:--  0:00:07 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0\\r  0     0    0     0    -:--:--  0:00:09 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0\\r  0     0    0     0    0     0      0      0 ---     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0\\r  0     0    0       0 --:--:--  0:00:16 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0\\r  0     0    0     0    0     0      0 8 --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: sonic-jenkins.westus2.cloudapp.a\": true, \"invocation\": {\"module_args\": {\"disk_used_pcent\": 8, \"new_image_url\": \"https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vsget/sonic-vs.bin\"}}}\n\n{\"msg\": \"Error: (<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x7f66b4c06830>)\", \"failed\": true, \"exceptionw traceback may *not* be related to the actual failure.\\n  File \\\"/tmp/ansible_reduce_and_add_sonic_images_payload_DkXc_T/__main__.py\\\", line 118, in main\\n age(module, new_image_url)\\n  File \\\"/tmp/ansible_reduce_and_add_sonic_images_payload_DkXc_T/__main__.py\\\", line 68, in install_new_sonic_image\\n    download, new_image_url, save_as)\\n  File \\\"/tmp/ansible_reduce_and_add_sonic_images_payload_DkXc_T/__main__.py\\\", line 56, in download_new_sonic_image\\n    msg=\\\"\")\\n  File \\\"/tmp/ansible_reduce_and_add_sonic_images_payload_DkXc_T/__main__.py\\\", line 35, in exec_command\\n    (msg, rc, out, err))\\n  File \\\"/tmp/ansic_images_payload_DkXc_T/ansible_reduce_and_add_sonic_images_payload.zip/ansible/module_utils/basic.py\\\", line 2071, in fail_json\\n    sys.exit(1)\\n\", \"invoc\": {\"disk_used_pcent\": 8, \"new_image_url\": \"https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin\
E               "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
E               "rc": 1
E           }

complex_args = {'new_image_url': 'https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin'}
filename   = '/data/sonic-mgmt/tests/upgrade_path/test_upgrade_path.py'
function_name = 'install_sonic'
index      = 0
line_number = 172
lines      = ['        res = duthost.reduce_and_add_sonic_images(new_image_url=image_url)\n']
module_args = ()
module_async = False
module_ignore_errors = False
previous_frame = <frame object at 0x2302e10>
res        = {'failed': True, '_ansible_no_log': False, 'rc': 1, 'msg': u'MODULE FAILURE\nS...ure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.b
self       = <tests.common.devices.SonicHost object at 0x7fedb8e20a90>

common/devices.py:95: RunAnsibleModuleFail
----------------------------------------------------------------- generated xml file: /data/sonic-mgmt/tests/logs/tr.xml ------------------------------------------
================================================================================ short test summary info ==========================================================
FAILED upgrade_path/test_upgrade_path.py::test_upgrade_path - RunAnsibleModuleFail: run module reduce_and_add_sonic_images failed, Ansible Results =>
=============================================================================== 1 failed in 64.24 seconds =========================================================

vaibhavhd · 2020-12-15T19:39:02Z

Are there any other steps required to download images from public Jenkins? I tried to pull the code and run the test using the command provided but got the following failure. Seems that the mgmt gw route is not added properly. Not sure if there is anything I missed.

Thanks for trying this, Shi.

Based on an offline discussion - there is a prerequisite before the test that allows KVM to download the images from Internet:
Execute https://github.com/Azure/sonic-mgmt/blob/master/ansible/setup-br1-nat.sh to set up NAT for br1 internal management network.

This step should ideally be part of testbed setup, as it is needed only for VS setup, and not necessarily for physical devices. For now, I am considering of adding it to this test itself, protected within a KVM check.

wangxin · 2021-01-04T13:35:56Z

+            new_route_added = False
+            break
+    else:
+        # Temprarily change the default route to mgmt-gateway address. This is done so that


Typo 'Temprarily' -> 'Temporarily'

Fixed now, thanks!

vaibhavhd · 2021-01-04T16:51:38Z

@shi-su I have added a verification step for checking service health status and reboot cause after warm-upgrade. This is to handle/catch any false-positives when syncd crashed or reboot-cause is COLD.

As far as connectivity to internet is concerned, this should be included in a testbed setup if we agree that it is generally safe. This can be dealt with in different PR.

With the present set of changes, test upgrade will work for KVM for locally (in container) stored images. If the network is configured manually to enable internet for the KVM switch, this test can also download remote images.

wangxin

I agree that configuration for internet access should be responsibility of testbed setup, not test script.

wangxin · 2021-01-05T07:41:57Z

retest vsimage please

vaibhavhd · 2021-01-07T06:08:55Z

retest vsimage please

Description of PR Summary: Fix bug introduced by #2661 where if new_image_url is not provided, the installation logic still proceeds. This leads to error in the Ansible module in upgrade_sonic.yml step for removing old sonic images. Approach What is the motivation for this PR? How did you do it? Added the check for no image URL. And do not execute installation logic if URL is None. How did you verify/test it? Tested on a physical lab device, and the Ansible execution of upgrade_sonic.yml passed.

…aemons] advance submodule head (sonic-net#13755) linkmgrd: * e191338 2023-02-10 | Fix the warning of unused variables (sonic-net#167) (HEAD -> 202205) [Longxiang Lyu] utilities: * 2c933b0a 2023-02-07 | [sai_failure_dump]Invoking dump during SAI failure (sonic-net#2633) (HEAD -> 202205) [Sudharsan Dhamal Gopalarathnam] * e949f318 2023-02-07 | [show] add support for gRPC show commands for `active-active` (sonic-net#2629) [vdahiya12] * 77723927 2023-01-30 | Fixed admin state config CLI for Backport interfaces (sonic-net#2557) [anamehra] * 32b1d4d6 2023-02-01 | [masic support] 'show run bgp' support for multi-asic (sonic-net#2427) [wenyiz2021] * a2252d8a 2022-10-11 | Filter port invalid MTU configuration (sonic-net#2378) [pettershao-ragilenetworks] * 0ffb4b6a 2023-02-09 | Add Transceiver PM basic CLI support to show output from TRANSCEIVER_PM table for ZR (sonic-net#2655) (github/202205) [longhuan-cisco] * 496a0774 2023-02-09 | Add asic id for linecards so "show fabric counters queue/port" can work for single chip systems (sonic-net#2656) [jfeng-arista] * 2591e8b5 2023-02-03 | multi asic support for show queue counter (sonic-net#2647) [zhixzhu] swss: * e0373a4 2023-02-07 | [autoneg]Fixing adv interface types to be set when AN is disabled (sonic-net#2638) (HEAD -> 202205, github/202205) [Sudharsan Dhamal Gopalarathnam] * 62a09a0 2023-02-09 | [sai_failure_dump]Invoking dump during SAI failure (sonic-net#2644) (sonic-net#2661) [Sudharsan Dhamal Gopalarathnam] * 076f63e 2023-02-08 | [202205] Revert "Revert "[voq][chassis]Add show fabric counters port/queue commands (sonic-net#2522)" (sonic-net#2612)" (sonic-net#2655) [kenneth-arista] * a35e074 2023-02-06 | [202205][voq][chassis] Remove created ports from the default vlan. (sonic-net#2651) [arista-nwolfe] swss-common: * b9d4284 2023-02-08 | [202205] Fix epoll and socket resource leak issue. (sonic-net#651) (sonic-net#741) (github/202205) [Kevin Petremann] sairedis: * 9d8e731 2023-02-08 | [Mellanox] Enable DSCP remapping by using SAI attribute (sonic-net#1188) (HEAD -> 202205, github/202205) [Stephen Sun] * 272a8bd 2023-02-10 | Fixing race condition for rif counters sonic-net#1136 (sonic-net#1202) [Suman Kumar] * 211365a 2023-02-08 | [202205][submodule][SAI]Advance SAI header (sonic-net#1207) [Richard.Yu] * 939c14b 2023-02-08 | [Submodule][upgrade]Upgrade SAI submodule (sonic-net#1203) [Richard.Yu] platform-daemons: * e5ccd40 2022-10-03 | [ycabled] fix naming error for error condition for CLI handling (sonic-net#302) (HEAD -> 202205, github/202205) [vdahiya12] * cdd354d 2022-09-29 | [ycabled] add some exception catching logic to some vendor specific API's (sonic-net#301) [vdahiya12] * cf58c08 2023-02-01 | Chassisd do an explicit stop of the config_manager (sonic-net#328) (sonic-net#336) [judyjoseph] Signed-off-by: Ying Xie <ying.xie@microsoft.com>

vaibhavhd added 2 commits December 14, 2020 14:06

Make test_upgrade ready for VS setup and support local image download

bd583f0

Fix flag use for new route that is added

2eb0252

vaibhavhd added the Pytest 🐍 label Dec 14, 2020

vaibhavhd requested review from a team and shi-su December 14, 2020 23:19

vaibhavhd changed the title ~~Support test_upgrade_path on KVM switch on VS testbed~~ [test_upgrade_path] Support test_upgrade_path on KVM switch on VS testbed Dec 15, 2020

Merge remote-tracking branch 'upstream/master' into upgrade-path-test-vs

92300f4

wangxin reviewed Jan 4, 2021

View reviewed changes

Check reboot-cause and services health after warm upgrade

6c8e082

wangxin approved these changes Jan 5, 2021

View reviewed changes

shi-su reviewed Jan 5, 2021

View reviewed changes

Comment thread tests/upgrade_path/test_upgrade_path.py Outdated

Comment thread tests/upgrade_path/test_upgrade_path.py Outdated

Address comments - handle image_name from upgrade test to Ansible module

5adb59c

shi-su reviewed Jan 7, 2021

View reviewed changes

Comment thread ansible/library/reduce_and_add_sonic_images.py

Check fo r file-path with tmpfs and then mount dir

89ab493

shi-su approved these changes Jan 7, 2021

View reviewed changes

vaibhavhd merged commit d97d086 into sonic-net:master Jan 7, 2021

vaibhavhd deleted the upgrade-path-test-vs branch January 7, 2021 16:57

vaibhavhd mentioned this pull request Jan 8, 2021

Upgrade path error fix - handle the cleanup only case without image URL #2783

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[test_upgrade_path] Support test_upgrade_path on KVM switch on VS testbed#2661

[test_upgrade_path] Support test_upgrade_path on KVM switch on VS testbed#2661
vaibhavhd merged 6 commits intosonic-net:masterfrom
vaibhavhd:upgrade-path-test-vs

vaibhavhd commented Dec 14, 2020

Uh oh!

shi-su commented Dec 15, 2020

Uh oh!

vaibhavhd commented Dec 15, 2020 •

edited

Loading

Uh oh!

wangxin Jan 4, 2021

Uh oh!

vaibhavhd Jan 4, 2021

Uh oh!

vaibhavhd commented Jan 4, 2021

Uh oh!

wangxin left a comment

Uh oh!

wangxin commented Jan 5, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vaibhavhd commented Jan 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

vaibhavhd commented Dec 14, 2020

Description of PR

Type of change

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

shi-su commented Dec 15, 2020

Uh oh!

vaibhavhd commented Dec 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangxin Jan 4, 2021

Choose a reason for hiding this comment

Uh oh!

vaibhavhd Jan 4, 2021

Choose a reason for hiding this comment

Uh oh!

vaibhavhd commented Jan 4, 2021

Uh oh!

wangxin left a comment

Choose a reason for hiding this comment

Uh oh!

wangxin commented Jan 5, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vaibhavhd commented Jan 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vaibhavhd commented Dec 15, 2020 •

edited

Loading