Skip to content

[Stress] Enable OMZ info_dumper.py in get_testdata.py#1485

Merged
vurusovs merged 7 commits intoopenvinotoolkit:masterfrom
vurusovs:vu/stress_enable_omz_info_dumper
Jul 31, 2020
Merged

[Stress] Enable OMZ info_dumper.py in get_testdata.py#1485
vurusovs merged 7 commits intoopenvinotoolkit:masterfrom
vurusovs:vu/stress_enable_omz_info_dumper

Conversation

@vurusovs
Copy link
Copy Markdown
Contributor

@vurusovs vurusovs commented Jul 27, 2020

Use of OMZ info_dumper.py required to avoid hardcoded paths and get more information about every model (such as framework, precisions etc.)

Details:

  1. Environment configs are removed as not required, info from it moved to test configs
  2. Changes in product-configs aren't required due no changes in CLI, --env_conf marked as deprecated
  3. Records in test configs with source="omz" will be extended with info from info_dumper.py (for example, full path to the model will be written after get_testdata.py call)
  4. Records with source="share" should be maintained manually (as before)

Requirements for test configs:

  1. "path" and "full_path" keys in "model" record are required to run C++ test (runtime will throw exception)
  2. "source" and "name" keys in "model" record are required to run get_testdata.py to call OMZ (script will ignore records which aren't contain one of that keys)
    Note: If "source" == "omz" but "name" isn't available in OMZ, that record will be removed from test config to prevent exception in C++ part (see point 1)
  3. Every another key is optional and may be used for uploading to a DB as additional information

@vurusovs vurusovs requested review from a team and asomsiko July 27, 2020 10:13
@vurusovs vurusovs force-pushed the vu/stress_enable_omz_info_dumper branch 6 times, most recently from d4561c0 to 262a807 Compare July 28, 2020 11:26
Comment thread tests/stress_tests/scripts/get_testdata.py
Comment thread tests/stress_tests/common/tests_utils.cpp
@vurusovs vurusovs force-pushed the vu/stress_enable_omz_info_dumper branch from 262a807 to de8cf10 Compare July 28, 2020 12:47
Comment thread tests/stress_tests/common/tests_utils.cpp Outdated
Comment thread tests/stress_tests/memcheck_tests/flags.h
Comment thread tests/stress_tests/scripts/run_memcheck.py Outdated
Comment thread tests/stress_tests/scripts/run_memcheck.py Outdated
Comment thread tests/stress_tests/scripts/run_memcheck.py Outdated
Comment thread tests/stress_tests/scripts/run_memcheck.py Outdated
@vurusovs vurusovs force-pushed the vu/stress_enable_omz_info_dumper branch from de8cf10 to 168b185 Compare July 28, 2020 16:52
@vurusovs vurusovs requested a review from asomsiko July 28, 2020 17:03
@vurusovs vurusovs force-pushed the vu/stress_enable_omz_info_dumper branch 3 times, most recently from 3048e9b to cb86490 Compare July 29, 2020 09:06
@vurusovs vurusovs force-pushed the vu/stress_enable_omz_info_dumper branch 2 times, most recently from 7ecd39c to d2cb1db Compare July 30, 2020 08:44
@vurusovs
Copy link
Copy Markdown
Contributor Author

vurusovs commented Jul 30, 2020

Think about config validators (for a manual run):

  1. Check config doesn't have duplicates (to prevent failure in C++ gtest part)
  2. Check required fields (such as "path" and "full_path") are presented (to prevent exception in C++ part)
  3. If unique_id (e.g. unique name in OMZ) will be added, check it's availability and uniqueness in scope of config
  4. If config contains records for OMZ models, check it's availability in OMZ

Comment thread tests/stress_tests/scripts/get_testdata.py
@vurusovs vurusovs force-pushed the vu/stress_enable_omz_info_dumper branch from d2cb1db to 79ee848 Compare July 30, 2020 12:10
@vurusovs vurusovs merged commit 25cadb6 into openvinotoolkit:master Jul 31, 2020
ilya-lavrenov pushed a commit to ilya-lavrenov/openvino that referenced this pull request Jul 31, 2020
…t#1485)

* [Stress] Support OMZ model_info.py in get_testdata.py

* [Stress] Copy IRs from OMZ models folder to IRs folder

* [Stress] Support modified configs in C++ tests

* [Stress] Deprecate support of --env_conf due refactoring of configs

* [Stress] Update configs:
1. Removed env configs due deprecation
2. Moved test configs to a new format

* [Stress] Extend MemCheck records with info from test config
RomanZm pushed a commit to RomanZm/openvino that referenced this pull request Aug 28, 2020
…t#1485)

* [Stress] Support OMZ model_info.py in get_testdata.py

* [Stress] Copy IRs from OMZ models folder to IRs folder

* [Stress] Support modified configs in C++ tests

* [Stress] Deprecate support of --env_conf due refactoring of configs

* [Stress] Update configs:
1. Removed env configs due deprecation
2. Moved test configs to a new format

* [Stress] Extend MemCheck records with info from test config
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants