OPENCV_PYTEST_FILTER fails to filter as expected due to a logic bug. It runs too many tests due to an errant wildcard.
For example, OPENCV_PYTEST_FILTER=test_cudaa runs 4 tests. I expect it to run 0 tests.
System information (version)
- OpenCV => 4.5.1
- Operating System / Platform => Microsoft Windows [Version 10.0.19042.746]
- Compiler => VS2019 v16.8.4 Community
Detailed description
Too many tests are run due to an errant wildcard in the test.py python test runner. There is also a minor output bug.
|
tests_pattern = os.environ.get('OPENCV_PYTEST_FILTER', 'test_') + '*.py' |
|
if tests_pattern != 'test_*py': |
|
print('Tests filter: {}'.format(tests_pattern)) |
Line 37 has an errant wildcard. It should not be present. Instead, the user should include a wildcard in OPENCV_PYTEST_FILTER if they want such behavior. Line 38 has the minor display bug. The coder intended to check for the default case but failed to include the .
It is easily possible to fix the python code to support both the intended default case and expectant wildcard behavior. Then OPENCV_PYTEST_FILTER would match the wildcard behavior of --gtest_filter. FYI: to achieve full test case name filtering, Python 3.7 is needed and use of unittest.testloader.testNamePatterns.
Steps to reproduce
- Setup Windows and Python 2.7
- Build a release version of OpenCV 4.5.1
- set env var
OPENCV_PYTEST_FILTER=test_cudaa
c:\python27\python opencv\modules\ts\misc\run.py . -a -t python2
Result
4 test found and 4 test run
...
Discovering python tests from: C:\repos-nobackup\opencv_contrib\modules\cudaarithm\misc\python\test
found 4 tests
...
test_arithmetic (test_cudaarithm.cudaarithm_test) ... ok
test_convolution (test_cudaarithm.cudaarithm_test) ... ERROR
test_cudaarithm (test_cudaarithm.cudaarithm_test) ... ok
test_logical (test_cudaarithm.cudaarithm_test) ... ok
Expected
No tests found and no tests run.
Issue submission checklist
OPENCV_PYTEST_FILTER fails to filter as expected due to a logic bug. It runs too many tests due to an errant wildcard.
For example,
OPENCV_PYTEST_FILTER=test_cudaaruns 4 tests. I expect it to run 0 tests.System information (version)
Detailed description
Too many tests are run due to an errant wildcard in the
test.pypython test runner. There is also a minor output bug.opencv/modules/python/test/test.py
Lines 37 to 39 in 1363496
Line 37 has an errant wildcard. It should not be present. Instead, the user should include a wildcard in
OPENCV_PYTEST_FILTERif they want such behavior. Line 38 has the minor display bug. The coder intended to check for the default case but failed to include the.It is easily possible to fix the python code to support both the intended default case and expectant wildcard behavior. Then OPENCV_PYTEST_FILTER would match the wildcard behavior of
--gtest_filter. FYI: to achieve full test case name filtering, Python 3.7 is needed and use ofunittest.testloader.testNamePatterns.Steps to reproduce
OPENCV_PYTEST_FILTER=test_cudaac:\python27\python opencv\modules\ts\misc\run.py . -a -t python2Result
4 test found and 4 test run
Expected
No tests found and no tests run.
Issue submission checklist
forum.opencv.org, Stack Overflow, etc and have not found solution