Make phpunit fail if a --filter does not execute any tests#2924
Make phpunit fail if a --filter does not execute any tests#2924ondrejmirtes merged 3 commits intophpstan:1.10.xfrom
Conversation
|
ohh interessting.. we had a test-case, which only runs in PHP 8.3+ phpstan-src/tests/PHPStan/Levels/LevelsIntegrationTest.php Lines 45 to 47 in f11add3 and this case did never run in CI, because level tests only execute in PHP 8.1. that was a problem even before the test-parallelizing PR landed. this turned into an error in the first commit of this PR, because the |
.github/workflows/tests.yml
Outdated
There was a problem hiding this comment.
should we run the level tests in all php versions? or maybe lowest and highest supported version?
There was a problem hiding this comment.
No it's fine to run it just once.
There was a problem hiding this comment.
E.g. there's nothing that changes for levels on different PHP versions. If some rule changes behaviour based on PHP version we test in that rule's test.
.github/workflows/tests.yml
Outdated
There was a problem hiding this comment.
I'd move it to phpunit.xml please - we always want to check this.
Also there are other interesting options that we might add:
executionOrder="random"
failOnWarning="true"
failOnRisky="true"
failOnEmptyTestSuite="true"
beStrictAboutOutputDuringTests="true"
From https://backendtea.com/post/phpunit-beyond-basic-config/#tldr
|
Any idea why this does not execute anything? https://github.com/phpstan/phpstan-src/actions/runs/7948332751/job/21698238578 The other executions work. |
yes, see #2924 (comment) |
|
Looks like the random execution order breaks something :) Please look into tests that set environment variables, and if each test sets up env variables it needs, and also if they are properly unset in tearDown. |
| /** | ||
| * @runTestsInSeparateProcesses | ||
| */ | ||
| class TableErrorFormatterTest extends ErrorFormatterTestCase |
There was a problem hiding this comment.
I just think that this test was influenced by some other test that set an env variable.
This diff is a workaround, not a solution to the problem.
There was a problem hiding this comment.
yeah, I am still in the process of finding a proper solution. the tests don't fail locally, so I push and look at CI results
|
I think we are good to go |
|
Awesome! |
we don't know yet how to properly escape the
--filtervalue, but we can use--fail-on-empty-test-suiteto get a non-zero exit code, so we at least get a failling github action error in case our filter is bogusfound in sebastianbergmann/phpunit#4314