-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Closed
Labels
QA:p3QA:teamdev:phpunit-testsp2-highEscalation, on top of current planning, release blockerEscalation, on top of current planning, release blocker
Milestone
Description
See PR #34972 and #34857 for a demo of the behavior:
If I break tests/lib/legacy/AppTest.php (see first commit for dumb easy example), then the phpunit test runs on drone:
- fail when running with
phpdbgand collecting coverage (all the drone coverage build jobs correctly report the 6 purposely induced unit test fails and exit with non-zero status) e.g. https://drone.owncloud.com/owncloud/core/16325/29 and https://drone.owncloud.com/owncloud/core/16325/38 - fail when running "ordinary" php7.1 and
phpunit(also correctly reports the 6 purposely induced unit test fails... like the previous point) e.g. https://drone.owncloud.com/owncloud/core/16325/47 - pass when running "ordinary" php7.2 or 7.3 and
phpunit(the 6 purposely induced unit test fails are logged byphpunitbutphpunitexits with status 0) this is a bad thing
PR #34972 has a cut-down example of the offending behavior.
- the first commit purposely puts a fail into
tests/lib/legacy/AppTest.phpso that we expect to get 6 test fails. - the 2nd commit:
- cuts down the drone jobs so just a few sqlite phpunit jobs are run
- cuts down
tests/phpunit-autotest.xmlto not bother running unit tests that did not effect the result - cuts down
tests/apps.phpso it only runs tests from thefilesapp - cuts down the tests for the files app to just
apps/files/tests/Command/ScanTest.php - cuts down the content of
ScanTestto just a single simplified test case
Running this results in the above behavior - some PHP7.2 and PHP 7.3 unit test runs come out with a "pass" on drone when actually 6 tests failed.
PR #34857 also has:
- the 3rd commit moves
ScanTest.phpout of the way, and puts the otherCommandtests back. In drone job https://drone.owncloud.com/owncloud/core/16326 all the phpunit jobs correctly report failure of 6 unit tests.
That demonstrates that it is something to do with ScanTest that messes up phpunit on PHP 7.2 and 7.3
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
QA:p3QA:teamdev:phpunit-testsp2-highEscalation, on top of current planning, release blockerEscalation, on top of current planning, release blocker