Skip to content

phpunit on PHP 7.2 or 7.3 can exit with 0 status when there are test fails #34858

@phil-davis

Description

@phil-davis

See PR #34972 and #34857 for a demo of the behavior:

If I break tests/lib/legacy/AppTest.php (see first commit for dumb easy example), then the phpunit test runs on drone:

PR #34972 has a cut-down example of the offending behavior.

  • the first commit purposely puts a fail into tests/lib/legacy/AppTest.php so that we expect to get 6 test fails.
  • the 2nd commit:
    • cuts down the drone jobs so just a few sqlite phpunit jobs are run
    • cuts down tests/phpunit-autotest.xml to not bother running unit tests that did not effect the result
    • cuts down tests/apps.php so it only runs tests from the files app
    • cuts down the tests for the files app to just apps/files/tests/Command/ScanTest.php
    • cuts down the content of ScanTest to just a single simplified test case

Running this results in the above behavior - some PHP7.2 and PHP 7.3 unit test runs come out with a "pass" on drone when actually 6 tests failed.

PR #34857 also has:

That demonstrates that it is something to do with ScanTest that messes up phpunit on PHP 7.2 and 7.3

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions