Full tests (npm run tests:full) currently take over an hour to run on a relatively powerful machine (Core i7 MacBook Pro) and there are some other issues related to tests as we reviewed them with @JanVoracek today.

UPDATE: In the end, this issue is a major update of the dockerized dev / testing environment last updated in #1329.

Highlights

• Full tests now run much faster thanks to volumes instead of host-mounted folders. On my machine, the time dropped from 1 hour 15 minutes to just 11 minutes. Use named volumes for tests #1396
• Permissions between containers (Debian vs. Alpine) have been fixed. Run CLI image as UID 33 (Debian's "www-data") #1397
• New run-tests.ts script that orchestrates how tests run. For example, it decides when to start a WordPress stack (if at all), cleans up after itself, etc. New run-tests.ts, making test runs more reliable #1401
• The main testing scripts is now just npm run tests (the :full suffix has been dropped).
  • Also, all the tests:custom scripts have been dropped as run-tests.ts now accepts parameters, for example:
    npm run tests -- --testsuite Unit --filter CursorTest
• Docker images are now fixed using their digests. No more surprises if upstream images change. Use digests for Docker images #1398
• Some effort has been put into testing and documenting how test work on Docker Toolbox. Docker Toolbox on Windows – notes #1403

During the work, I've also encountered several other things that are noted in the "Deferred items" section below.

TODO list

• Split testing documentation from Dev Setup, it's quite long on its own. Docs: separate page on Testing #1392
  • ... and smaller related content changes. Minor docs updates around dev setup and testing #1393
• Split docker-compose services into two files, docker-compose.yml and docker-compose.tests.yml. Separate docker-compose files for dev and tests #1395
• ⭐ Permissions: Run CLI image as UID 33 (Debian's "www-data") Run CLI image as UID 33 (Debian's "www-data") #1397
• ⭐ Volumes: use named volumes (i.e., don't map to host OS) & have some way to inspect the data later. Use named volumes for tests #1396
• Use Docker image digests instead of tags. Use digests for Docker images #1398
  • They are uglier, e.g., busybox:latest becomes busybox@sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d, but it's the only way to make runs predictable (docker-compose lacks a concept of a lock file).
  • This has already been considered in Dev setup updates – spring 2018 #1329 but we eventually didn't go for it.
  • docker-compose config --resolve-image-digests can help with that.
  • Awesome post about the problem and solutions: Overcoming Docker’s mutable image tags
    • (BTW, it's by Renovate, a tool that we should consider in the future anyway to keep dependencies up to date.)
• Stop containers after the tests finish running. Stop containers after tests #1399 (package.json impl), then New run-tests.ts, making test runs more reliable #1401 (run-tests.ts impl)
• ⭐ Introduce run-tests.ts and make test runs more reliable. Make tests pass on the updated testing infrastructure #1400 (PR New run-tests.ts, making test runs more reliable #1401)
• Verify how things work in Docker Toolbox. Docker Toolbox on Windows – notes #1403
• Possibly remove "test:custom" npm scripts, just document how to do it. Done as part of New run-tests.ts, making test runs more reliable #1401.
• Explore using Samba to speed up files inspection, see Improve Dockerized testing infrastructure #1389 (comment). Explored but not successfully, see Small updates of exploring the test WP site after tests have run #1411.
• Document how to stop tests on Mac – Ctrl+C doesn't work, I have to open another terminal, type docker ps, copy the container ID and do docker stop <id>. Resolved as part of New run-tests.ts, making test runs more reliable #1401 – Ctrl+C now kills the Node.js script.
  • Ctrl-C/SIGINT has erratic behaviour on "docker-compose up" docker/compose#3317
  • [1.7] CTRL+C fails to stop containers when running compose in the foreground docker/compose#3347
  • Might be an issue with wordpress:cli entrypoint and how it handles signals.
• ⭐ Make npm run tests pass. Make tests pass on the updated testing infrastructure #1400

Deferred items

These should be done separately in future PRs.

• Rename images to something like versionpress/wordpress and versionpress/cli, i.e., make "CLI" part of the image name so that it's clear when digests are used.
• Only enable Xdebug for ...:debug scripts.
  • Right now, it's always enabled and just not configured with the right idekey in non-debug scripts which makes things slower. Here and here.
  • Update Xdebug to 2.7.0. (Let's still stay on PHP 7.2 for tests for now.)
• Make PHP 7.2 the minimum required version? (We don't have an issue for it yet, will create it.)
• The WP-CLI installation into the WP image looks a bit weird, review it.
• Try updating PHPUnit (I remember it wasn't so easy due to supported PHP versions or something but it's worth looking again).
• Don't base our CLI image on wordpress:cli, it does a lot more than we need.
• Explore re-running failed tests: Implement "previous failures first" mode sebastianbergmann/phpunit#46
• Close Fix remaining tests under Docker, post-beta #1284 (Fix remaining tests under Docker, post-beta) and port those to a new issue
• Review backlog from the "Noted for future work" section in Dev setup updates – spring 2018 #1329
• Another big one is WP-CLI / Selenium worker switching – revised approach #1258 (WP-CLI / Selenium worker switching). That issue is a bit old but overall, we should still be able to run all tests with both workers, or pick a worker with a flag like npm run tests --worker=selenium. Things like a worker or a specific test site to use shouldn't be hardcoded in test-config.yml.
  • Related: How to parametrize PHPUnit tests.