Split unit test over multiple jobs in GitHub Actions, lowering the total run time#4985
Split unit test over multiple jobs in GitHub Actions, lowering the total run time#4985muglug merged 2 commits intovimeo:masterfrom
Conversation
ea9a1e2 to
45205e5
Compare
45205e5 to
a1a9533
Compare
…t run as a standalone Test
a1a9533 to
f83896c
Compare
|
Thanks! CircleCI is faster (and I was thinking of ditching phpunit in Github Actions) but this is a great solution |
|
Yeah, the current GA runners have low resources, while circle's have lots. |
…tal run time (vimeo#4985) * Split unit test over multiple jobs in GitHub Actions, lowering the total run time * rename FileManipulationTest to FileManipulationTestCase so it does not run as a standalone Test
|
|
||
| mkdir -p build/parallel/ build/phpunit/logs/ | ||
|
|
||
| find tests -name '*Test.php' | shuf --random-source=<(get_seeded_random) > build/tests_all |
There was a problem hiding this comment.
This doesn't sound right to me: this line shuffles the tests based on a seed that is unique for each spawned jobs.tests, because get_seeded_random gets called indeed inside each of them.
So every job shuffles them differently and this lead to some test being run twice or worse, not being run at all.
The random-source should be generated only once in the chuck-matrix job.
There was a problem hiding this comment.
the random source is consistent. the seed is always vimeo/psalm
you can verfy this yourself by running this multiple times
find tests -name '*Test.php' | shuf --random-source=<(openssl enc -aes-256-ctr -pass pass:"vimeo/psalm" -nosalt </dev/zero 2>/dev/null)There was a problem hiding this comment.
There was a problem hiding this comment.
Yeah, I confirm I always saw error in jobs to be consistent from one run to another
There was a problem hiding this comment.
the random source is consistent.
This is good news indeed.
I wonder why shuf is needed in the first place then 🤔
Maybe you know you have some long-running tests and other quick tests, and you want to spread them evenly across the jobs? It would be good to write a comment about the why, just to clear the things out
Hi Matt,
I noticed that the unit test took quite a while to run in GitHub Actions, with the last few runs taking over 20 minutes to complete.
This is quite a long feedback loop, so I've tried to speed up it up.
What I've done is:
paratest.::error::to make failing tests easy to spot.Started with 5 jobs with 5 parallel
phpunitprocesses each, but can tweak this easily over time.Runtime should be down to around 5 minutes, which is a much better feedback loop.
Thanks.