Run test sets in different Travis jobs #632
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t and sit there forever
…sitting at the end waiting for the final job to finish. Makes better use of the parallel job-runners.
hugovk
added a commit
to hugovk/Pillow
that referenced
this pull request
Apr 18, 2014
It's always the slowest and we should give it a head start so we're not waiting for it to finish at the end. It means we're making the most use of our parallel job-runners for the quicker jobs. See also python-pillow#632.
Merged
Member
|
Closing for now, I don't think it was a successful experiment. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Experiment
This wasn't as successful as hoped, but there was some improvement.
Hypothesis
Running tests in separate jobs will speed up the Travis CI builds.
Results
Initially, build slowed down. Probably due to the overhead of installing dependencies and building Pillow at the start. If that bit can be sped up (possibly via caching) then this approach may help.
Changing Python order sped up build a bit (total waiting time), but uses more CPU time.
Details
Using an environment variable, we can run different test sets in different build jobs.
Reference:
http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-your-builds-across-virtual-machines
Currently we have two main tests sets: self-test, and the main tests.
Using an environment variable, Travis will run a different job for each value.
Then instead of running both in
script:just run one:
This changes build times from:
26:32 https://travis-ci.org/hugovk/Pillow/builds/22906842
24:25 https://travis-ci.org/hugovk/Pillow/builds/23269205
to:
36:52 https://travis-ci.org/hugovk/Pillow/builds/23268580
33:26 https://travis-ci.org/hugovk/Pillow/builds/23269260
Hmm. That may be partly because the pypy build is 2-3x slower and run last. We're just sitting around waiting for it to finish on the last job-runner. If we run pypy first, we can make better use of the parallel job-runners:
39:48 https://travis-ci.org/hugovk/Pillow/builds/23269381
35:48 https://travis-ci.org/hugovk/Pillow/builds/23269381
Still bad. Most of this is probably from the dependency install and Pillow build overhead at the start of each job.
But hang on! These times are the total CPU time used across all jobs. The actual time from start to finish is much less.
Here's the actual elapsed start-to-end times:
Before: ~12m (about the time of the slowest pypy job)
With env: ~13m
pypy at end: ~10m (about the time of the slowest pypp job)
It uses more CPU time, but us humans get the build results in a bit less time.
Conclusion
Should we include this change? Perhaps it'd be better to until #540 has been finalised and test again, as the dependency/build overhead will remain.
At the very least, pypy should come first as that's the slowest and needs a head start. I've made another PR for that (#633).