bats: support parallel execution of tests#172
Merged
sublimino merged 5 commits intobats-core:masterfrom Mar 5, 2019
Merged
Conversation
74c9b1e to
a2cf056
Compare
This was referenced Feb 5, 2019
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Feb 6, 2019
Use the *not-yet-merged* parallelism support I've been working on in bats[1] in order to speed up our tests massively. This requires that we use my home project on OBS, but hopefully my patches will be merged eventually. [1]: bats-core/bats-core#172 Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Feb 6, 2019
Use the *not-yet-merged* parallelism support I've been working on in bats[1] in order to speed up our tests massively. This requires that we use my home project on OBS, but hopefully my patches will be merged eventually. [1]: bats-core/bats-core#172 Signed-off-by: Aleksa Sarai <asarai@suse.de>
0b61dbc to
3d3378c
Compare
On distributions like openSUSE, libexec doesn't really exist so we need to install bats in other places in the source tree. We previously had out-of-tree patches for this, but it looks like they weren't pushed upstream. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
It appears as though this test didn't actually test what it was supposed to in 25505bd ("Skip pretty formatting if the first line isn't a TAP plan"). It was supposed to test that bats-format-tap-stream would not try to format non-TAP data -- but instead it just showed that the preamble of test files was executed before printing the TAP header (something that is not going to be the case with the suite refactor). Fixes: 25505bd ("Skip pretty formatting if the first line isn't a TAP plan") Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
This allows us to remove most of the complicated handling between bats-exec-test and bats-exec-suite (which had a complicated interplay due to the fact they both supported multi-test execution). Instead, multi-test execution is entirely left to bats-exec-suite and bats-exec-test only handles a single test. This entirely removes the need for the magic output handling in bats-exec-test -- and opens the door to suite-wide parallelism in one fell swoop. The only downside is that bats-preprocess is run multiple times on the same source file, but a follow-up patch will improve the caching of pre-processed files such that they will only be processed once. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
3d3378c to
f5f2794
Compare
Most well-behaved bats tests (where they aren't writing to shared data between test executions) are able to be executed in parallel, but to date bats has always executed them in serial. This is incredibly wasteful for trivially parallelisable tests, and can result in fairly inflated test times for no good reason. Parallel exeuction is done suite-wide, so it makes no difference how the tests are structured. Unfortunately, because some suites might be implicitly serial (like bats' own test suite) we cannot make the default mode execute in parallel. In addition, there is currently no way of tagging that a *specific* test has to be run in serial (so that the rest can be run in parallel) -- but that can be done in future work. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Testing that parallel execution actually works is a bit difficult, so the most trivial way of doing it is to just have a bunch of sleep tests and make sure that they run faster than would be possible if they were run in serial. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
f5f2794 to
7e6dd94
Compare
Contributor
Author
|
Any updates? @sublimino? |
Member
Contributor
Author
|
Neat. I will send the randomised ordering PR next. |
2 tasks
|
This commit: 97975a0 Introduces Stacktrace for those living on the bleeding edge running from source: /usr/bin/readlink: illegal option -- f
usage: readlink [-n] [file ...]
|
Contributor
Author
|
Yes. I was considering re-implementing |
Contributor
Author
|
Scratch that, I just figured out I can implement it with a really hacky use of |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This allows for tests to be run in parallel, by using GNU parallel (if
available). It's a fairly trivial extension (though it required a fair
amount of work in order for it to be usable).
Most well-behaved bats tests (where they aren't writing to shared data
between test executions) are able to be executed in parallel, but to
date bats has always executed them in serial. This is incredibly
wasteful for trivially parallelisable tests, and can result in fairly
inflated test times for no good reason. Parallel exeuction is done
suite-wide, so it makes no difference how the tests are structured.
Unfortunately, because some suites might be implicitly serial (like
bats' own test suite) we cannot make the default mode execute in
parallel. In addition, there is currently no way of tagging that a
specific test has to be run in serial (so that the rest can be run in
parallel) -- but that can be done in future work.
Fixes #171
Signed-off-by: Aleksa Sarai cyphar@cyphar.com