Expected Behavior
The Test task should have configuration that makes it possible to order test classes pseudo randomly.
test {
pseudoRandomClassOrder()
}
A the randomization seed would be configurable via a test.seed property.
The test seed would need to be logged on the console, part of the output, etc.
Current Behavior
There is no way to influence or control in which order test classes are executed.
There is a precedent for ordering based on previous failures, but that can not be influenced by build script authors.
Context
The Elasticsearch and Lucene tests make heavy use of randomized testing in general.
See: http://labs.carrotsearch.com/randomizedtesting.html.
This includes randomization of test class, test method order, random parameters, exceptions etc.
We are currently running tests with carrotsearch randomizetesting ant runner ( which we call from Gradle ) and we would like to move our testing infrastructure to Gradle replacing the ant runner, but we are missing this functionality. Randomizing test class order will help uncover any unwanted dependency among test classes and it is something the teams feel strongly about.
I already looked at how the Test task is implemented and would like to contribute an implementation for it, but would like to have a conversation about it first.
Expected Behavior
The
Testtask should have configuration that makes it possible to order test classes pseudo randomly.A the randomization seed would be configurable via a
test.seedproperty.The test seed would need to be logged on the console, part of the output, etc.
Current Behavior
There is no way to influence or control in which order test classes are executed.
There is a precedent for ordering based on previous failures, but that can not be influenced by build script authors.
Context
The Elasticsearch and Lucene tests make heavy use of randomized testing in general.
See: http://labs.carrotsearch.com/randomizedtesting.html.
This includes randomization of test class, test method order, random parameters, exceptions etc.
We are currently running tests with carrotsearch randomizetesting ant runner ( which we call from Gradle ) and we would like to move our testing infrastructure to Gradle replacing the ant runner, but we are missing this functionality. Randomizing test class order will help uncover any unwanted dependency among test classes and it is something the teams feel strongly about.
I already looked at how the
Testtask is implemented and would like to contribute an implementation for it, but would like to have a conversation about it first.