Update: Meanwhile I found the time to prepare an open source library that provides you with a ready-to-use ParameterizedSuite! See my Github repo for detailed instructions.

---

When you come across this blog entry you certainly know the features of JUnit’s Suite and Parameterized classes: Suite lets you execute multiple tests as a test suite (i.e. as a group), Parameterized allows to define parameters for a single test case and execute each of the test’s methods with each tuple of the parameter set.

Now obvious questions are: How do I define parameters for a test suite? Why is there no ParameterizedSuite class in JUnit? How do I implement a parameterised suite by myself?

What I found funny when I was searching the Internet was that many questions seemed to came up in the context of Selenium. And this was exactly the reason why I needed to parameterise a test suite! My parameters would be the different browsers where I wanted to run the same tests. Seems as if developers and QA testers around the world stumble across the same problems but nobody offers a solution. It’s time for change! 😉

Why is there no out-of-the-box implementation?

I think, you don’t find a ready-to-use implementation of a ParameterizedSuite because:

1. The implementation of such a class is far from trivial. It is even hindered by JUnit’s “closed code” that doesn’t really allow extension and reuse. There are also some complex internal concepts in JUnit a developer normally doesn’t touch (e.g. Runners and Statements), but these are key to implement an own ParameterizedSuite.
2. Parameterized and Suite follow slightly different concepts of JUnit. Parameters are applied to instances of test classes, whereas during the processing of a Suite the class isn’t instantiated at all.
3. There can’t be a single one-size-fits-all implementation of ParameterizedSuite since there are quite a few design questions regarding the runtime behaviour of the class. These implementation details depend on your use case. I tried to summarise some of these questions in the next chapter.

The good news is: I was able to implement a ParameterizedSuite for my own use case.
The bad news is: It will take me some more time to share the code with you.

However, you certainly have other requirements anyway, so you don’t get around implementing your own ParameterizedSuite. In this blog post I’d like to give valuable insight for this implementation task.

For implementing your own ParameterizedSuite you can certainly copy a lot of code from Suite and Parameterized. The bigger challenge is to understand the internal concepts of JUnit, the runtime lifecycle of a test and how to combine both classes.

One last remark: You should double-check if you have alternatives to implementing your own ParameterizedSuite! It won’t only be very tedious and time consuming, but it could also be hard to maintain in the future when your requirements change.
One alternative I can think of is to parameterise the build job on your Continuous Integration machine and start a job for every parameter tuple you have. This is much easier to implement and maintain! You’ve been warned 😉

Design questions

Before you start your implementation, you should put some thought in your requirements (Well, that’s always true, isn’t it?). Try to answer the following:

1. Where should the parameters (the @Parameters annotation) be defined?
   If you define parameters only at the Suite level, how should a single, parameterised test case behave if you start it standalone, e.g. from your IDE? Should it have own parameters as a fallback?
2. Will you implement a hierarchy of suites, e.g. a suite running suites running tests?
   Do these sub-suites also have parameters? Do they add to the superior parameters, or overwrite them? Or do they act as a fallback like in the previous question?
3. How to transfer parameters top down?
   When a suite provides parameters, how does a parameterised test case access them?
4. How do you want to execute the cross-product of test cases and parameters? Execute all tests per parameter first or all parameters per test before going over to the next test?
5. Do you want support for JUnit’s annotation based features like @Before, @BeforeClass, @After, @AfterClass, @Rule, @ClassRule, @TestName, @ExpectedException, @Timeout etc. in your ParameterizedSuites? Do you want to evaluate these once for your suite or for every parameter?

Internal concepts in JUnit

There are a few concepts you should be aware of.

Runners

Suite and Parameterized are linked to your test class via the @RunWith annotation. Both are Runners and replace the BlockJUnit4ClassRunner, which is the default Runner and used to run single test classes. All three extend the ParentRunner class, which means that in practice in your IDE they appear as a node in your hierarchy of tests that has leaves or other nodes (so called children). The children of a BlockJUnit4ClassRunner are the test class’ methods. The children of a Suite are the classes from the @SuiteClasses annotation.

Statements

A runner builds a chain of statements in its run() method and executes it directly afterwards. This chain of statements is the actual execution order of a test’s features (See design question #5).

For BlockJUnit4ClassRunner the smallest piece of statement is the execution of a test method. Statements triggered by annotations like @Before or @AfterClass encapsulate this statement and execute own logic prior or after that.

A Suite maintains a list of child runners and generates a statement, which on execution calls run() for each of these children sequentially. This statement is decorated with evaluations of @BeforeClass, @AfterClass and @ClassRule, which annotate static methods and fields.

Instantiation of the test class

BlockJUnit4ClassRunner creates an instance of the test class for every test method. This happens in the methodBlock() method and is necessary to run the test methods. The annotations @Before and @After are evaluated in that method. For that reason those annotations are linked to non-static methods, while @BeforeClass and @AfterClass are not. The latter annotations are inherited from ParentRunner (and thus shared with Suite also), which doesn’t operate on test objects at all, but only on the not instantiated classes.

For this reason you’re not allowed to use non-static fields and methods for JUnit features in Suites.

Implementation details

Storing the parameters in a context

To pass the singularised parameters top-down, I decided to introduce a singleton ParameterContext. It holds the parameters for one test execution (i.e. Object[]). Later on the JUnit framework will pass this Object array via reflection as method arguments to the test class’ constructor. So if your constructor takes two arguments, your @Parameters method should produce Object[][2] and ParameterContext will store Object[2].

The ParameterContext is set by the ParameterizedSuite while iterating the set of parameters.

The test cases look up the parameters in the ParameterContext. If the context hasn’t been set (i.e. if you start the test case standalone, not within a suite), they have a set of fall back parameters that becomes active.

Depending on your decision on design question #2, non-top-level suites combine the behaviour of both parameterized suites and test cases: If no context is set, they set it. Otherwise they might extend the context or take no action at all.

Other solution designs are possible as well, but could be far more invasive to existing framework classes, e.g. by passing parameters to the RunnerBuilders.

ParameterizedSuite

To have greatest flexibility, my implementation extends ParentRunner directly and not Suite or Parameterized.

To implement a blended behaviour of both, you will certainly have to copy a dozen helper methods from these two classes, because most of them are private (as of JUnit 4.12).

• Start with the constructor! Suite has a bunch of them, but you actually only need a 2-arg-constructor. Call super(Class) from there.

    public ParameterizedSuite(Class<?> suiteTestClass, RunnerBuilder runnerBuilder) throws Throwable {
super(suiteTestClass);

•  As in Suite, you already construct all your child runners in the constructor, so you have them available when ParentRunner asks you for them through getChildren().
• If you have a multi-level hierarchy of ParameterizedSuites (i.e. a ParameterizedSuite of ParameterizedSuites of test cases) you will have to distinct two cases:
  1. Is this ParameterizedSuite the top-level suite? Then you have to singularise the parameters!
  1. If this is not, you don’t have to care about the parameters, but you might want to execute the annotation-based features (design question #5)
• To determine the cases, you could check if the ParameterContext is already set.

Case: Top-level

• Determine the set of parameters (i.e. Object[][] !). For each single parameter (Object[]) instantiate a new SingleParameterSuite() and pass that single parameter to it.
• If the test class has annotation-based features you certainly want to execute these in the SingleParameterSuites but skip those now. So you have to overwrite ParentRunner.classBlock()  and only call childrenInvoker().

 Case: Non-top-level

• “Just do, what a normal Suite would do”
• In classBlock() be sure to run the annotation processing. Here you will have to “reuse” (i.e. copy) a lot of code.

 SingleParameterSuite

Here you can extend Suite. But why do we need a specialised class for this at all?
First of all you should overwrite ParentRunner.getName() to return a unique name for every different parameter (like I did in the figure above). If you don’t and return a static name, at least in Eclipse only the first suite will be executed and the others with the same name will be skipped.

Here I also extended the annotation-processing of Suite: I instantiated the test class (lots of copied code again) and processed annotations like @Before, which are linked to non-static methods.

You should pass a reference of the test class (annotated with ParameterizedSuite) to the constructor SingleParameterSuite to:

1. to pass it as parameter “klass” to the super-constructor Suite(RunnerBuilder, Class<?>, Class<?>[]). This way the test execution logic will work as if we would directly run on the test class and not on the virtual sub-suite.
2. have direct access to its annotations methods.

Closing words

Although it will take me some more time to prepare the code for you, I hope this guide makes your life a bit easier already.

If you have questions, leave a comment! 🙂