Bing Xu opened SUREFIRE-2284 and commented

Hi again from Atlassian

Bug description
If a Test fails (either fails (via an assertion or directly calling fail) or throws an {}error{}) in the @BeforeAll method, the interaction with the rerunFailingTestsCount property doesn't work properly. Even if the test succeeds in a re-run, the Surefire Mojo will throw an exception and fail the build.

Scenario/Steps to reproduce
A test fails or throws an exception (error) in the @BeforeAll method on an initial run (Run #1).
And then succeeds in a subsequent execution (Run #2).

Current behaviour
The surefire plugin will throw an exception because of the failure/error in the initial run of the test class.

Expected behaviour:
The build should pass if a subsequent rerun of the test and no exception should be thrown.

Root cause
The root cause seems to be because surefire has logic to merge the result of multiple runs of the same test inside the following class:

maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/DefaultReporterFactory.java

This class does so by matching test method stats with the same testClassMethodName. This does not work when the test fails in @BeforeAll because the test method name is null, so it appears to surefire as if this is a separate method altogether that has never succeeded. 

This is the problematic part of the code:

Map<String, List<TestMethodStats>> mergedTestHistoryResult = new HashMap<>();
// Merge all the stats for tests from listeners
for (TestSetRunListener listener : listeners) {
    for (TestMethodStats methodStats : listener.getTestMethodStats()) {
        List<TestMethodStats> currentMethodStats =
                mergedTestHistoryResult.get(methodStats.getTestClassMethodName());
        if (currentMethodStats == null) {
            currentMethodStats = new ArrayList<>();
            currentMethodStats.add(methodStats);
            mergedTestHistoryResult.put(methodStats.getTestClassMethodName(), currentMethodStats);
        } else {
            currentMethodStats.add(methodStats);
        }
    }
} 

3 votes, 4 watchers