Having the @unroll annotation on a test which has a groovy compilation error in where clause results in the test being excluded and not marked as failed. Really dangerous! 🔺

Bug appears (ie as if test never existed):

• IntelliJ: Running the test by package, even if only one class with one test in that package.
• IntelliJ: Running all unit tests (probably same as above)
• Gradle: Always, even if you -Ptest.include only a single class/test

Bug does not appear (ie test is marked as failed):

• IntelliJ: Running the particular test (right-clicking green arrow on right -> run)
• IntelliJ: Running the test by running all tests in class (by green arrow as above)
• IntelliJ: Removing the @unroll

Imagine, you've done some refactoring and introduced compilation errors in the tests.
You run all your groovy tests in IntelliJ and all is false-positively green.
You run gradle tests and all is green. You run your Jenkins pipeline and all is green.

Update: I found that this bug only appears with junit-jupiter all versions
Perhaps the exception thrown by spock is swallowed silently by jupiter

How to reproduce

Add a java class that you want to test, say User.java in a package.
Create a test for User.java annotated with @unroll
Initialise the User in the test wrongly by constructor in where clause
Run the tests by package
User.java

public class User {

    public String id;

    public User(String id) {
        this.id = id;
    }
}

UserTest.groovy

import spock.lang.Specification
import spock.lang.Unroll

class UserTest extends Specification {

    @Unroll
    def "testRevealingUnrollBug"() {
        when:
        def nothing = "here"

        then:
        user.id == user_id

        where:
        user                    || user_id
        new User("id")          || "id"
        new User("id", "extra") || "id"
    }

}

Java/JDK

openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)
(Tried java jdk1.8.0_162 as well)

Groovy version

2.5.6

Gradle

Gradle 4.10.3

IDE

IntelliJ 2018.3.4

Spock version

spock-core:1.2-groovy-2.5
spock-spring:1.2-groovy-2.5