In order to speed up compilation on CI I think we should stop forking the java compiler. My tests on a 6 core machine show that this could give us a 2x speed-up.

Not forking for compilation:

./gradlew clean compileJava compileTestJava
5m 11s

./gradlew clean compileJava compileTestJava --parallel
2m 59s

Forking for compilation:

./gradlew clean compileJava compileTestJava
1m 55s

./gradlew clean compileJava compileTestJava --parallel
1m 26s

The code where we do forking has the comment

// we fork because compiling lots of different classes in a shared jvm can eventually trigger GC overhead limitations

but I have not seen such issues. If we run ./gradlew clean compileJava compileTestJava --parallel in a separate build step (for example in elasticsearch-ci), then this should pose no issue. We could also make this a configurable thing (-Pfork.compiler=false) so the fast mode can be enabled for people that want faster compilation.