Make hot compilation 2x faster by properly reusing classloaders#393
Merged
lihaoyi merged 1 commit intocom-lihaoyi:masterfrom Jul 27, 2018
Merged
Make hot compilation 2x faster by properly reusing classloaders#393lihaoyi merged 1 commit intocom-lihaoyi:masterfrom
lihaoyi merged 1 commit intocom-lihaoyi:masterfrom
Conversation
So far, Mill was caching ScalaInstance which contains a classloader but this is not enough: Zinc creates its own classloader by combining the ScalaInstance classloader with the path to the compiler-bridge. Zinc takes care of caching this classloader in each instance of ScalaCompiler. We can take advantage of this by caching an instance of Compilers since it contains everything we want to cache (ScalaCompiler and ScalaInstance). This significantly reduces the amount of classloading that happens at each compilation step, as measured by running: export _JAVA_OPTIONS="-XX:+UnlockDiagnosticVMOptions -XX:+TraceClassLoading -XX:+TraceClassUnloading" mill -i foo.compile Then looking at the output of `out/mill-worker-1/logs` while adding a new line in a source file in `foo` and running `mill -i foo.compile` again. The speedup is going to depend on the project, but I measured a ~2x improvement when running on the Mill build `time(core.compile())` and `rm -rf out/core/compile/` in a loop until results stabilized. See also the results that were obtained when a very similar issue was fixed in sbt itself: sbt/sbt#2754
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
So far, Mill was caching ScalaInstance which contains a classloader but
this is not enough: Zinc creates its own classloader by combining the
ScalaInstance classloader with the path to the compiler-bridge. Zinc
takes care of caching this classloader in each instance of
ScalaCompiler.
We can take advantage of this by caching an instance of Compilers since
it contains everything we want to cache (ScalaCompiler and
ScalaInstance).
This significantly reduces the amount of classloading that happens
at each compilation step, as measured by running:
Then looking at the output of
out/mill-worker-1/logswhile adding anew line in a source file in
fooand runningmill -i foo.compileagain.The speedup is going to depend on the project, but I measured a ~2x
improvement when running on the Mill build
time(core.compile())andrm -rf out/core/compile/in a loop until results stabilized. See alsothe results that were obtained when a very similar issue was fixed in
sbt itself: sbt/sbt#2754