Skip to content

Experimental support for outline type checking#8077

Merged
retronym merged 7 commits intoscala:2.12.xfrom
retronym:topic/outline-typing
May 26, 2019
Merged

Experimental support for outline type checking#8077
retronym merged 7 commits intoscala:2.12.xfrom
retronym:topic/outline-typing

Conversation

@retronym
Copy link
Member

@retronym retronym commented May 22, 2019 •
edited
Loading

A new mode for PipelineMain that can shorten the critical path of a build and extract more parallelism from the post-typer compiler phases (at the cost of instantiating multiple instances of Global).

First, we compile all sources with -Youtline -Ystop-after:pickler. The first option disables typechecking of the RHS of definitions that have ascribed types.

The RHS is not skipped when it contains a super call that might need a super accessor. (hat tip to @smarter for finding this corner case in dotty). Are there more like it that we haven't identified?

The pickles are then exported and made available to downstream compilation, as is also the case with the strategy=Pipeline. The sources for the current projects are then compiled in parallel in chunks, also using the exported pickles on the classpath.

Testing this out guardian-frontend. In the past I've experimented with the full multi-module build, but in this case I want to focus on the common project that is the bottleneck the build.

Pipeline + Outline

$ mkdir -p /tmp/pickle-cache && ./build/pack/bin/scala -J-Dsun.reflect.inflationThreshold=0  -J-Xmx4G -Dscala.pipeline.cache.name.table=true -Dscalac.filebasedcache.defer.close.ms=30000 -Dscala.pipeline.picklecache=/tmp/pickle-cache -Dscala.pipeline.group.size=128  -Dscala.pipeline.strategy=outlinetypepipeline -Dscala.pipeline.strip.external.classpath=true -Dscala.pipeline.cache.macro.classloader=true -Dscala.pipeline.use.jar=true scala.tools.nsc.PipelineMainTest /Users/jz/code/guardian-frontend/common/target/compile.args
...
====== ITERATION 20=======
parallelism = 8, strategy = OutlineTypePipeline
Exported external classpath in 4 ms
Wrote project dependency graph to: /Users/jz/code/scala/projects.dot
/Users/jz/code/guardian-frontend/common:compile: scalac outline: start
/Users/jz/code/guardian-frontend/common:compile: scalac outline: done 1378 ms
/Users/jz/code/guardian-frontend/common:compile: scalac (1/4): start
/Users/jz/code/guardian-frontend/common:compile: scalac (2/4): start
/Users/jz/code/guardian-frontend/common:compile: scalac (4/4): start
/Users/jz/code/guardian-frontend/common:compile: scalac (3/4): start
/Users/jz/code/guardian-frontend/common:compile: scalac (4/4): done 1729 ms
/Users/jz/code/guardian-frontend/common:compile: scalac (3/4): done 1898 ms
/Users/jz/code/guardian-frontend/common:compile: scalac (1/4): done 3074 ms
/Users/jz/code/guardian-frontend/common:compile: scalac (2/4): done 4698 ms
PROGRESS: 6 / 6
 Wall Clock: 6077 ms
Chrome trace written to /Users/jz/code/scala/build-OutlineTypePipeline-20.trace

Pipeline

$ ... -Dscala.pipeline.strategy=pipeline ...
====== ITERATION 20=======
parallelism = 8, strategy = Pipeline
Exported external classpath in 4 ms
Wrote project dependency graph to: /Users/jz/code/scala/projects.dot
/Users/jz/code/guardian-frontend/common:compile: scalac: start
/Users/jz/code/guardian-frontend/common:compile: scalac outline: done 4232 ms
/Users/jz/code/guardian-frontend/common:compile: scalac: exported pickles 52 ms
/Users/jz/code/guardian-frontend/common:compile: scalac: done 4523 ms
PROGRESS: 3 / 3
 Wall Clock: 8807 ms
Chrome trace written to /Users/jz/code/scala/build-Pipeline-20.trace

Traditional

$ ... -Dscala.pipeline.strategy=traditional ...
====== ITERATION 20=======
parallelism = 8, strategy = Traditional
Wrote project dependency graph to: /Users/jz/code/scala/projects.dot
/Users/jz/code/guardian-frontend/common:compile: scalac (1/1): start
/Users/jz/code/guardian-frontend/common:compile: scalac (1/1): done 9343 ms
PROGRESS: 3 / 3
Wall Clock: 9409 ms
Chrome trace written to /Users/jz/code/scala/build-Traditional-20.trace

Summary

  • The main benefit of the new strategy here is reducing the "time-to-pickles" down to 1.4s (vs 9.3 for traditional vs 4.2 for non-outline pipeline)
  • The wall clock time is also reduced somewhat (6.1s vs 8.8s), which may be interesting in itself when enough cores are available. There will be rapidly diminishing returns as the chunk size gets too small, however, due to the current approach of using a Global instance per-chunk.

Caveats

  • Incompatible with the non-local optimizations in the backend as .class files for symbols backed by exported pickles aren't available. This shouldn't be a big problem in "dev mode".
  • To get the most of out of the caching of macro classloaders, a hand crafted -Ymacro-classpath setting is needed with the macro implementation.

Future work:

@scala-jenkins scala-jenkins added this to the 2.12.9 milestone May 22, 2019
@retronym retronym force-pushed the topic/outline-typing branch from 3cfb31c to e4a2db1 Compare May 22, 2019 04:32
retronym added 2 commits May 22, 2019 15:55
@retronym
PipelineMain: add test, make it more testable and less buggy.
039b118 
  - Allow user specified reporter
    - funnel javac errors through it
    - funnel PipelineMain's logging through it, too.
  - Use a separate FileManager for each javac invocation to avoid an
    apparent race condition.
  - Expose config knobs programatically rather than only through
    system properties.
@retronym
Add experimental support for outline typing
79ace8a 
In this new mode, the RHS of definitions is only typechecked
if the definition lacks an explicit type ascription, or or it
may contain a super call that is compiled to a trait super
accessor. Refer to the new test case for a motivating example.
@retronym retronym force-pushed the topic/outline-typing branch from e4a2db1 to 79ace8a Compare May 22, 2019 05:55
@retronym
Copy link
Member Author

retronym commented May 22, 2019

I'm testing this out now on some real world projects. Will mark as WIP while I do that.

@retronym retronym added the WIP label May 22, 2019
@retronym retronym force-pushed the topic/outline-typing branch from 826cd8c to a669e91 Compare May 22, 2019 06:15
adriaanm and others added 2 commits May 22, 2019 17:15
@adriaanm @retronym
[backport] Drop leaky encoding from JavaParser
1a16514 
It's subsumed by Jason's improvements to name resolution in scala#7671.

The leak yielded spurious errors in mixed Scala/Java compilation (akka-http).

(cherry picked from commit 8529be7)
@retronym
Test for backported fix in prior commit (pls merge forward)
5e83e4a
@retronym
Copy link
Member Author

retronym commented May 22, 2019 •
edited
Loading

Here's how I've exported suitable .args files for the Akka build and build with PipelineMain-s three strategies: https://github.com/retronym/akka/pull/1/files

I'm still analysing the results.

One functional problem so far:

  • chome trace output is broken under this new mode.

adriaanm
adriaanm reviewed May 24, 2019
View reviewed changes
@adriaanm
Copy link
Contributor

adriaanm commented May 24, 2019

Very nice! I would like to see a short motivation for each test case, but LGTM otherwise.

@retronym retronym force-pushed the topic/outline-typing branch from ce06564 to 08e697d Compare May 26, 2019 07:27
@retronym retronym removed the WIP label May 26, 2019
@retronym retronym merged commit ad99127 into scala:2.12.x May 26, 2019
@eed3si9n eed3si9n added the release-notes worth highlighting in next release notes label Aug 1, 2019
@eed3si9n eed3si9n mentioned this pull request Aug 1, 2019
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@adriaanm adriaanm adriaanm left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

release-notes worth highlighting in next release notes

Projects

None yet

Milestone

2.12.9

Development

Successfully merging this pull request may close these issues.

4 participants

@retronym @adriaanm @eed3si9n @scala-jenkins