Skip to content

Sbt build for Scala#2

Closed
gkossakowski wants to merge 258 commits into2.11.xfrom
sbt-build
Closed

Sbt build for Scala#2
gkossakowski wants to merge 258 commits into2.11.xfrom
sbt-build

Conversation

@gkossakowski
Copy link
Owner

The new, sbt-based build definition for Scala

This is a PR submitted against my own fork. I'll move it over to scala/scala when it matures a bit.

What you see here is very much work-in-progress. Basics like compiling and packaging jars (into the right location) work. Everything else is missing.

Tasks up for grabs

  • building docs (doc task produces errors at the moment)
  • placing shell scripts in build/pack/bin (so you can run the compiler or repl easily) (@adriaanm)
  • running partest tests (@michaelpigg)
  • compiling and running JUnit tests (@retronym)
  • add Eclipse support (see this comment)
  • check if this project imports into IntelliJ (@retronym)
  • disable docs generation for asm and forkjoin
  • figure out what's the role of library-aux and how to plug it into sbt build
  • generate .properties files that are placed in jars (see for example library.properties) (@dumpstate, Generating .properties files that are placed in jars. #3)
  • generate OSGi bundle information for all subprojects we publish
  • implement OSGi testing
  • add sbt compiler interface testing (checking if we are not breaking sbt's incremental compilation with changes to the Scala compiler)
  • integrate MiMa for binary compatibility checking
  • make dependency on ant provided so it doesn't show up as transitive dependency
  • make compiler/run and repl/run work (@gkossakowski)

If you would like to grab any of those tasks just drop a comment below. I'll update the list with your github id.


You'll notice that this build definition is much more complicated than your typical sbt build. The main reason is that we are not benefiting from sbt's conventions when it comes project layout. For that reason we have to configure a lot more explicitly. I've tried explain in comments the less obvious settings.

This nicely leads me to explaning goal and non-goals of this build definition.

Goals

  • to be easy to tweak it in case a bug or small inconsistency is found
  • to mimic Ant's behavior as closely as possible
  • to be super explicit about any departure from standard sbt settings
  • to achieve functional parity with Ant build as quickly as possible
  • to be readable and not necessarily succint
  • to provide the nicest development experience for people hacking on Scala

Non-goals

  • to have the shortest sbt build definition possible; we'll beat Ant definition easily and that will thrill us already
  • to remove irregularities from our build process right away
  • to modularize the Scala compiler or library further

It boils down to

Simple rules

  • project laytout is set in stone for now
  • if you need to work on convincing sbt to follow non-standard layout then explain everything you did in comments
  • constantly check where Ant build produces class files, artifacts, what kind of other files generates and port all of that to here

Note on bootstrapping

Let's start with reminder what bootstrapping means in our context. It's an answer to this question: which version of Scala are using to compile Scala? The fact that the question sounds circular suggests trickiness. Indeed, bootstrapping Scala compiler is a tricky process.

The Ant build used to have involved system of bootstrapping Scala. It would consist of three layers: starr, locker and quick. The sbt build for Scala ditches layering and strives to be as standard sbt project as possible. This means that we are simply building Scala with latest stable release of Scala. See this discussion for more details behind this decision.

lrytz and others added 8 commits February 7, 2015 07:45
Trait implementation classes and specialized classes are always
considered top-level in terms of the InnerClass / EnclosingMethod
attributes. These attributes describe source-level properties, and
such classes are a compilation artifact.

Note that the same is true for delambdafy:method closure classes
(they are always top-level).
Members of value classes are moved over to the companion object early.
This change ensures that closure classes nested in value classes
appear that way to Java reflection.

This commit also changes the EnclosingMethod attribute for classes
(and anonymous functions) nested in anonymous function bodies. Before,
the enclosing method was in some cases the function's apply method.
Not always though:
  () => { class C ... val a = { class D ...} }
The class C used to be nested in the function's apply method, but not
D, because the value definition for a was lifted out of the apply.

After this commit, we uniformly set the enclosing method of classes
nested in function bodies to `null`. This is consistent with the
source-level view of the code.

Note that under delambdafy:method, closures never appear as enclosing
classes (this didn't change in this commit).
Run a number of Java reflection operations on all classes in
scala library / reflect / compiler.

The test is based on a draft by Jason.
The test is corrected (inverted) and the extractor is made
more succinct.  Succinctness isn't enforced by the test,
but I checked it manually.
Now both of the overloaded variants have a parameter list.
This seems to make type inference happier. Or it makes someone
happier.

The user is unaware whether `subsets()` takes a default arg.
But happily, empty application still kicks in.
Updated documentation of + and ++ in SetLike to explicitly state that duplicate elements
are not added to the set.
build.sbt Outdated
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will likely cause significant issues for IDEs. They definitely want each project to live in a different directory, don't they? Or is baseDirectory not use by, say, sbteclipse?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! The baseDirectory shows up in this search: https://github.com/typesafehub/sbteclipse/search?utf8=%E2%9C%93&q=baseDirectory

It's definitively a goal to provide good experience for both Eclipse and IntelliJ users. I added this as a task to PR description.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adressed in c44ecb9.

@michaelpigg
Copy link

I'm interesting in working on getting partest working.

@michaelpigg
Copy link

Having problems with the project not loading in sbt, though: https://gist.github.com/michaelpigg/9cacd81ed43780fefa7d

@gkossakowski
Copy link
Owner Author

hey @michaelpigg!

Assigned to you! We have an implementation of sbt's testing interface for partest already: https://github.com/scala/scala-partest-interface
Not sure if it's used in any of our subprojects, though. I'll check and let you know.

@gkossakowski
Copy link
Owner Author

What sbt plugins do you have enabled? I think one of them might not be compatible with sbt's autoPlugin mechanism. As a work-around comment out disablePlugins(plugins.IvyPlugin) lines from the build.

@sjrd
Copy link

sjrd commented Feb 12, 2015

@gkossakowski
Copy link
Owner Author

Thanks! What was the reason for forking?

@sjrd
Copy link

sjrd commented Feb 12, 2015

At least, we had to replace
https://github.com/scala/scala-partest-interface/blob/master/src/main/scala/tools/partest/TestingInterface.scala#L75
by
https://github.com/scala-js/scala-js/blob/master/partest/src/main/scala/scala/tools/partest/scalajs/PartestInterface.scala#L97
and
https://github.com/scala/scala-partest-interface/blob/master/src/main/scala/tools/partest/TestingInterface.scala#L90
by
https://github.com/scala-js/scala-js/blob/master/partest/src/main/scala/scala/tools/partest/scalajs/PartestInterface.scala#L113

I don't remember if there were other things we needed to change.

Btw: the Scala.js build is much more complex than the tiny build this PR currently contains, so

You'll notice that this build definition is much more complicated than your typical sbt build.

made me smile a bit :-p

@michaelpigg
Copy link

@gkossakowski You're right, good time to trim those plugins down a bit.

@gkossakowski
Copy link
Owner Author

Thanks for references! I created scala/scala-partest#30 for future reference.

This build is pretty complicated for what it does (really just building and packaging). I'm pretty sure we'll be growing down the road.

Users of German & Swedish keyboards can again
type `]` in the REPL without crazy acrobatics.

Fixed upstream in jline/jline2#175
@vsalvis
Copy link

vsalvis commented Feb 12, 2015

I'm going to start work on making partest more configurable soon so it can also be used for Dotty, and it makes sense to integrate the scala.js work and possibly other requests too.

@gkossakowski
Copy link
Owner Author

Thanks @vsalvis! Please keep us in the loop when this happens.

JustinPihony and others added 7 commits February 12, 2015 21:22
Improve attribute parsing and propagate errors
across event thread. Otherwise tests just hang.
This is tested, right?

This is an upstream port of scala-xml
5f2cfadeb9e8574ed66f37dc7a7a868eb129a8a9
SI-8759 bump jline: support alt-gr-based keyboards
SI-8841 report named arg / assignment ambiguity also in silent mode.
Use `j.l.Class.isInstance` for Scala.js, so that `unapply`
works correctly when referencing raw JavaScript classes.
Inline overloaded calls to `unapply`, so we can get rid of them in 2.13.

Note that theres a lot of `box(unbox(x))` going on behind the scenes.
Is this needed?
gkossakowski and others added 20 commits April 15, 2015 01:08
Static resources are stored along source files in our current layout.
Therefore we set resourceDirectory to be sourceDirectory but make sure
that includeFilter includes nothing by default. Subprojects like library
or compiler set their own include filters.

The include filters have been ported from the Ant build definition.
As it turns out, we generate properties files only for those three
subprojects.
We mimic what Ant build does.
Before this commit, we used to have common way of handling names of
subproject. We would construct the name by prepending "scala-" suffix to
project id (which is simply the name of directory in src/). It turns out,
our projects are not named uniformly. For example, we have just "repl" and
not "scala-repl".

In this commit, we give up on central naming logic and have manual settings
for each subproject that requires special handling.
Let's override `sources` task to return empty collections of sources to
compile. Otherwise, sbt by default would pick up sources in the root
directory which is not what we want.
Actors get properties files generated so we need to add settings for that.
Also, the jar is called "scala-actors.jar" so we need to tweak the name
of the project.
Shell scripts are generated with `mkBin` sbt task that calls ScalaTool.
ScalaTool is defined in project/ and is a port of Ant task that lives
in scala.tools.ant.ScalaTool. While porting, we've simplified the code
significantly.

The `mkBin` task is defined in a new subproject: dist. This subproject
will become responsible for building the distribution in the future.
This way we don't need to worry about mixing classes for Scala compiler
that sbt is invoking and the one we just built. For an example of
possible problem, check out:

#2 (comment)
Those jars are put by Ant build in build/pack/lib but they are not needed
to be there.
Common library dependencies are extracted into vals.
Those dependencies define "scala-library" as an excluded transitive
dependency so we don't get eviction warnings during `update`.
Lastly, version numbers for dependencies are obtained from
`versions.properties` file.
 - Explicitly set [un]managed[re]sourceDirectories for all
   projects to improve the import into IntelliJ
 - Add partest extras project, that contains scala/scala specific
   extensions to partest for use in partest and junit tests.
 - Add junit project, which contains unit tests for the compiler and
   library.
All partests pass. Partest tests are defined in Integration configuration.
This means you need to run them with `test/it:test` command. We put them
in separate configuration because they are slow to run so user needs to
ask explicitly for partest to run.
Introduce a common way of setting jar location so it can be reused between
`scalaSubprojectSettings` and `partestJavaAgent`. This reduces code
duplication.
The 6e6632e introduced a junit test
for scaladoc. We need to take that into account in sbt build.
Change pattern in .gitignore file so git ignores all target directories.
Add welcome message that is shown at sbt's startup. The message clearly
indicates experimental status of sbt build.

Also, add a short paragraph to README.md explaining status of sbt build.
@gkossakowski gkossakowski mentioned this pull request Apr 16, 2015
16 tasks
adriaanm pushed a commit to adriaanm/scala that referenced this pull request Apr 17, 2015
This way we don't need to worry about mixing classes for Scala compiler
that sbt is invoking and the one we just built. For an example of
possible problem, check out:

gkossakowski#2 (comment)
adriaanm pushed a commit to adriaanm/scala that referenced this pull request Apr 17, 2015
This way we don't need to worry about mixing classes for Scala compiler
that sbt is invoking and the one we just built. For an example of
possible problem, check out:

gkossakowski#2 (comment)
@SethTisue SethTisue mentioned this pull request Sep 14, 2015
20 tasks
gkossakowski pushed a commit that referenced this pull request Apr 21, 2016
The log messages intented to chronicle implicit search were
always being filtered out by virtue of the fact that the the tree
passed to `printTyping` was already typed, (e.g. with an implicit
MethodType.)

This commit enabled printing in this case, although it still
filters out trees that are deemed unfit for typer tracing,
such as `()`. In the context of implicit search, this happens
to filter out the noise of:

```
|    |    |    [search #2] start `()`, searching for adaptation to pt=Unit => Foo[Int,Int] (silent: value <local Test> in Test) implicits disabled
|    |    |    [search #3] start `()`, searching for adaptation to pt=(=> Unit) => Foo[Int,Int] (silent: value <local Test> in Test) implicits disabled
|    |    |    \-> <error>
```

... which I think is desirable.

The motivation for this fix was to better display the interaction
between implicit search and type inference. For instance:

```
class Foo[A, B]
class Test {
  implicit val f: Foo[Int, String] = ???
  def t[A, B](a: A)(implicit f: Foo[A, B]) = ???
  t(1)
}
```

````
% scalac -Ytyper-debug sandbox/instantiate.scala
...
|    |-- t(1) BYVALmode-EXPRmode (site: value <local Test> in Test)
|    |    |-- t BYVALmode-EXPRmode-FUNmode-POLYmode (silent: value <local Test> in Test)
|    |    |    [adapt] [A, B](a: A)(implicit f: Foo[A,B])Nothing adapted to [A, B](a: A)(implicit f: Foo[A,B])Nothing
|    |    |    \-> (a: A)(implicit f: Foo[A,B])Nothing
|    |    |-- 1 BYVALmode-EXPRmode-POLYmode (site: value <local Test> in Test)
|    |    |    \-> Int(1)
|    |    solving for (A: ?A, B: ?B)
|    |    solving for (B: ?B)
|    |    [search #1] start `[A, B](a: A)(implicit f: Foo[A,B])Nothing` inferring type B, searching for adaptation to pt=Foo[Int,B] (silent: value <local Test> in Test) implicits disabled
|    |    [search #1] considering f
|    |    [adapt] f adapted to => Foo[Int,String] based on pt Foo[Int,B]
|    |    [search #1] solve tvars=?B, tvars.constr= >: String <: String
|    |    solving for (B: ?B)
|    |    [search #1] success inferred value of type Foo[Int,=?String] is SearchResult(Test.this.f, TreeTypeSubstituter(List(type B),List(String)))
|    |    |-- [A, B](a: A)(implicit f: Foo[A,B])Nothing BYVALmode-EXPRmode (site: value <local Test> in Test)
|    |    |    \-> Nothing
|    |    [adapt] [A, B](a: A)(implicit f: Foo[A,B])Nothing adapted to [A, B](a: A)(implicit f: Foo[A,B])Nothing
|    |    \-> Nothing
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.