Skip to content

SI-9920 Avoid linkage errors with captured local objects + self types#5397

Merged
adriaanm merged 1 commit intoscala:2.12.0from
retronym:ticket/9920
Sep 28, 2016
Merged

SI-9920 Avoid linkage errors with captured local objects + self types#5397
adriaanm merged 1 commit intoscala:2.12.0from
retronym:ticket/9920

Conversation

@retronym
Copy link
Member

@retronym retronym commented Sep 13, 2016

An outer parameter of a nested class is typed with the self type
of the enclosing class:

class C; trait T { _: C => def x = 42; class D { x } }

leads to:

    class D extends Object {
      def <init>($outer: C): T.this.D = {
        D.super.<init>();
        ()
      };
      D.this.$outer().$asInstanceOf[T]().x();

Note that a cast is inserted before the call to x.

If we modify that a little, to instead capture a local module:

class C; trait T { _: C => def y { object O; class D { O } } }

Scala 2.11 used to generate (after lambdalift):

    class D$1 extends Object {
      def <init>($outer: C, O$module$1: runtime.VolatileObjectRef): C#D$1 = {
        D$1.super.<init>();
        ()
      };
      D$1.this.$outer().O$1(O$module$1);

That isn't type correct, D$1.this.$outer() : C does not
have a member O$1.

However, the old trait encoding would rewrite this in mixin to:

T$class.O$1($outer, O$module$1);

Trait implementation methods also used to accept the self type:

trait T$class {
   final <stable> def O$1($this: C, O$module$1: runtime.VolatileObjectRef): T$O$2.type
}

So the problem was hidden.

This commit changes replaces manual typecheckin of the selection in LambdaLift with
a use of the local (erasure) typer, which will add casts as needed.

For run/t9220.scala, this changes the post LambdaLift AST as follows:

     class C1$1 extends Object {
       def <init>($outer: C0, Local$module$1: runtime.VolatileObjectRef): T#C1$1 = {
         C1$1.super.<init>();
         ()
       };
-      C1$1.this.$outer.Local$1(Local$module$1);
+      C1$1.this.$outer.$asInstanceOf[T]().Local$1(Local$module$1);
       <synthetic> <paramaccessor> <artifact> private[this] val $outer: C0 = _;
       <synthetic> <stable> <artifact> def $outer(): C0 = C1$1.this.$outer
     }

@scala-jenkins scala-jenkins added this to the 2.12.0-RC2 milestone Sep 13, 2016
@retronym
Copy link
Member Author

TODO:

@retronym
Copy link
Member Author

Review by @lrytz

@lrytz
Copy link
Member

lrytz commented Sep 14, 2016

Sorry about the lengthy comment...

I modified the example a little bit and moved the object O into class C:

class C { object O }
trait T { _: C =>
  def foo {
    class D { O }
  }
}

In this case there should not be a cast, and indeed we get

  class T$D$1 extends Object {
    def <init>($outer: C): T$D$1 = {
      T$D$1.super.<init>();
      $outer.O();
      ()
    }
  }

This is correct, but it seems to conflict with what you say in the commit message: "An outer parameter of a nested class is typed with the self type of the enclosing class". When I spin up the debugger in ExplicitOuter.outerSelect, the $outer field has type UniqueThisType(T). The base tree in method outerPath therefore has that type, the tree shape is D.this.$outer. We are in the ExplicitOuter tree transform, the current phase is "erasure" (there's an exitingExplicitOuter).

When using the example that your patch actually fixes (move object O into method foo):

class C
trait T { _: C =>
  def foo {
    object O
    class D { O }
  }
}

We enter method outerPath trough a different trace: we're in the LambdaLift transformation, the current phase is "constructors" (again there's an afterOwnPhase). In this case, the tree shape of the base tree is D$1.this.$outer, with type TypeRef(C). The tree is still constructed by the outerSelect methods. But the $outer filed now has the following type history: explicitouter: <refinement>.type, posterasure: C. In this case your patch actually introduces the cast (and fixes the bug).

So I'm wondering if that's all intended..

@retronym
Copy link
Member Author

retronym commented Sep 15, 2016

@lrytz

This is correct, but it seems to conflict with what you say in the commit message: "An outer parameter of a nested class is typed with the self type of the enclosing class".

I don't think there is a conflict (maybe my commit message was ambiguous and you've interpreted it differently?) The outer parameter of the nested class D is not of type T, but rather of type C (the self type of C). Calling a member of C does not incur a cast in any Scala version.

@retronym
Copy link
Member Author

retronym commented Sep 15, 2016

I guess it is more accurate to say that the outer parameter is typed with the ThisType of the trait, which in some cases erases to a type that does not extend the trait itself.

scala> trait T { _: C => }; class C
defined trait T
defined class C

scala> ThisType(symbolOf[T])
res1: $r.intp.global.Type = <refinement>.type

scala> ThisType(symbolOf[T]).baseTypeSeq
res2: $r.intp.global.BaseTypeSeq = BTS(<refinement>.type,T with C,T,C,Object,Any)

scala> erasure.scalaErasure(res1)
res5: $r.intp.global.erasure.global.Type = C

@lrytz
Copy link
Member

lrytz commented Sep 15, 2016

My concern is the following:

scala> trait T { _: C => class D }; class C
defined trait T
defined class C

scala> val entering = enteringErasure(typeOf[T#D].member(newTermName("$outer ")).info)
entering: $r.intp.global.Type = <refinement>.type

scala> val exiting = exitingErasure(typeOf[T#D].member(newTermName("$outer ")).info)
exiting: $r.intp.global.Type = C

scala> val c = symbolOf[C]
c: $r.intp.global.TypeSymbol = class C

scala> entering.typeSymbol.isNonBottomSubClass(c)
res31: Boolean = false

scala> exiting.typeSymbol.isNonBottomSubClass(c)
res32: Boolean = true

Your patch uses isNonBottomSubClass, so the result depends on the phase. For the example

class C { object O }
trait T { _: C =>
  def foo {
    class D { O }
  }
}

the check happens to be executed enteringErasure, so no cast is added, we get $outer.O - a cast to T would be wrong, as T doesn't have an O.

For the example that your patch fixes, the check happens to be executed exitingErasure, so we do get the cast.

This looks a bit like we're just lucky.

@retronym
Copy link
Member Author

@lrytz Thanks for digging deep.

Another irregularity that I don't understand is why inner classes and traits differ in whether they use the self type of the class type of the outer class.

scala> class C; trait T { _: C => class IC; trait IT { T.this.toString } };
defined class C
defined trait T

scala> :javap -private T$IT T$IC
Compiled from "<console>"
public interface T$IT {
  public abstract T $line6$$read$T$IT$$$outer();
}
Compiled from "<console>"
public class T$IC {
  public final C $outer;
  public C $line6$$read$T$IC$$$outer();
  public T$IC(C);
}
  def newOuterAccessor(clazz: Symbol) = {
    val accFlags = SYNTHETIC | ARTIFACT | STABLE | ( if (clazz.isTrait) DEFERRED else 0 )
    val sym      = clazz.newMethod(nme.OUTER, clazz.pos, accFlags)
    val restpe   = if (clazz.isTrait) clazz.outerClass.tpe_* else clazz.outerClass.thisType

    sym expandName clazz
    sym.referenced = clazz
    sym setInfo MethodType(Nil, restpe)
  }

@szeiger
Copy link
Contributor

szeiger commented Sep 20, 2016

I don't know if it's the same problem, but all tests in MiMa fail with an AbstractMethodError in both, RC1 and this version:

$ sbt -Dmima.testScalaVersion=2.12.0-c215d5f-SNAPSHOT testFunctional
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024M; support was removed in 8.0
[info] Loading global plugins from /Users/szeiger/.sbt/0.13/plugins/project
[info] Loading global plugins from /Users/szeiger/.sbt/0.13/plugins
[info] Loading project definition from /Users/szeiger/code/migration-manager/project
[info] Resolving key references (73348 settings) ...
[info] Set current project to mima (in build file:/Users/szeiger/code/migration-manager/)
java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
        at com.typesafe.tools.mima.core.Config$.setup(Config.scala:58)
        at com.typesafe.tools.mima.core.Config$.setup(Config.scala:54)
        at com.typesafe.tools.mima.lib.CollectProblemsTest.runTest(CollectProblemsTest.scala:13)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at MimaBuild$$anonfun$runTest$1.apply(Build.scala:252)
        at MimaBuild$$anonfun$runTest$1.apply(Build.scala:234)
        at scala.Function6$$anonfun$tupled$1.apply(Function6.scala:35)
        at scala.Function6$$anonfun$tupled$1.apply(Function6.scala:34)
        at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
        at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
        at sbt.std.Transform$$anon$4.work(System.scala:63)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
        at sbt.Execute.work(Execute.scala:237)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
        at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
...

Update: This could be a classpath problem in MiMa instead. I'm still working on a standalone reproduction.

@szeiger
Copy link
Contributor

szeiger commented Sep 20, 2016

Never mind, the MiMa problem was indeed caused by mixing the wrong classpaths.

@retronym
Copy link
Member Author

@lrytz I've pushed a new patch that changes the caller to outerPath in LambdaLift, rather than changing outerPath itself.

@lrytz
Copy link
Member

lrytz commented Sep 21, 2016

VerifyError: Bad type on operand stack. I'll take a look later today.

BTW, @szeiger, sbt's error output is humongous, check https://scala-ci.typesafe.com/job/scala-2.12.0-validate-test/29/console

@szeiger
Copy link
Contributor

szeiger commented Sep 21, 2016

@lrytz That's the downside of running all tests and then printing all results. I should be able to prune it down quite a bit by not printing the same failure twice.

@lrytz
Copy link
Member

lrytz commented Sep 23, 2016

Minimized regression by the latest diff:

  def o = {
    def i(n: Int) {
      i(n)
      i(n)
    }
    i(0)
  }

crashes the compiler with

scala.tools.asm.tree.analysis.AnalyzerException: While processing Test$.i$1
    at scala.tools.nsc.backend.jvm.analysis.BackendUtils$AsmAnalyzer.<init>(BackendUtils.scala:40)
Caused by: java.lang.IndexOutOfBoundsException: Cannot pop operand off an empty stack.

I'll take a look at it today.

@lrytz
Copy link
Member

lrytz commented Sep 23, 2016

We have a phase ordering issue with recursive local methods, not sure how to get around it. Given:

object Test {
  def o = {
    def i: Int = { i; 0 }
    i
  }
}

LambdaLift's preTransform changes the recursive call Ident(i) to Select(Test$, i$1), this is the location of your patch.

The problem is that only postTransform will perform the actual lifting of def i and re-write the symbol's owner. So when calling localTyper.typed(Select(Test$, iSymbol)), method adaptMember in the Eraser performs the following check:

  <Test$>.tpe.typeSymbol isSubClass iSymbol.owner

At this point, iSymbol.owner is still method o, so the check fails, and a cast is inserted:

cast(<Test$>, iSymbol.owner.tpe.resultType) // cast to Int

Note that this check / inserting cast is exactly the logic that fixes the actual bug.

It would work to change add .enclClass in the Eraser, but that seems too much of a hack. Ideas?

An outer parameter of a nested class is typed with the self type
of the enclosing class:

```
class C; trait T { _: C => def x = 42; class D { x } }
```

leads to:

```
    class D extends Object {
      def <init>($outer: C): T.this.D = {
        D.super.<init>();
        ()
      };
      D.this.$outer().$asInstanceOf[T]().x();
```

Note that a cast is inserted before the call to `x`.

If we modify that a little, to instead capture a local module:

```
class C; trait T { _: C => def y { object O; class D { O } } }
```

Scala 2.11 used to generate (after lambdalift):

```
    class D$1 extends Object {
      def <init>($outer: C, O$module$1: runtime.VolatileObjectRef): C#D$1 = {
        D$1.super.<init>();
        ()
      };
      D$1.this.$outer().O$1(O$module$1);
```

That isn't type correct, `D$1.this.$outer() : C` does not
have a member `O$1`.

However, the old trait encoding would rewrite this in mixin to:

```
T$class.O$1($outer, O$module$1);
```

Trait implementation methods also used to accept the self type:

```
trait T$class {
   final <stable> def O$1($this: C, O$module$1: runtime.VolatileObjectRef): T$O$2.type
}
```

So the problem was hidden.

This commit changes replaces manual typecheckin of the selection in LambdaLift with
a use of the local (erasure) typer, which will add casts as needed.

For `run/t9220.scala`, this changes the post LambdaLift AST as follows:

```
     class C1$1 extends Object {
       def <init>($outer: C0, Local$module$1: runtime.VolatileObjectRef): T#C1$1 = {
         C1$1.super.<init>();
         ()
       };
-      C1$1.this.$outer.Local$1(Local$module$1);
+      C1$1.this.$outer.$asInstanceOf[T]().Local$1(Local$module$1);
       <synthetic> <paramaccessor> <artifact> private[this] val $outer: C0 = _;
       <synthetic> <stable> <artifact> def $outer(): C0 = C1$1.this.$outer
     }
```
@retronym
Copy link
Member Author

@lrytz Hmmm, it is tricky. I'm trying a middle ground between my first and second attempts: I'm manually adding the cast, but only doing this in LambdaLift.

@adriaanm adriaanm self-assigned this Sep 27, 2016
@lrytz
Copy link
Member

lrytz commented Sep 28, 2016

LGTM

@retronym
Copy link
Member Author

@adriaanm will leave you to hit merge on this after you pass another set of eyes over it.

@adriaanm
Copy link
Contributor

👀

@adriaanm adriaanm merged commit 3b5d4e2 into scala:2.12.0 Sep 28, 2016
@adriaanm adriaanm added the release-notes worth highlighting in next release notes label Oct 18, 2016
@adriaanm adriaanm modified the milestone: 2.12.0-RC2 Oct 29, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes worth highlighting in next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants