Skip to content

Improvements to the type variable solver.#6127

Closed
milessabin wants to merge 2 commits intoscala:2.13.xfrom
milessabin:topic/bounds-propagation
Closed

Improvements to the type variable solver.#6127
milessabin wants to merge 2 commits intoscala:2.13.xfrom
milessabin:topic/bounds-propagation

Conversation

@milessabin
Copy link
Contributor

@milessabin milessabin commented Oct 12, 2017

This is an attempt to fix some issues and make some small improvements in the type variable solver. It's not complete, but I'm pushing to get feedback and so that I can run a community build against it.

The main change is that I've added a more sophisticated test for the currently-being-solved TypeVar being used in the bounds of another type variable, allowing more information to be propagated. This also fixes a bug where the test case TypeRef(_, `tparam`, _) allows a fully applied type to be added to the bounds of a higher kinded type resulting in a blow up in glb/lub.

I've left various open issues in comments ... these should be pursued if this is followed up on.

Nb. the test cases compile in Dotty, so this is a move in the right direction.

@milessabin milessabin added the WIP label Oct 12, 2017
@milessabin milessabin requested a review from retronym October 12, 2017 11:29
@scala-jenkins scala-jenkins added this to the 2.12.5 milestone Oct 12, 2017
@adriaanm adriaanm modified the milestones: 2.12.5, 2.13.0-M4 Oct 12, 2017
@adriaanm
Copy link
Contributor

Changes to type inference and other core parts of the compiler should go to 2.13 at this point.

@milessabin
Copy link
Contributor Author

@adriaanm gotcha ... I'll rebase.

@milessabin
Copy link
Contributor Author

Green community build here.

@milessabin milessabin force-pushed the topic/bounds-propagation branch from 24442fe to 7bd9851 Compare October 17, 2017 12:00
@milessabin milessabin changed the base branch from 2.12.x to 2.13.x October 17, 2017 12:01
@milessabin
Copy link
Contributor Author

Rebased to 2.13.x ...

milessabin added a commit to milessabin/scala that referenced this pull request Nov 17, 2017
This fixes typelevel#172 reported
against the literal types implementation, but is a symptom of more
general issues in the solver that I've been exploring in
scala#6127.

Ideally this would be an independent PR, but it's very difficult to
reproduce without ValueOf which would have to be replicated as a macro,
so on balance I think it's better to in include it here.
milessabin added a commit to milessabin/scala that referenced this pull request Nov 17, 2017
This fixes typelevel#172 reported
against the literal types implementation, but is a symptom of more
general issues in the solver that I've been exploring in
scala#6127.

Ideally this would be an independent PR, but it's very difficult to
reproduce without ValueOf which would have to be replicated as a macro,
so on balance I think it's better to in include it here.
milessabin added a commit to milessabin/scala that referenced this pull request Dec 8, 2017
This fixes typelevel#172 reported
against the literal types implementation, but is a symptom of more
general issues in the solver that I've been exploring in
scala#6127.

Ideally this would be an independent PR, but it's very difficult to
reproduce without ValueOf which would have to be replicated as a macro,
so on balance I think it's better to in include it here.
milessabin added a commit to milessabin/scala that referenced this pull request Jan 2, 2018
This fixes typelevel#172 reported
against the literal types implementation, but is a symptom of more
general issues in the solver that I've been exploring in
scala#6127.

Ideally this would be an independent PR, but it's very difficult to
reproduce without ValueOf which would have to be replicated as a macro,
so on balance I think it's better to in include it here.
milessabin added a commit to milessabin/scala that referenced this pull request Jan 4, 2018
This fixes typelevel#172 reported
against the literal types implementation, but is a symptom of more
general issues in the solver that I've been exploring in
scala#6127.

Ideally this would be an independent PR, but it's very difficult to
reproduce without ValueOf which would have to be replicated as a macro,
so on balance I think it's better to in include it here.
milessabin added a commit to milessabin/scala that referenced this pull request Jan 10, 2018
This fixes typelevel#172 reported
against the literal types implementation, but is a symptom of more
general issues in the solver that I've been exploring in
scala#6127.

Ideally this would be an independent PR, but it's very difficult to
reproduce without ValueOf which would have to be replicated as a macro,
so on balance I think it's better to in include it here.
milessabin added a commit to milessabin/scala that referenced this pull request Jan 13, 2018
This fixes typelevel#172 reported
against the literal types implementation, but is a symptom of more
general issues in the solver that I've been exploring in
scala#6127.

Ideally this would be an independent PR, but it's very difficult to
reproduce without ValueOf which would have to be replicated as a macro,
so on balance I think it's better to in include it here.
milessabin added a commit to milessabin/scala that referenced this pull request Jan 17, 2018
This fixes typelevel#172 reported
against the literal types implementation, but is a symptom of more
general issues in the solver that I've been exploring in
scala#6127.

Ideally this would be an independent PR, but it's very difficult to
reproduce without ValueOf which would have to be replicated as a macro,
so on balance I think it's better to in include it here.
def propagate(bound0: Type, instantiate: => Type): Unit = bound0 match {
case TypeRef(_, `tparam`, _) if !tvar.isHigherKinded && !bound0.isHigherKinded =>
addBound(instantiate)
case TypeRef(_, `tparam`, _) if tvar.isHigherKinded && bound0.isHigherKinded && tparam.typeParams.size == bound0.typeParams.size =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused. When can this happen? If bound0.isHigherKinded shouldn't it be a PolyType and not a TypeRef? Also checking the size of type parameters might not be enough. Basically we care only if the bound is F[x, y, z[_]] <: G[x, y, z], but it could be F[x, y, z[_]] <: G[z, x, y] or F[x, y, z[_]] <: G[x, y, List].

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't remember the scenario now. It's possible that it's something that came up with incorrect code.

// mentioned in the active bound are solved first.
// 1. Is this reordering desirable? Should solving always be left to right instead?
// 2. Should this take account of the variance of tparam2 and flip the bounds if
// necessary?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When would it be necessary to flip the bounds?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flip the bounds depending on whether the param is co/inv vs. contra variant.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

up depends on the variance of tparam and the variance of tparam2 shouldn't matter I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to be convinced of that :-)

@joroKr21
Copy link
Member

@milessabin do you need help with this? What more needs to be done?

@milessabin
Copy link
Contributor Author

I wanted to get things to a state where t7289_status_quo.scala was consistent between the covariant and bounded existential cases and all around reasonable, without breaking a ton of other stuff. I ran out of time ... it'd be great if you could try and make some more progress on it.

@joroKr21
Copy link
Member

joroKr21 commented Mar 18, 2018

I think I made some progress on my fork that improves the status quo test. It also shows that the reordering of type variables is necessary to make that happen (otherwise A would be inferred as Nothing). But there are still some things I am concerned about:

  • Blindly propagating the type constructor - think Nil.type <:< List[x] but Nil.type doesn't have a type constructor. However, I couldn't come up with a counterexample that compiled before.
  • Blindly propagating the type arguments - think Map[k, v] <:< Traversable[(k, v)]. Again, couldn't find a counterexample.
  • Respecting variance when propagating type arguments - this is trickier. What if we have bounds like this: [X, Y, F <: X => Y]?

@adriaanm
Copy link
Contributor

adriaanm commented Apr 3, 2018

I added a few comments on the PR commit. Looks like it's heading in the right direction, but I want to be clear that there's a high bar for changes to type inference -- especially in a library release cycle (language changes should normally go to 2.14, unless needed now for some other reason, e.g. collections rework).

Can you elaborate on why it's safe/needed to check kind conformance from solveOne? Do we risk adding constraints? How about cycles? Performance?

@adriaanm
Copy link
Contributor

adriaanm commented Apr 3, 2018

That said I can be convinced to merge this, of course. Will also need a bit more comments, with motivation / brain dump, while it's fresh. Green community build.

@milessabin
Copy link
Contributor Author

I'd like to add this to #6378 and test with the new collections. So, far there haven't been any problems.

@joroKr21 how do you want to proceed? I'm happy for you to make a PR from your branch, or I could push your changes here ... you choose :-)

@joroKr21
Copy link
Member

joroKr21 commented Apr 3, 2018

@adriaanm I don't feel strongly about checking kind conformance. I was mostly worried about the first two points from #6127 (comment), but like I said I couldn't find any counterexamples. I'm inclined to agree that it's not worth the cost, hence I will remove the check.

  • Blindly propagating the type constructor - think Nil.type <:< List[x] but Nil.type doesn't have a type constructor. However, I couldn't come up with a counterexample that compiled before.
  • Blindly propagating the type arguments - think Map[k, v] <:< Traversable[(k, v)]. Again, couldn't find a counterexample.

@milessabin this is still mostly your work, so I can PR my additions to your branch.

@milessabin milessabin force-pushed the topic/bounds-propagation branch from b6a2076 to 018b64e Compare April 3, 2018 18:18
@milessabin
Copy link
Contributor Author

Rebased and merged @joroKr21's updates.

@milessabin
Copy link
Contributor Author

@joroKr21 good work on that :-)

This must fix a bunch of type inference- and hkt-related tickets ... do we know which?

@milessabin milessabin changed the title [WIP] Attempt to improve type variable solver. Improvements to the type variable solver. Apr 3, 2018
@milessabin milessabin added typelevel and removed WIP labels Apr 3, 2018
@milessabin milessabin requested a review from adriaanm April 3, 2018 18:24
// Propagate type constructor: F[A] <: B => F <: B.typeConstructor
// Propagate type arguments: F[+A, -B] <: C => A <: C.typeArgs(0) && B >: C.typeArgs(1)
case TypeRef(_, tsym, targs) =>
val inst = toInst(instantiate)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check for NoType?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addBound checks for NoType

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just added a check for NoType returned from toInst.

case TypeRef(_, tsym, targs) =>
val inst = toInst(instantiate)
if (tsym == tparam) addBound(variance, inst.typeConstructor)
foreach3(tsym.typeParams, targs, inst.typeArgs) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced of this part. Subtyping only moves to arguments once the type constructors are equal, not just in some subtype relation. You'd have to do something like relating F's type args with C.baseType(F).typeArgs, but we don't support doing that stuff on type vars. Can you show with an example that this code path is crucial, and how it works?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we're talking about the same thing over and over. This was my concern to begin with. For example:

  • Nil.type <:< List[x] but they don't have the same type constructor
  • Map[k, v] <:< Traversable[(k, v)] but they don't have the same type arguments

I tried to be on the safer side by doing a kind conformance check, but as you noted it's not a 100% guarantee and it's costly.

Propagating the type arguments is necessary for t7289_status_quo. Otherwise implicitly[Ext[_ <: List[List[Int]]]] and implicitly[ExtCov[List[List[Int]]]] are ambiguous.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In any case we have type variables on both sides so in the worst case we will infer something that doesn't respect the bounds and it will fail to compile when we check them. I couldn't find an example that compiled before and would fail with the more aggressive propagation, but if you can find one, that would be awesome.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for bearing with me, this is tricky stuff and it's core to the type checker, so we need to take our time to understand the change and make sure it's right.

I don't see how this step propagates the constraints to the right type args, since addBound will still be constraining the type var that was originally bounded by the applied type constructor, and now we are recursing on its args without changing the type var that addBound works on.

Even if we did propagate those args to the right type var somehow, I'm not sure it would help. When evaluating the progress made in the test, it's important to keep in mind the actual types inferred using -Xprint:typer. In the covariant case (and equivalents) we just end up with Nothing. In the example, this seems to be an accident, because the type args in bounds of CC end up constraining CC itself. (Turn the debugLog in addBound into a println to see the trace.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriaanm you're right, we are propagating constraints to the type var currently under consideration. We're getting those constraints from bounds on other type variables that it appears in.

For instance, consider def foo[A <: List[B], B](x: A): Option[B] = None where A is solved as List[Int] (see, test/files/pos/bounds-propagation.scala). Now when we come to solve for B we inspect the bounds that B appears in, here the bound on A. From that we can infer an Int bound on B and so solve B as Int. With the status quo we don't consider B's occurrence in other bounds so the best we get is Nothing.

Apropos the use of baseType in this context, at least one of the participants will be an unsolved type var, so we'll not get anything useful out of it. As @joroKr21 says we'll be checking for within bounds later anyway, so I don't think there's a problem here.


val variance = if (up) Variance.Covariant else Variance.Contravariant
val bound = if (up) tparam2.info.bounds.lo else tparam2.info.bounds.hi
propagate(variance, bound.dealias, tparam2.tpeHK.instantiateTypeParams(tparams, tvars))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not do a foreach2(tparams, tvars)(propagateBounds) below, so that tparam2.tpeHK.instantiateTypeParams(tparams, tvars) can simply become tvar2? The tpeHK of a type parameter tparam2 is TypeRef(NoPrefix, tparam2, Nil), upon which we subst that to the corresponding tvar...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize you didn't write this code, but now that we're all on this crazy ride down the rabbit hole...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This one is 13 years deep!)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriaanm I'm not quite following what you're suggesting ... propagateBounds takes a single argument, so it doesn't fit as an argument to foreach2. We do already have tparams.foreach(propagateBounds) and propagateBounds closes over tvars.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, also in that case instantiate doesn't need to be a by-name parameter.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a cleanup of the existing code that I see as uncontroversial: https://github.com/adriaanm/scala/tree/review/6127. I would be grateful if you could rebase your PR on top of this. I know this has already been a lot of work, but I'd be more comfortable if we move slowly here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh ... OK, got it. Trying that now.

Copy link
Contributor

@adriaanm adriaanm Apr 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My commit has that refactoring already in place. The remaining bit would be to dive into higher-order args, and peel off eta-expansion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flippin' 'eck ... the solver has barely been touched in 13 years!?!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup... All the more reason to be weary of users having come to depend on its precise (mal)functioning.... Experience over the years has shown that touching old code is bound to upset users. As I said before, 2.13 is focused on library changes, so we must be especially careful as we're counting on spending our "breakage budget" on the collections rework.

adriaanm added a commit to adriaanm/scala that referenced this pull request Apr 4, 2018
Based on work by joroKr21 and milessabin in scala#6127
@milessabin
Copy link
Contributor Author

Looks like Jenkins fell over ...

@milessabin
Copy link
Contributor Author

/rebuild

@milessabin
Copy link
Contributor Author

/synch

@adriaanm
Copy link
Contributor

adriaanm commented Apr 4, 2018

Artifactory has been crashing recently. Not sure why. Jenkins itself has been pretty stable.

@milessabin
Copy link
Contributor Author

testAll passes for me locally. rebuild again or leave it for a bit?

@milessabin
Copy link
Contributor Author

@adriaanm your foreach2 suggestion works out just fine. I noticed you've been playing around a bit too ... shall I push or are you working on an alternative?

@adriaanm
Copy link
Contributor

adriaanm commented Apr 4, 2018

I've pushed my work to https://github.com/adriaanm/scala/tree/review/6127, planning to leave it at adriaanm@dc39256. Would appreciate if you could rebase on top of that as our baseline, so that the review can focus on the core of the contribution.

adriaanm and others added 2 commits April 4, 2018 15:55
@milessabin milessabin force-pushed the topic/bounds-propagation branch from 1b81615 to 0a78e27 Compare April 4, 2018 15:15
@milessabin
Copy link
Contributor Author

Squashed and rebased onto @adriaanm's cleanup.

@adriaanm
Copy link
Contributor

adriaanm commented Apr 5, 2018

To avoid causing more back-and-forth, I'll adopt the part of this PR I agree with. I don't think we should go look in the type arguments. Closing in favor of #6492

@adriaanm adriaanm closed this Apr 5, 2018
@milessabin
Copy link
Contributor Author

That's disappointing. We'll continue in Typelevel Scala.

@SethTisue SethTisue removed this from the 2.13.0-M4 milestone May 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants