Parse `+_` and `-_` in types as identifiers under `-Xsource:3` to support Scala 3.2 placeholder syntax by neko-kai · Pull Request #9605 · scala/scala

neko-kai · 2021-05-03T22:27:51Z

This change allows kind-projector plugin to rewrite +_ and -_ tokens to type lambdas,
in line with proposed syntax for Scala 3.2 in http://dotty.epfl.ch/docs/reference/changed-features/wildcards.html

See also https://scalacenter.github.io/scala-3-migration-guide/docs/tutorials/kind-projector.html (Soon on https://docs.scala-lang.org/scala3/guides/migration/compatibility-intro.html)

When used in conjunction with -Xsource:3 this will let the user use ? for wildcards and _ for placeholders, letting the user cross-compile the same sources with Scala 3 with -source:3.2 flag.

This change is not source breaking since currently +_ and -_ fail to parse entirely,
this change also does not allow the user to declare types with these names without backticks,
they can only be used as part of a type tree.

This does not add a special error when these symbols are used without kind-projector plugin – just "identifier not found", which is however in line with the behavior of other symbols currently used as placeholders in kind-projector – ? & *

See also:

Original issue in kind-projector repository Scala 3 mode with _ as a placeholder and ? as wildcard? typelevel/kind-projector#120

…eholder syntax This change allows `kind-projector` plugin to rewrite `+_` and `-_` tokens to type lambdas, in line with proposed syntax for Scala 3.2 in http://dotty.epfl.ch/docs/reference/changed-features/wildcards.html When used in conjunction with `-Xsource:3` this will let the user use `?` for wildcards and `_` for placeholders, letting the user cross-compile the same sources with Scala 3 with `-source:3.2` flag. This change is not source breaking since currently `+_` and `-_` fail to parse entirely, this change also does not allow the user to declare types with these names without backticks, they can only be used as part of a type tree.

smarter

This change allows kind-projector plugin to rewrite +_ and -_ tokens to type lambdas,
in line with proposed syntax for Scala 3.2 in http://dotty.epfl.ch/docs/reference/changed-features/wildcards.html

This page mentions _ but not +_ or -_, and I don't believe the latter two will become syntax in a future Scala 3 release, because we already disallowed variance annotations on type lambdas:

class Foo[F[+_]]
def foo(x: Foo[[+X] =>> Either[Int, X]]): Unit = {}
//              ^
//              no `+/-` variance annotation allowed here

This was disallowed because the compiler now infers the variance of a type lambda from its body, so the following works:

def foo(x: Foo[[X] =>> Either[Int, X]]): Unit = {}

Therefore in the future we can expect the following syntax to work too:

def foo(x: Foo[Either[Int, _]]): Unit = {}

But not a hypothetical syntax with +_.

neko-kai · 2021-05-04T12:54:53Z

@smarter

This page mentions _ but not +_ or -_, and I don't believe the latter two will become syntax in a future Scala 3 release, because we already disallowed variance annotations on type lambdas:

I was not aware of this change! However, they can probably be allowed to parse and be treated as just _ under -Ykind-projector for compatibility – along with +* and -*, that were omitted in initial -Ykind-projector PR – and that are required for cross-compiling variant type lambdas anyway, since 2.12 and 2.13 unfortunately do not infer variance by default.

The other glaring thing I was not aware of is that dotty doesn't actually implement underscore syntax right now. I was fooled by scastie! It was silently crashing on old versions of dotty when flag -source:3.2 was passed, giving the impression that source compiled correctly. There was never actually a -source:3.2 mode at all, let alone with underscores. 🤦‍♀️ (https://scastie.scala-lang.org/AAoaybtxQYaM40EsEl9gYA)

smarter · 2021-05-04T12:59:45Z

However, they can probably be allowed to parse and be treated as just _ under -Ykind-projector for compatibility – along with +* and -*, that were omitted in initial -Ykind-projector PR

That's one possibility, but it seems weird to me to introduce a new syntax that never existed in kind-projector under -Ykind-projector, the goal of which is to help existing code to compile, not to enable new ways to write code.

neko-kai · 2021-05-04T14:13:40Z

@smarter

That's one possibility, but it seems weird to me to introduce a new syntax that never existed in kind-projector under -Ykind-projector, the goal of which is to help existing code to compile, not to enable new ways to write code.

It seems weird now, but if this PR gets merged, then kind-projector will have such syntax – Scala 2 projects in the wild will use it and then this rule relaxation in -Ykind-projector will indeed help existing Scala 2 code to compile. Note that this would not enable new ways to write code, only accept existing Scala 2 constructs written in future syntax (that would have existed and made sense in Scala 3 if it was not obviated by variance inference).

SethTisue · 2021-05-07T15:14:35Z

It looks like this isn't actually gatewayed by -Xsource:3? Was that just an oversight?

You could argue that it doesn't matter, but especially since the new syntax is a no-op, it feels weird to me to simply accept it by default; I expected to have to opt-in.

dwijnand

Complicated matters, but I think I've understood the situation so this change LGTM.

@smarter am I right in taking your merging of the Dotty PR as agreeing to this change too?

smarter · 2021-05-07T15:49:47Z

The Dotty PR was about +* and -*, not about +_ and -_, I still stand by #9605 (comment), but I don't have strong feelings either way as long as I don't have to document this :).

dwijnand · 2021-05-07T16:24:36Z

Good point. With * deprecated in 3.2 and removed in 3.3, this PR looks to finding a way to compile code with Scala 2.13 with -Xsource:3 and Scala 3 with -source:3.2.

So basically it only makes sense if both this PR and a Scala 3 "be allowed to parse and be treated as just _ under -Ykind-projector for compatibility" PR is merged, right?

smarter · 2021-05-07T16:29:30Z

With * deprecated in 3.2 and removed in 3.3

Note that the timeline in http://dotty.epfl.ch/docs/reference/changed-features/wildcards.html is just there to give a rough idea of how things could go, it's not a fixed plan (in particular we may bump our minor version at any time just to signal that we broke forward binary compatibility).

So basically it only makes sense if both this PR and a Scala 3 "be allowed to parse and be treated as just _ under -Ykind-projector for compatibility" PR is merged, right?

Yep.

neko-kai · 2021-05-07T17:28:46Z

@SethTisue

It looks like this isn't actually gatewayed by -Xsource:3? Was that just an oversight?

You could argue that it doesn't matter, but especially since the new syntax is a no-op, it feels weird to me to simply accept it by default; I expected to have to opt-in.

My reasoning for not gating was that it's not directly related to -Xsource:3 - instead it enables a kind-projector feature which may be used independently of -Xsource:3 (although without a ? substitute you would just lose any way to write a wildcard).
Although, I have no strong opinion and would gate or not depending on what you decide basically.

@dwijnand

Good point. With * deprecated in 3.2 and removed in 3.3, this PR looks to finding a way to compile code with Scala 2.13 with -Xsource:3 and Scala 3 with -source:3.2.

Yes, the timeline assumes a period when * is disabled, rendering cross-compilation impossible. My goal is to just jump straight to the end of the timeline and use _ everywhere in my libraries without losing cross-compilation (using appropriate future-enabling flags in builds).

So basically it only makes sense if both this PR and a Scala 3 "be allowed to parse and be treated as just _ under -Ykind-projector for compatibility" PR is merged, right?

Well, I assumed Scala 3 already implements underscores, but I was wrong.
However there is a public plan to do this, so it will happen eventually, that's why I consider this change meaningful for cross-compilation even if the current RC or 3.0 final may not have underscores – because one of the next versions will eventually have them.

@smarter I can submit a quick PR that will allow use of _/+_/-_ as type lambda placeholders in -source:future using current the -Ykind-projector codepath. However this will not close scala/scala3#5379, because there's a question of what should F[_[_]] mean – in kind-projector this means [A[_]] =>> F[A], but in value-level lambdas f(_(_)) means f((a, b) => a(b)), so to avoid deciding anything on this question I would only be able to support *-kinded variables, in line with what -Ykind-projector supports right now.

By the way I've added the change to kind-projector typelevel/kind-projector#188 & verified that it works for my usecase 7mind/izumi#1479

smarter · 2021-05-07T17:41:35Z

I can submit a quick PR that will allow use of _/+_/-_ as type lambda placeholders in -source:future using current the -Ykind-projector codepath.

kind-projector repurposing _ right now would break a lot of existing code, to do that I think we'd need a deprecation period (in kind-projector or scala 2 itself under the -Xsource:3 flag) encouraging folks to switch to ? for wildcards.

neko-kai · 2021-05-07T17:46:51Z

kind-projector repurposing _ right now would break a lot of existing code, to do that I think we'd need a deprecation period (in kind-projector or scala 2 itself under the -Xsource:3 flag) encouraging folks to switch to ? for wildcards.

I thought Dotty's -source:future already disables _ for wildcards, so there is no syntax overlap?

For Scala 2 with typelevel/kind-projector#188 it repurposes _ only if the option -P:kind-projector:underscore-placeholders is passed, so my thinking is to get new syntax you would use:

in Scala 2 - -Xsource:3 -P:kind-projector:underscore-placeholders
in Scala 3 - -source:future

And if you pass no flags then nothing changes (And you'll get new syntax only in Scala 3.2/3.3 when it becomes the default)

smarter · 2021-05-07T17:50:53Z

I thought Dotty's -source:future already disables _ for wildcards, so there is no syntax overlap?

It only deprecates it for now.

it repurposes _ only if the option -P:kind-projector:underscore-placeholders

That could work for now yeah.

neko-kai · 2021-05-07T17:56:14Z

@smarter

It only deprecates it for now.

Got it, what about a new -Y flag, such as -Yunderscore-placeholders or a sub-option in -Ykind-projector such as -Ykind-projector:underscores?

I think that -source:3.2 could also work for that case, but there seems to have been a movement back to future instead of version numbers, and I don't think -source:future-future would work.

smarter · 2021-05-07T18:00:30Z

Honestly not sure, t'm wary of multiplying the flags since it makes everything more complicated, I guess we could go for the sub-option for now.

neko-kai · 2021-05-07T21:50:01Z

@smarter Submitted at scala/scala3#12378

…ption to existing code

neko-kai · 2021-05-08T23:12:00Z

I've decided to gate the change behind -Xsource:3 to minimize any possible disruption to the majority of existing code

lrytz · 2021-05-10T12:36:13Z

as I don't have to document this :)

Resonates with me :-) @neko-kai could you take this on? Would be really helpful to have a page that explains the status quo, the goal, and all the possible paths through the forest of combinations of

(wildcard, placeholder) x
(variance) x
(2.13 2.13 -Xsource:3, 3.0, 3, 3 -source:3.1, 3 -source:3.2) x
(-P:kind-projector:underscore-placeholders) x
(-Yunderscore-placeholders)

@smarter where should that documentation go?

smarter · 2021-05-10T12:38:42Z

@smarter where should that documentation go?

Great question, maybe a dedicated page in the migration guide ? /cc @adpi2

neko-kai · 2021-05-10T12:40:02Z

@lrytz I've documented the Dotty part in the "Changes to wildcards", I've also expanded kind-projector README in the related kind-projector PR.
For scalac obviously this will be mentioned in release notes, but I don't know where to put the rest though (maybe just in comments or descriptions related to -Xsource:3 ?)

lrytz · 2021-05-10T12:51:08Z

Since this is mainly about migration to Scala 3, I think the migration guide is a good place. We can always link to it from elsewhere. It would be really good to have it all in one place.

I've documented the Dotty part in the "Changes to wildcards",

Where's that?

expanded kind-projector README

https://github.com/typelevel/kind-projector/pull/188/files?

neko-kai · 2021-05-10T12:53:31Z

@lrytz

Where's that?

https://github.com/lampepfl/dotty/pull/12378/files#diff-08f1a956e9a74ed84ae35bea06852879c16776956f075ec5f5b5748e117c6ecaR46
Here, but admittedly this probably deserves a separate page in migration guide

https://github.com/typelevel/kind-projector/pull/188/files?

Yep

dwijnand · 2021-05-10T14:21:30Z

So basically it only makes sense if both this PR and a Scala 3 "be allowed to parse and be treated as just _ under -Ykind-projector for compatibility" PR is merged, right?

Yep.

Got a verbal ok for that Scala 3 part being acceptable, so that's a re-✅ from me.

neko-kai · 2021-05-10T20:37:00Z

Migration guide updated at scalacenter/scala-3-migration-guide#203

scala-jenkins added this to the 2.13.6 milestone May 3, 2021

neko-kai mentioned this pull request May 3, 2021

Scala 3 mode with _ as a placeholder and ? as wildcard? typelevel/kind-projector#120

Closed

SethTisue added the release-notes worth highlighting in next release notes label May 3, 2021

smarter requested changes May 4, 2021

View reviewed changes

neko-kai mentioned this pull request May 5, 2021

Support variance annotations in -Ykind-projector mode scala/scala3#12341

Merged

Fix a bug with +_/-_ not parsing in type constructor position

946b129

dwijnand approved these changes May 7, 2021

View reviewed changes

This was referenced May 7, 2021

Swap ? placeholder and _ wildcard using kind-projector 0.13.0 & scalac 2.13.6 7mind/izumi#1479

Merged

Add option to use underscore symbol _ instead of * to define anonymous type lambdas typelevel/kind-projector#188

Merged

This was referenced May 7, 2021

Allow _ as a type lambda placeholder in -Ykind-projector:underscores compatiblity mode scala/scala3#12378

Merged

Differentiate Scala 2 and Scala 3 wildcard identifier names #9614

Merged

Gate -_/+_ parsing behind -Xsource:3 to guarantee minimal disru…

196c656

…ption to existing code

neko-kai mentioned this pull request May 8, 2021

[backport] Underscore changes for -Xsource:3: parse +_/-_ and prefix wildcard variables introduced by ?-syntax with ? #9620

Merged

Merge branch '2.13.x' into parse-variant-underscore-placeholders

0a00169

lrytz approved these changes May 10, 2021

View reviewed changes

lrytz merged commit 8e4f76b into scala:2.13.x May 10, 2021

neko-kai mentioned this pull request May 10, 2021

Document migration to Scala 3 Underscore Type Lambdas syntax and cross-compiling with kind-projector scalacenter/scala-3-migration-guide#203

Merged

dwijnand changed the title ~~Parse +_ and -_ in types as identifiers to support Scala 3.2 placeholder syntax~~ Parse +_ and -_ in types as identifiers under -Xsource:3 to support Scala 3.2 placeholder syntax May 14, 2021

neko-kai mentioned this pull request May 17, 2021

Parse underscore type lambdas (And +_ and -_ symbols) scalameta/scalameta#2347

Closed

dos65 mentioned this pull request May 24, 2021

Allow -_ and +_ as type names for -Xsource3 flag scalameta/scalameta#2348

Merged

Conversation

neko-kai commented May 3, 2021 • edited by lrytz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

neko-kai commented May 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter commented May 4, 2021

Uh oh!

neko-kai commented May 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SethTisue commented May 7, 2021

Uh oh!

dwijnand left a comment

Choose a reason for hiding this comment

Uh oh!

smarter commented May 7, 2021

Uh oh!

dwijnand commented May 7, 2021

Uh oh!

smarter commented May 7, 2021

Uh oh!

neko-kai commented May 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter commented May 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neko-kai commented May 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter commented May 7, 2021

Uh oh!

neko-kai commented May 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter commented May 7, 2021

Uh oh!

neko-kai commented May 7, 2021

Uh oh!

neko-kai commented May 8, 2021

Uh oh!

lrytz commented May 10, 2021

Uh oh!

smarter commented May 10, 2021

Uh oh!

neko-kai commented May 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lrytz commented May 10, 2021

Uh oh!

neko-kai commented May 10, 2021

Uh oh!

dwijnand commented May 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neko-kai commented May 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

neko-kai commented May 3, 2021 •

edited by lrytz

Loading

smarter left a comment •

edited

Loading

neko-kai commented May 4, 2021 •

edited

Loading

neko-kai commented May 4, 2021 •

edited

Loading

neko-kai commented May 7, 2021 •

edited

Loading

smarter commented May 7, 2021 •

edited

Loading

neko-kai commented May 7, 2021 •

edited

Loading

neko-kai commented May 7, 2021 •

edited

Loading

neko-kai commented May 10, 2021 •

edited

Loading

dwijnand commented May 10, 2021 •

edited

Loading