Skip to content

Invalid bytecode generated for deserialization when many lambdas are declared #10232

New issue
New issue
Closed
Closed
Invalid bytecode generated for deserialization when many lambdas are declared#10232
Assignees
retronym
Labels
runtime crashserialization
Milestone
2.12.3
@scabug

Description

@scabug
scabug
opened on Mar 17, 2017

Deserialization is broken when many lambdas are declared in the same class. The problem is that that the compiler seems to be generating bytecode for deserialization logic that runs into some JVM limitations.

How to reproduce

E.g.:

import java.io.{ByteArrayInputStream, ByteArrayOutputStream, ObjectInputStream, ObjectOutputStream}

object A {
	val surplus: List[Any => String] = List(
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" },
	{ (t: Any) => "ab" }
		)

  def main(args: Array[String]): Unit = {
  	val lambda = surplus.head
    val outStream = new ByteArrayOutputStream
    val oo = new ObjectOutputStream(outStream)
    oo.writeObject(lambda)

    val inStream = new ByteArrayInputStream(outStream.toByteArray)
    val oi = new ObjectInputStream(inStream)
    val lambda2 = oi.readObject().asInstanceOf[Any => String]
    println(lambda2(1))
  }
}

fails with:

[gkk@mbp ~/tmp/scalac-lambda-surplus]$ scala A
java.io.IOException: unexpected exception type
	at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1582)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1154)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2022)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
	at A$.main(A.scala:271)
	at A.main(A.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.reflect.internal.util.ScalaClassLoader.$anonfun$run$2(ScalaClassLoader.scala:98)
	at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:32)
	at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:30)
	at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:129)
	at scala.reflect.internal.util.ScalaClassLoader.run(ScalaClassLoader.scala:98)
	at scala.reflect.internal.util.ScalaClassLoader.run$(ScalaClassLoader.scala:90)
	at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:129)
	at scala.tools.nsc.CommonRunner.run(ObjectRunner.scala:22)
	at scala.tools.nsc.CommonRunner.run$(ObjectRunner.scala:21)
	at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:39)
	at scala.tools.nsc.CommonRunner.runAndCatch(ObjectRunner.scala:29)
	at scala.tools.nsc.CommonRunner.runAndCatch$(ObjectRunner.scala:28)
	at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:39)
	at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:61)
	at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:88)
	at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:99)
	at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:104)
	at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1148)
	... 27 more
Caused by: java.lang.BootstrapMethodError: too many bootstrap method arguments
	at java.lang.invoke.CallSite.makeSite(CallSite.java:320)
	at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307)
	at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297)
	at A$.$deserializeLambda$(A.scala)
	... 37 more

How to work around

Given that this seems to be per-class limit. Moving lambdas to different classes (e.g. introducing an inner class for declaring lambdas) is a possible work-around.

It's really problematic that you find out about broken bytecode only at runtime when deserialization is executed.

This problem has been found in scalding-core during Stripe's experiment to update to Scala 2.12 (we're curious to see whether the new optimizer helps our ML infra perf).

Metadata

Metadata

Assignees

Labels

runtime crashserialization

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions