Skip to content

Adapt to REPL changes planned for Scala 2.12.11#1

Open
retronym wants to merge 2 commits intomasterfrom
topic/pr-snapshots
Open

Adapt to REPL changes planned for Scala 2.12.11#1
retronym wants to merge 2 commits intomasterfrom
topic/pr-snapshots

Conversation

@retronym
Copy link
Owner

@retronym retronym commented Feb 12, 2020 •
edited
Loading

The renovations of -Yrepl-class-based (scala/scala#8712) require opting into the the newly added -Yrepl-use-magic-imports (scala/scala#8576).

@retronym
Add profile to enable testing builds of Scala from scala/scala's CI
391ba6d 
For instance:

```
 mvn -P scala-pr-validation-snapshots -Dscala.version=2.12.11-bin-ee22df6-SNAPSHOT clean install
```
@retronym
Enable -Yrepl-use-magic-imports in compiler versions with that setting
35e8500 
This not only reduces the bloat of the REPL wrapper code, but
is also necessary to keep the Spark REPL test suites passing
after the Scala REPLs implementation of -Yrepl-class-based has
been updated in Scala 2.12.11.

Tested with:

```
./build/mvn -P scala-pr-validation-snapshots -Dscala.version=2.12.10 -pl :spark-repl_2.12 -Dtest=none -DwildcardSuites='org.apache.spark.repl.ReplSuite,org.apache.spark.repl.SingletonReplSuite,org.apache.spark.repl.ExecutorClassLoaderSuite' test
./build/mvn -P scala-pr-validation-snapshots -Dscala.version=2.12.11-bin-ee22df6-SNAPSHOT -pl :spark-repl_2.12 -Dtest=none -DwildcardSuites='org.apache.spark.repl.ReplSuite,org.apache.spark.repl.SingletonReplSuite,org.apache.spark.repl.ExecutorClassLoaderSuite' test
```
@retronym retronym changed the title Add profile to enable testing builds of Scala from scala/scala's CI Adapt to REPL changes planned for Scala 2.12.11 Feb 13, 2020
@retronym
Copy link
Owner Author

retronym commented Feb 13, 2020

Test with:

./build/mvn -P scala-pr-validation-snapshots -Dscala.version=2.12.11-bin-ee22df6-SNAPSHOT -pl :spark-repl_2.12 -Dtest=none -DwildcardSuites='org.apache.spark.repl.ReplSuite,org.apache.spark.repl.SingletonReplSuite,org.apache.spark.repl.ExecutorClassLoaderSuite' test

@retronym retronym mentioned this pull request Feb 13, 2020
Merged
1 task
@dwijnand
Copy link

dwijnand commented Feb 13, 2020

Failures when using that 2.12.11 build and without the magic imports (cc @lrytz):

SingletonReplSuite:
Spark context available as 'sc' (master = local-cluster[2,1,1024], app id = app-20200213105349-0000).
Spark session available as 'spark'.
- simple foreach with accumulator
- external vars

- external classes *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala>      |      | defined class C

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 12, grey.mynet, executor 1): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$res$1(<console>:24)
  	at scala.runtime.java8.JFunction1$mcII$sp.apply(JFunction1$mcII$sp.java:23)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at scala.collection.Iterator.foreach(Iterator.scala:943)
  	at scala.collection.Iterator.foreach$(Iterator.scala:943)
  	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
  	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
  	at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
  	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
  	at scala.collection.AbstractIterator.to(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
  	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
  	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
  	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
  	at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
  	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
  	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 35 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$res$1(<console>:24)
    at scala.runtime.java8.JFunction1$mcII$sp.apply(JFunction1$mcII$sp.java:23)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    at scala.collection.AbstractIterator.to(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
    at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
    at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>      | _result_1581591233942: Int = 1

  scala> (SingletonReplSuite.scala:106)

- external functions
- external functions that access vars
- broadcast vars
- interacting with files
- local-cluster mode
- SPARK-1199 two instances of same class don't type check.
- SPARK-2452 compound statements.

- SPARK-2576 importing implicits *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> import spark.implicits._

  scala> defined class TestCaseClass

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 16.0 failed 4 times, most recent failure: Lost task 0.3 in stage 16.0 (TID 44, grey.mynet, executor 1): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$res15$1(<console>:30)
  	at $anonfun$res15$1$adapted(<console>:30)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
  	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
  	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:339)
  	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
  	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
  	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
  	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
  	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:384)
    at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3507)
    at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2831)
    at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3497)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:102)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:162)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:89)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3495)
    at org.apache.spark.sql.Dataset.collect(Dataset.scala:2831)
    ... 37 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$res15$1(<console>:30)
    at $anonfun$res15$1$adapted(<console>:30)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:339)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>
  scala>
  scala> res16: Array[TestCaseClass] = Array(TestCaseClass(1))

  scala>      | _result_1581591240142: Int = 1

  scala> (SingletonReplSuite.scala:106)

- Datasets and encoders

- SPARK-2632 importing a method from non serializable class and not using it. *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> defined class TestClass

  scala> t: TestClass = TestClass@1bbf2d84

  scala> import t.testMethod

  scala> defined class TestCaseClass

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 19.0 failed 4 times, most recent failure: Lost task 0.3 in stage 19.0 (TID 55, grey.mynet, executor 1): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$res20$1(<console>:38)
  	at $anonfun$res20$1$adapted(<console>:38)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at scala.collection.Iterator.foreach(Iterator.scala:943)
  	at scala.collection.Iterator.foreach$(Iterator.scala:943)
  	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
  	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
  	at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
  	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
  	at scala.collection.AbstractIterator.to(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
  	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
  	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
  	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
  	at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
  	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
  	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 39 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$res20$1(<console>:38)
    at $anonfun$res20$1$adapted(<console>:38)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    at scala.collection.AbstractIterator.to(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
    at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
    at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>      | _result_1581591243727: Int = 1

  scala> (SingletonReplSuite.scala:106)

- collecting objects of class defined in repl *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> defined class Foo

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 57, not attempting to retry it. Exception during serialization: java.io.NotSerializableException: TestClass
  Serialization stack:
  	- object not serializable (class: TestClass, value: TestClass@1bbf2d84)
  	- field (class: $iw, name: t, type: class TestClass)
  	- object (class $iw, $iw@623a8812)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@2a0edbaa)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@a82317a)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@ba68685)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@31f4cc52)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@429285ed)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@5e1d4d)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3dea2a3e)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3fd0c411)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@71a6d5b5)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@728ae1fc)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@121b000e)
  	- field (class: $read, name: $iw, type: class $iw)
  	- object (class $read, $read@5cb921d7)
  	- field (class: $iw, name: read, type: class $read)
  	- object (class $iw, $iw@68fdc77c)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@6f6837e3)
  	- field (class: Foo, name: $outer, type: class $iw)
  	- object (class Foo, Foo(1))
  	- element of array (index: 0)
  	- array (class [LFoo;, size 10)
  	- field (class: scala.collection.mutable.WrappedArray$ofRef, name: array, type: class [Ljava.lang.Object;)
  	- object (class scala.collection.mutable.WrappedArray$ofRef, WrappedArray(Foo(1), Foo(2), Foo(3), Foo(4), Foo(5), Foo(6), Foo(7), Foo(8), Foo(9), Foo(10)))
  	- writeObject data (class: org.apache.spark.rdd.ParallelCollectionPartition)
  	- object (class org.apache.spark.rdd.ParallelCollectionPartition, org.apache.spark.rdd.ParallelCollectionPartition@cf9)
  	- field (class: org.apache.spark.scheduler.ResultTask, name: partition, type: interface org.apache.spark.Partition)
  	- object (class org.apache.spark.scheduler.ResultTask, ResultTask(20, 0))
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 39 elided

  scala>      | _result_1581591244762: Int = 1

  scala> (SingletonReplSuite.scala:106)

- collecting objects of class defined in repl - shuffling *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> defined class Foo

  scala> list: List[(Int, Foo)] = List((1,Foo(1)), (1,Foo(2)))

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 58, not attempting to retry it. Exception during serialization: java.io.NotSerializableException: TestClass
  Serialization stack:
  	- object not serializable (class: TestClass, value: TestClass@1bbf2d84)
  	- field (class: $iw, name: t, type: class TestClass)
  	- object (class $iw, $iw@623a8812)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@2a0edbaa)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@a82317a)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@ba68685)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@31f4cc52)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@429285ed)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@5e1d4d)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3dea2a3e)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3fd0c411)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@71a6d5b5)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@728ae1fc)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@121b000e)
  	- field (class: $read, name: $iw, type: class $iw)
  	- object (class $read, $read@5cb921d7)
  	- field (class: $iw, name: read, type: class $read)
  	- object (class $iw, $iw@253e26ef)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@4d910314)
  	- field (class: Foo, name: $outer, type: class $iw)
  	- object (class Foo, Foo(1))
  	- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
  	- object (class scala.Tuple2, (1,Foo(1)))
  	- element of array (index: 0)
  	- array (class [Lscala.Tuple2;, size 1)
  	- field (class: scala.collection.mutable.WrappedArray$ofRef, name: array, type: class [Ljava.lang.Object;)
  	- object (class scala.collection.mutable.WrappedArray$ofRef, WrappedArray((1,Foo(1))))
  	- writeObject data (class: org.apache.spark.rdd.ParallelCollectionPartition)
  	- object (class org.apache.spark.rdd.ParallelCollectionPartition, org.apache.spark.rdd.ParallelCollectionPartition@d22)
  	- field (class: org.apache.spark.scheduler.ShuffleMapTask, name: partition, type: interface org.apache.spark.Partition)
  	- object (class org.apache.spark.scheduler.ShuffleMapTask, ShuffleMapTask(21, 0))
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 39 elided

  scala>      | _result_1581591245289: Int = 1

  scala> (SingletonReplSuite.scala:106)

- replicating blocks of object with class defined in repl *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> timeout: Int = 60000

  scala> start: Long = 1581591245957

  scala>      |      |      |
  scala>      |      |
  scala> import org.apache.spark.storage.StorageLevel._

  scala> defined class Foo

  scala> ret: org.apache.spark.rdd.RDD[Foo] = ParallelCollectionRDD[43] at parallelize at <console>:39

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 59, not attempting to retry it. Exception during serialization: java.io.NotSerializableException: TestClass
  Serialization stack:
  	- object not serializable (class: TestClass, value: TestClass@1bbf2d84)
  	- field (class: $iw, name: t, type: class TestClass)
  	- object (class $iw, $iw@623a8812)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@2a0edbaa)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@a82317a)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@ba68685)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@31f4cc52)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@429285ed)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@5e1d4d)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3dea2a3e)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3fd0c411)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@71a6d5b5)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@728ae1fc)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@121b000e)
  	- field (class: $read, name: $iw, type: class $iw)
  	- object (class $read, $read@5cb921d7)
  	- field (class: $iw, name: read, type: class $read)
  	- object (class $iw, $iw@2e516148)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@78e16ce)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@86a622b)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@6db87f43)
  	- field (class: Foo, name: $outer, type: class $iw)
  	- object (class Foo, Foo(1))
  	- element of array (index: 0)
  	- array (class [LFoo;, size 10)
  	- field (class: scala.collection.mutable.WrappedArray$ofRef, name: array, type: class [Ljava.lang.Object;)
  	- object (class scala.collection.mutable.WrappedArray$ofRef, WrappedArray(Foo(1), Foo(2), Foo(3), Foo(4), Foo(5), Foo(6), Foo(7), Foo(8), Foo(9), Foo(10)))
  	- writeObject data (class: org.apache.spark.rdd.ParallelCollectionPartition)
  	- object (class org.apache.spark.rdd.ParallelCollectionPartition, org.apache.spark.rdd.ParallelCollectionPartition@d74)
  	- field (class: org.apache.spark.scheduler.ResultTask, name: partition, type: interface org.apache.spark.Partition)
  	- object (class org.apache.spark.scheduler.ResultTask, ResultTask(23, 0))
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1227)
    ... 41 elided

  scala> res: Int = 0

  scala>      | _result_1581591245815: Int = 1

  scala> (SingletonReplSuite.scala:106)

- should clone and clean line object in ClosureCleaner *** FAILED ***
  isContain was true Interpreter output contained 'AssertionError':

  scala> import org.apache.spark.rdd.RDD

  scala>
  scala> lines: org.apache.spark.rdd.RDD[String] = pom.xml MapPartitionsRDD[45] at textFile at <console>:38

  scala> defined class Data

  scala> dataRDD: org.apache.spark.rdd.RDD[Data] = MapPartitionsRDD[46] at map at <console>:40

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 24.0 failed 4 times, most recent failure: Lost task 0.3 in stage 24.0 (TID 66, grey.mynet, executor 0): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$dataRDD$1(<console>:40)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
  	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
  	at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
  	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
  	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
  	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
  	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
  	at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1227)
    ... 41 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$dataRDD$1(<console>:40)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
    at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
    at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala> repartitioned: org.apache.spark.rdd.RDD[Data] = MapPartitionsRDD[50] at repartition at <console>:38

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 25.0 failed 4 times, most recent failure: Lost task 0.3 in stage 25.0 (TID 75, grey.mynet, executor 1): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$dataRDD$1(<console>:40)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
  	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
  	at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
  	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
  	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
  	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
  	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
  	at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
  	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
  	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
  	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
  	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
  	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
  	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1227)
    ... 41 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$dataRDD$1(<console>:40)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
    at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
    at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>
  scala>      |      | getCacheSize: (rdd: org.apache.spark.rdd.RDD[_])Long

  scala> cacheSize1: Long = 0

  scala> cacheSize2: Long = 0

  scala>
  scala>
  scala> deviation: Double = NaN

  scala>      | java.lang.AssertionError: assertion failed: deviation too large: NaN, first size: 0, second size: 0
    at scala.Predef$.assert(Predef.scala:223)
    ... 42 elided

  scala>      | _result_1581591246847: Int = 1

  scala> (SingletonReplSuite.scala:106)

- newProductSeqEncoder with REPL defined class

- create encoder in executors *** FAILED ***
  isContain was false Interpreter output did not contain 'r: Foo = Foo(bar)':

  scala> defined class Foo

  scala>
  scala> import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder

  scala>
  scala>      | org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 27.0 failed 4 times, most recent failure: Lost task 1.3 in stage 27.0 (TID 80, grey.mynet, executor 0): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$r$1(<console>:42)
  	at $anonfun$r$1$adapted(<console>:42)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at scala.collection.Iterator.foreach(Iterator.scala:943)
  	at scala.collection.Iterator.foreach$(Iterator.scala:943)
  	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
  	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
  	at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
  	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
  	at scala.collection.AbstractIterator.to(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
  	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
  	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
  	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
  	at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
  	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
  	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 42 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$r$1(<console>:42)
    at $anonfun$r$1$adapted(<console>:42)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    at scala.collection.AbstractIterator.to(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
    at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
    at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>      | _result_1581591249369: Int = 1

  scala> (SingletonReplSuite.scala:100)

@dwijnand
Copy link

dwijnand commented Feb 13, 2020

I wasn't able to extract from that how we can reproduce that in a test in our suite. 😕

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@retronym @dwijnand