Skip to content

[VL] Bad optional access with lag function in Window #7631

@zhztheplayer

Description

@zhztheplayer

Test name:

SPARK-16633: lead/lag should return the default value if the offset row does not exist

Error:

2024-10-22T06:07:14.2572740Z - SPARK-16633: lead/lag should return the default value if the offset row does not exist *** FAILED ***
2024-10-22T06:07:14.2573967Z   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 152.0 failed 1 times, most recent failure: Lost task 0.0 in stage 152.0 (TID 109) (da070ca603ce executor driver): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: bad optional access
2024-10-22T06:07:14.2574400Z 	at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
2024-10-22T06:07:14.2574708Z 	at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
2024-10-22T06:07:14.2575092Z 	at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
2024-10-22T06:07:14.2575433Z 	at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
2024-10-22T06:07:14.2575751Z 	at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
2024-10-22T06:07:14.2576103Z 	at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
2024-10-22T06:07:14.2576371Z 	at scala.collection.Iterator.isEmpty(Iterator.scala:387)
2024-10-22T06:07:14.2576631Z 	at scala.collection.Iterator.isEmpty$(Iterator.scala:387)
2024-10-22T06:07:14.2576979Z 	at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.isEmpty(IteratorsV1.scala:90)
2024-10-22T06:07:14.2577467Z 	at org.apache.gluten.execution.VeloxColumnarToRowExec$.toRowIterator(VeloxColumnarToRowExec.scala:121)
2024-10-22T06:07:14.2577935Z 	at org.apache.gluten.execution.VeloxColumnarToRowExec.$anonfun$doExecuteInternal$1(VeloxColumnarToRowExec.scala:77)
2024-10-22T06:07:14.2578172Z 	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:856)
2024-10-22T06:07:14.2578432Z 	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:856)
2024-10-22T06:07:14.2578719Z 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
2024-10-22T06:07:14.2578973Z 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
2024-10-22T06:07:14.2579140Z 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
2024-10-22T06:07:14.2579421Z 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
2024-10-22T06:07:14.2579678Z 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
2024-10-22T06:07:14.2579844Z 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
2024-10-22T06:07:14.2580213Z 	at org.apache.spark.sql.execution.SQLExecutionRDD.$anonfun$compute$1(SQLExecutionRDD.scala:52)
2024-10-22T06:07:14.2580502Z 	at org.apache.spark.sql.internal.SQLConf$.withExistingConf(SQLConf.scala:158)
2024-10-22T06:07:14.2580827Z 	at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:52)
2024-10-22T06:07:14.2581078Z 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
2024-10-22T06:07:14.2581243Z 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
2024-10-22T06:07:14.2581519Z 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
2024-10-22T06:07:14.2581766Z 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
2024-10-22T06:07:14.2581931Z 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
2024-10-22T06:07:14.2582190Z 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
2024-10-22T06:07:14.2582484Z 	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
2024-10-22T06:07:14.2582672Z 	at org.apache.spark.scheduler.Task.run(Task.scala:139)
2024-10-22T06:07:14.2582971Z 	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
2024-10-22T06:07:14.2583216Z 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)
2024-10-22T06:07:14.2583472Z 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
2024-10-22T06:07:14.2583808Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2024-10-22T06:07:14.2584104Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2024-10-22T06:07:14.2584230Z 	at java.lang.Thread.run(Thread.java:748)
2024-10-22T06:07:14.2584520Z Caused by: org.apache.gluten.exception.GlutenException: bad optional access
2024-10-22T06:07:14.2584876Z 	at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
2024-10-22T06:07:14.2585292Z 	at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
2024-10-22T06:07:14.2585674Z 	at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
2024-10-22T06:07:14.2585760Z 	... 35 more

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions