[GLUTEN-11088] Fix GlutenDataFrameFunctionsSuite in Spark-4.0#11195
[GLUTEN-11088] Fix GlutenDataFrameFunctionsSuite in Spark-4.0#11195zhouyuan merged 3 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
| Row(null) | ||
| ) | ||
|
|
||
| intercept[SparkException](df1.selectExpr("map_concat(map1, map2)").collect()) |
There was a problem hiding this comment.
I think there are similar tests found when i work on the first patch to enable unit tests. Is it too heavy to modify the map_concat function to throw the right exception?
There was a problem hiding this comment.
I haven't found anything similar yet. Would the impact be greater if GlutenException were changed to SparkRuntimeException?
There was a problem hiding this comment.
Can GlutenException inherit SparkRuntimeException? Now it inherits RuntimeException, I'm not sure if it is more feasible.
There was a problem hiding this comment.
If it inherits SparkRuntimeException, it throws GlutenException, not SparkRuntimeException.
https://github.com/apache/spark/blob/29434ea766b0fc3c3bf6eaadb43a8f931133649e/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L2928-L2937
Vanilla spark throw SparkRuntimeException, gluten throw SparkException.
Related issue: #11088