[VL] Track on Spark-4.1.x failed unit tests #11400
Description
Description
After #11353 and #11380. We still have failed unit tests left here:
legend
✔️ Fixed
🚫 Blocker, must be addressed
‼️ Need Help from community
❌️ No need support until now
| Description | Owner | Category | Cause | Affected Files |
|---|---|---|---|---|
Improve extractShuffleIds to find AdaptiveSparkPlanExec anywhere in plan tree fixed in 4.1.1 |
✔️ #11426 @baibaichen | SQL | #52157 | gluten-ut/spark41/.../VeloxTestSettings.scala |
Fix SPARK-47939: Explain should work with parameterized queries |
✔️ @marin-ma | TEST | #11252 | gluten-ut/spark41/.../VeloxTestSettings.scala |
| Support Checksum in Column ShuffleWriters | 🚫 | CORE | #50230 | gluten-ut/spark41/.../velox/VeloxTestSettings.scala gluten-ut/spark41/.../GlutenMapStatusEndToEndSuite.scala Excluded tests: - GlutenMapStatusEndToEndSuite (entire suite) |
Support spark.sql.unionOutputPartitioning=true |
✔️ #11455 @baibaichen | SQL | #51623 | .github/workflows/velox_backend_x86.yml, gluten-ut/spark41/.../VeloxTestSettings.scala, tools/gluten-it/common/.../Suite.scalaExcluded tests: - GlutenBroadcastExchangeSuite.SPARK-52962 - GlutenDataFrameSetOperationsSuite.SPARK-52921* |
| Fixes a Spark Parquet read bug where missing struct fields caused the entire struct to be read as NULL. | PARQUET | SPARK-53535 | gluten-ut/spark41/.../VeloxTestSettings.scalaExcluded tests: - SPARK-53535* - vectorized reader: missing all struct fields* |
|
| Infer Variant shredding schema when writing to Parquet | PARQUET | #52406 | gluten-ut/spark41/.../velox/VeloxTestSettings.scala Excluded test: - "infer shredding with mixed scale" in GlutenFileBasedDataSourceSuite |
|
| NullType/VOID/UNKNOWN Type Support in Parquet | PARQUET | SPARK-54220 | gluten-ut/spark41/.../VeloxTestSettings.scalaExcluded tests: - SPARK-54220* |
|
| Update CI python to 3.10 | #11519@ReemaAlzaid | PYTHON | #51259 | backends-velox/.../python/ArrowEvalPythonExecSuite.scala |
Align with Spark split |
SQL | #48470 | gluten-ut/spark41/.../VeloxTestSettings.scala, backends-velox/.../VeloxStringFunctionsSuite.scalaExcluded tests: - GlutenRegexpExpressionsSuite.SPLIT - VeloxStringFunctionsSuite: split test |
|
Fix additional Spark 4.1 KeyGroupedPartitioningSuite tests |
SQL | #53132, #53142 | gluten-ut/spark41/.../VeloxTestSettings.scalaExcluded tests: - SPARK-53322* - SPARK-54439* |
|
| Fix failing SQL tests on Spark 4.1. | SQL | N/A | gluten-ut/spark41/.../velox/VeloxSQLQueryTestSettings.scala Excluded tests: - cast.sql - describe.sql - nonansi/cast.sql - nonansi/st-functions.sql - scripting/randomly_generated_scripts.sql - st-functions.sql - type-coercion-edge-cases.sql - variant-field-extractions.sql |
|
| Support memory based thresholds for shuffle spill | ❌️ | SQL | #47856 | gluten-ut/spark41/.../VeloxTestSettings.scalaExcluded tests: - SPARK-49386: Window spill with more than the inMemoryThreshold and spillSizeThreshold - SPARK-49386: test SortMergeJoin (with spill by size threshold) |
| Fix additional Spark 4.1 STRUCTURED STREAMING tests. | ❌️ | SS | #52645 | gluten-ut/spark41/.../VeloxTestSettings.scalaExcluded tests: - SPARK-53942: changing the number of stateless shuffle partitions via config - SPARK-53942: stateful shuffle partitions are retained from old checkpoint |
| Fix additional Spark 4.1 STRUCTURED STREAMING tests. | ❌️ | SS | #52473 #52870 #52891 |
gluten-ut/spark41/.../velox/VeloxTestSettings.scala Excluded tests: - GlutenStreamRealTimeModeAllowlistSuite: "rtm operator allowlist", "repartition not allowed", "stateful queries not allowed" - GlutenStreamRealTimeModeE2ESuite: "foreach", "to_json and from_json round-trip", "generateExec passthrough" - GlutenStreamRealTimeModeSuite: "processAllAvailable" |
Gluten version
None