Description

Our plan is to ensure that Spark 3.4 passes all unit tests. This issue will track all the failed unit tests in Spark 3.4, and we will work on fixing them one by one.

GlutenSQLQueryTestSuite

• - describe-query.sql *** FAILED ***
• - sql-compatibility-functions.sql *** FAILED ***
• - cast.sql *** FAILED ***
• - show-tables.sql *** FAILED ***
• - intersect-all.sql *** FAILED ***
• - cte-nested.sql *** FAILED ***
• - random.sql *** FAILED ***
• - datetime-formatting-invalid.sql *** FAILED ***
• - group-by.sql *** FAILED ***
• - show-create-table.sql *** FAILED ***
• - cte-nonlegacy.sql *** FAILED ***
• - inline-table.sql *** FAILED ***
• - describe-table-column.sql *** FAILED ***
• - map.sql *** FAILED ***
• - describe.sql *** FAILED ***
• - datetime-parsing-invalid.sql *** FAILED ***
• - show_columns.sql *** FAILED ***
• - comments.sql *** FAILED ***
• - udaf.sql *** FAILED ***
• - json-functions.sql *** FAILED ***
• - describe-table-after-alter-table.sql *** FAILED ***
• - show-tblproperties.sql *** FAILED ***
• - grouping_set.sql *** FAILED ***
• - charvarchar.sql *** FAILED ***
• - table-valued-functions.sql *** FAILED ***
• - regexp-functions.sql *** FAILED ***
• - ilike-any.sql *** FAILED ***
• - change-column.sql *** FAILED ***
• - string-functions.sql *** FAILED ***
• - columnresolution-negative.sql *** FAILED ***
• - group-by-ordinal.sql *** FAILED ***
• - timestamp-ntz.sql *** FAILED ***
• - interval.sql *** FAILED ***
• - date.sql *** FAILED ***
• - bitwise.sql *** FAILED ***
• - natural-join.sql *** FAILED ***
• - timezone.sql *** FAILED ***
• - ilike-all.sql *** FAILED ***
• - datetime-legacy.sql *** FAILED ***
• - csv-functions.sql *** FAILED ***
• - pivot.sql *** FAILED ***
• - group-by-filter.sql *** FAILED ***
• - except-all.sql *** FAILED ***
• - join-lateral.sql *** FAILED ***
• - like-any.sql *** FAILED ***
• - table-aliases.sql *** FAILED ***
• - transform.sql *** FAILED ***
• - query_regex_column.sql *** FAILED ***
• - count.sql *** FAILED ***
• - show-views.sql *** FAILED ***
• - timestamp.sql *** FAILED ***
• - try_arithmetic.sql *** FAILED ***
• - extract.sql *** FAILED ***
• - cte.sql *** FAILED ***
• - cte-legacy.sql *** FAILED ***
• - order-by-ordinal.sql *** FAILED ***
• - literals.sql *** FAILED ***
• - limit.sql *** FAILED ***
• - like-all.sql *** FAILED ***
• - tablesample-negative.sql *** FAILED ***
• - ansi/cast.sql *** FAILED ***
• - ansi/decimalArithmeticOperations.sql *** FAILED ***
• - ansi/map.sql *** FAILED ***
• - ansi/datetime-parsing-invalid.sql *** FAILED ***
• - ansi/string-functions.sql *** FAILED ***
• - ansi/interval.sql *** FAILED ***
• - ansi/date.sql *** FAILED ***
• - ansi/timestamp.sql *** FAILED ***
• - ansi/try_arithmetic.sql *** FAILED ***
• - ansi/literals.sql *** FAILED ***
• - timestampNTZ/timestamp-ansi.sql *** FAILED ***
• - timestampNTZ/timestamp.sql *** FAILED ***
• - udf/udf-intersect-all.sql - Scala UDF *** FAILED ***
• - udf/udf-except-all.sql - Scala UDF *** FAILED ***
• - udf/udf-udaf.sql - Scala UDF *** FAILED ***
• - udf/udf-group-by.sql - Scala UDF *** FAILED ***
• - udf/udf-except.sql - Scala UDF *** FAILED ***
• - udf/udf-pivot.sql - Scala UDF *** FAILED ***
• - udf/udf-inline-table.sql - Scala UDF *** FAILED ***
• - udf/postgreSQL/udf-select_having.sql - Scala UDF *** FAILED ***
• - typeCoercion/native/windowFrameCoercion.sql *** FAILED ***
• - typeCoercion/native/decimalPrecision.sql *** FAILED ***
• - typeCoercion/native/ifCoercion.sql *** FAILED ***
• - typeCoercion/native/dateTimeOperations.sql *** FAILED ***
• - typeCoercion/native/booleanEquality.sql *** FAILED ***
• - typeCoercion/native/mapZipWith.sql *** FAILED ***
• - typeCoercion/native/caseWhenCoercion.sql *** FAILED ***
• - typeCoercion/native/widenSetOperationTypes.sql *** FAILED ***
• - typeCoercion/native/promoteStrings.sql *** FAILED ***
• - typeCoercion/native/stringCastAndExpressions.sql *** FAILED ***
• - typeCoercion/native/inConversion.sql *** FAILED ***
• - typeCoercion/native/division.sql *** FAILED ***
• - typeCoercion/native/mapconcat.sql *** FAILED ***
• - subquery/in-subquery/in-basic.sql *** FAILED ***
• - subquery/negative-cases/subq-input-typecheck.sql *** FAILED ***
• - subquery/negative-cases/invalid-correlation.sql *** FAILED ***
• - subquery/scalar-subquery/scalar-subquery-select.sql *** FAILED *** (Gluten throws exception: more than one row returned by a subquery used as an expression)
• - subquery/in-subquery/in-limit.sql *** FAILED *** (inconsistent result)
• - postgreSQL/create_view.sql *** FAILED ***
• - postgreSQL/boolean.sql *** FAILED ***
• - postgreSQL/aggregates_part1.sql *** FAILED ***
• - postgreSQL/window_part3.sql *** FAILED ***
• - postgreSQL/strings.sql *** FAILED ***
• - postgreSQL/window_part4.sql *** FAILED ***
• - postgreSQL/int8.sql *** FAILED ***
• - postgreSQL/union.sql *** FAILED ***
• - postgreSQL/aggregates_part4.sql *** FAILED ***
• - postgreSQL/int4.sql *** FAILED ***
• - postgreSQL/numeric.sql *** FAILED ***
• - postgreSQL/window_part2.sql *** FAILED ***
• - postgreSQL/with.sql *** FAILED ***
• - postgreSQL/interval.sql *** FAILED ***
• - postgreSQL/date.sql *** FAILED ***
• - postgreSQL/join.sql *** FAILED ***
• - postgreSQL/select_having.sql *** FAILED ***
• - postgreSQL/text.sql *** FAILED ***
• - postgreSQL/limit.sql *** FAILED ***
• - postgreSQL/float4.sql *** FAILED ***
• - postgreSQL/select_implicit.sql *** FAILED ***
• - postgreSQL/aggregates_part3.sql *** FAILED ***

GlutenBroadcastJoinSuite will hang.

VeloxDataTypeValidationSuite

• Velox Parquet Write

VeloxParquetWriteForHiveSuite

• test hive static partition write table
• test hive write table

VeloxParquetWriteSuite

• test write parquet with compression codec
• test ctas
• test parquet dynamic partition write

TPCDSPlan

• check simplified (tpcds-v1.4/$q)

GlutenParquetV1FilterSuite

• filter pushdown - StringPredicate

GlutenParquetV2FilterSuite

• filter pushdown - StringPredicate
• Gluten - filter pushdown - date

GlutenDynamicPartitionPruningV1SuiteAEOff

• - SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec
• - static scan metrics
• - Gluten - static scan metrics

GlutenDynamicPartitionPruningV1SuiteAEOn

• SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan

• SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan

• SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV2SuiteAEOff

• SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV2SuiteAEOn

• SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV2SuiteAEOnDisableScan

• SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan

• SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec

GlutenCoalesceShufflePartitionsSuite

• - Gluten - SPARK-24705 adaptive query execution works correctly when exchange reuse enabled

GlutenAdaptiveQueryExecSuite

• - SPARK-30291: AQE should catch the exceptions when doing materialize
• - SPARK-33933: Materialize BroadcastQueryStage first in AQE
• - gluten Logging plan changes for AQE
• - gluten test log level

GlutenEnsureRequirementsSuite

• - SPARK-41986: Introduce shuffle on SinglePartition

GlutenKeyGroupedPartitioningSuite

• - SPARK-41413: partitioned join: partition values from one side are subset of those from the other side
• - SPARK-41413: partitioned join: partition values from both sides overlaps
• - SPARK-41413: partitioned join: non-overlapping partition values from both sides
• - SPARK-41413: partitioned join: partition values from one side are subset of those from the other side
• - SPARK-41413: partitioned join: partition values from both sides overlaps
• - SPARK-41413: partitioned join: non-overlapping partition values from both sides
• - SPARK-42038: partially clustered: with different partition keys and both sides partially clustered
• - SPARK-42038: partially clustered: with different partition keys and missing keys on left-hand side
• - SPARK-42038: partially clustered: with different partition keys and missing keys on right-hand side
• - SPARK-42038: partially clustered: left outer join
• - SPARK-42038: partially clustered: right outer join
• - SPARK-42038: partially clustered: full outer join is not applicable
• - SPARK-42038: partially clustered: with dynamic partition filtering
• - SPARK-44641: duplicated records when SPJ is not triggered

GlutenParquetSchemaSuite

• - SPARK-40819: parquet file with TIMESTAMP(NANOS, true) (with nanosAsLong=true)
• - SPARK-40819: parquet file with TIMESTAMP(NANOS, true) (with default nanosAsLong=false)

GlutenWriteDistributionAndOrderingSuite

• - ordered distribution and sort with same exprs: append
• - ordered distribution and sort with same exprs: overwrite
• - ordered distribution and sort with same exprs: overwriteDynamic
• - clustered distribution and sort with same exprs: append *** FAILED ***
• - clustered distribution and sort with same exprs: overwrite *** FAILED ***
• - clustered distribution and sort with same exprs: overwriteDynamic *** FAILED ***
• - clustered distribution and sort with extended exprs: append *** FAILED ***
• - clustered distribution and sort with extended exprs: overwrite *** FAILED ***
• - clustered distribution and sort with extended exprs: overwriteDynamic *** FAILED ***
• - ordered distribution and sort with manual global sort: append *** FAILED ***
• - ordered distribution and sort with manual global sort: overwrite *** FAILED ***
• - ordered distribution and sort with manual global sort: overwriteDynamic *** FAILED ***
• - ordered distribution and sort with incompatible global sort: append *** FAILED ***
• - ordered distribution and sort with incompatible global sort: overwrite *** FAILED ***
• - ordered distribution and sort with incompatible global sort: overwriteDynamic *** FAILED ***
• - ordered distribution and sort with manual local sort: append *** FAILED ***
• - ordered distribution and sort with manual local sort: overwrite *** FAILED ***
• - ordered distribution and sort with manual local sort: overwriteDynamic *** FAILED ***
• - clustered distribution and local sort with manual global sort: append *** FAILED ***
• - clustered distribution and local sort with manual global sort: overwrite *** FAILED ***
• - clustered distribution and local sort with manual global sort: overwriteDynamic *** FAILED ***
• - clustered distribution and local sort with manual local sort: append *** FAILED ***
• - clustered distribution and local sort with manual local sort: overwrite *** FAILED ***
• - clustered distribution and local sort with manual local sort: overwriteDynamic *** FAILED ***

GlutenInsertSuite

• - INSERT rows, ALTER TABLE ADD COLUMNS with DEFAULTs, then SELECT them
• - SPARK-39557 INSERT INTO statements with tables with array defaults
• - SPARK-39557 INSERT INTO statements with tables with struct defaults
• - SPARK-39557 INSERT INTO statements with tables with map defaults
• - SPARK-39844 Restrict adding DEFAULT columns for existing tables to certain sources
• - INSERT INTO TABLE - complex type but different names

GlutenJoinSuite

• - SPARK-43113: Full outer join with duplicate stream-side references in condition (SMJ)
• - SPARK-45882: BroadcastHashJoinExec propagate partitioning should respect CoalescedHashPartitioning

GlutenFallbackSuite

• - test fallback logging
• - test fallback event

GlutenJsonV2Suite

• - SPARK-39731: Correctly parse dates and timestamps with yyyyMMdd pattern

GlutenParquetIOSuite

• - Read TimestampNTZ and TimestampLTZ for various logical TIMESTAMP types
• - SPARK-40128 read DELTA_LENGTH_BYTE_ARRAY encoded strings
• - SPARK-41096: FIXED_LEN_BYTE_ARRAY support

GlutenCTEHintSuite

• - Resolve join hint in CTE

GlutenExpressionsSchemaSuite

• - Check schemas for expression examples

GlutenDataSourceV2Suite

• - ordering and partitioning reporting

GlutenSQLQuerySuite

• - the escape character is not allowed to end with
• - SPARK-40245: Fix FileScan canonicalization when partition or data filter columns are not read
• SPARK-38548: try_sum should return null if overflow happens before merging

GlutenQueryExecutionErrorsSuite

• - FAILED_EXECUTE_UDF: execute user defined function
• - UNRECOGNIZED_SQL_TYPE: unrecognized SQL type -100
• - INVALID_BUCKET_FILE: error if there exists any malformed bucket files
• - SCALAR_SUBQUERY_TOO_MANY_ROWS: More than one row returned by a subquery used as an expression
• - UNSUPPORTED_FEATURE.MULTI_ACTION_ALTER: The target JDBC server hosting table does not support ALTER TABLE with multiple actions.

GlutenDataFrameSuite

• - SPARK-41048: Improve output partitioning and ordering with AQE cache

GlutenFileMetadataStructSuite

• - SPARK-41896: Filter on row_index and a stored column at the same time
• - SPARK-43450: Filter on aliased _metadata.row_index

GlutenCSVv1Suite

• - SPARK-23786: warning should be printed if CSV header doesn't conform to schema
• - lineSep with 2 chars when multiLine set to

GlutenCSVv2Suite

• - SPARK-23786: warning should be printed if CSV header doesn't conform to schema
• - lineSep with 2 chars when multiLine set to
• - test for FAILFAST parsing mode
• - SPARK-39731: Correctly parse dates and timestamps with yyyyMMdd pattern

GlutenCSVLegacyTimeParserSuite

• SPARK-23786: warning should be printed if CSV header doesn't conform to schema
• lineSep with 2 chars when multiLine set to

GlutenFileSourceSQLInsertTestSuite

• - SPARK-41982: treat the partition field as string literal when keepPartitionSpecAsStringLiteral is enabled

GlutenDSV2SQLInsertTestSuite

• -SPARK-41982: treat the partition field as string literal when keepPartitionSpecAsStringLiteral is enabled

GlutenArithmeticExpressionSuite

• - SPARK-45786: Decimal multiply, divide, remainder, quo

GlutenMathFunctionsSuite

• - SPARK-44973: conv must allocate enough space for all digits plus negative sign