[GLUTEN-7600][VL] Remove EmptySchemaWorkaround#7620
Merged
zhztheplayer merged 9 commits intoapache:mainfrom Oct 22, 2024
Merged
[GLUTEN-7600][VL] Remove EmptySchemaWorkaround#7620zhztheplayer merged 9 commits intoapache:mainfrom
zhztheplayer merged 9 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
zhztheplayer
commented
Oct 21, 2024
| checkNullTypeRepartition( | ||
| spark.table("lineitem").selectExpr("null as x", "null as y").repartition(), | ||
| 1 | ||
| 0 |
Member
Author
There was a problem hiding this comment.
Query plan changed from
VeloxColumnarToRow
+- ShuffleQueryStage 0
+- ColumnarExchange SinglePartition, REBALANCE_PARTITIONS_BY_NONE, [plan_id=979], [shuffle_writer_type=hash], [OUTPUT] ArraySeq(x:NullType, y:NullType), [OUTPUT] ArraySeq(x:NullType, y:NullType)
+- VeloxResizeBatches 1024, 2147483647
+- RowToVeloxColumnar
+- *(1) Project [null AS x#296, null AS y#297]
+- *(1) ColumnarToRow
+- FileScan parquet [] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/opt/gluten/backends-velox/target/scala-2.13/test-classes/tpch-da..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<>
to
VeloxColumnarToRow
+- ShuffleQueryStage 0
+- ColumnarExchange SinglePartition, REBALANCE_PARTITIONS_BY_NONE, [plan_id=957], [shuffle_writer_type=hash], [OUTPUT] ArraySeq(x:NullType, y:NullType), [OUTPUT] ArraySeq(x:NullType, y:NullType)
+- VeloxResizeBatches 1024, 2147483647
+- ^(12) ProjectExecTransformer [null AS x#296, null AS y#297]
+- ^(12) NativeFileScan parquet [] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/opt/gluten/backends-velox/target/scala-2.13/test-classes/tpch-da..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<>
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
zhztheplayer
commented
Oct 22, 2024
| Sig[CheckOverflow](CHECK_OVERFLOW), | ||
| Sig[MakeDecimal](MAKE_DECIMAL), | ||
| Sig[PromotePrecision](PROMOTE_PRECISION), | ||
| Sig[MonotonicallyIncreasingID](MONOTONICALLY_INCREASING_ID), |
zhztheplayer
commented
Oct 22, 2024
Comment on lines
-213
to
-219
| if ((list == null || list.isEmpty) && childCtx != null) { | ||
| // The computing for this project is not needed. | ||
| // the child may be an input adapter and childCtx is null. In this case we want to | ||
| // make a read node with non-empty base_schema. | ||
| context.registerEmptyRelToOperator(operatorId) | ||
| return childCtx | ||
| } |
Member
Author
There was a problem hiding this comment.
context.registerEmptyRelToOperator(operatorId) looks to be suitable only when the operator simply outputs all the inputs it receives. Which is not the case here for list.isEmpty
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
zhztheplayer
commented
Oct 22, 2024
Comment on lines
+94
to
+97
| if (child.output.isEmpty) { | ||
| // See: https://github.com/apache/incubator-gluten/issues/7600. | ||
| return Some("Shuffle with empty schema is not supported") | ||
| } |
Member
Author
There was a problem hiding this comment.
Empty schema batches should pass through shuffle so that they can be handled in reducer-side operators within the row number information they carried on.
This is a temporary code path to disable shuffle for empty schema input. We should finally remove it to add the support.
cc @marin-ma
zhztheplayer
commented
Oct 22, 2024
|
|
||
| private static BatchType identifyBatchType(ColumnarBatch batch) { | ||
| if (batch.numCols() == 0) { | ||
| // zero-column batch considered as heavy batch |
Member
Author
There was a problem hiding this comment.
comment should be updated
zhztheplayer
commented
Oct 22, 2024
Comment on lines
+903
to
+904
| .exclude( | ||
| "SPARK-16633: lead/lag should return the default value if the offset row does not exist") |
|
Run Gluten Clickhouse CI |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #7600
Remove rules
EmptySchemaWorkaround.PlanOneRowRelation/EmptySchemaWorkaround.FallbackEmptySchemaRelation/OffloadHashAggregate. Inline some workaround logics into utility classColumnarBatchesor into operator validation procedures.