What is the problem the feature request solves?
Just tried comet locally with spark-shell, it turns out that comet requires that all the input shall be columnar based. So, it's not possible to test comet with the following code:
spark.range(10, 1000, 10).selectExpr("id", "id + 1 as val").repartition(10, col("id"))
Adding the CometRowToColumnar operator on top of the leaf node(RangeExec) will make it easy to test comet out.
In the long term, I think the CometRowToColumnar could be used to wrap around Spark's row-base source exec node and makes all the following operators columnar-based and leverages comet's columnar execution.
Describe the potential solution
Add CometRowToColumnar operator
Additional context
No response
What is the problem the feature request solves?
Just tried comet locally with
spark-shell, it turns out that comet requires that all the input shall be columnar based. So, it's not possible to test comet with the following code:Adding the
CometRowToColumnaroperator on top of the leaf node(RangeExec) will make it easy to test comet out.In the long term, I think the
CometRowToColumnarcould be used to wrap around Spark's row-base source exec node and makes all the following operators columnar-based and leverages comet's columnar execution.Describe the potential solution
Add CometRowToColumnar operator
Additional context
No response