[GLUTEN-2169][VL] fix: some spark 33 ut fixed#2563
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/oap-project/gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
gluten-core/src/main/scala/io/glutenproject/execution/BatchScanExecTransformer.scala
Outdated
Show resolved
Hide resolved
|
Would you add newly supported suites name in PR description? |
updated thanks |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
@gaoyangxiaozhu This command can help you. |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
anyone help checking for what thing goes wrong in Gluten ClickHouse CI ? |
|
Run Gluten Clickhouse CI |
|
Hi @gaoyangxiaozhu, you can log in ClickHouse CI with a public account/password. See https://github.com/oap-project/gluten/blob/main/docs/get-started/ClickHouse.md#new-ci-system. |
Run Gluten Clickhouse CI |
|
|
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
anyone help for ClickHouse CI issue, i manually trigger multiple times always fail but i can't find RCA from the log |
|
@zzcclp It seem the CI on clickhouse showed all tests passed, but still failed on exit, could you please help to take a look on this? thanks, -yuan |
|
Run Gluten Clickhouse CI |
1 similar comment
|
Run Gluten Clickhouse CI |
|
|
||
| def pushedAggregate(fileFormat: String): Option[Aggregation] = { | ||
| fileFormat match { | ||
| case "parquet" => scan.asInstanceOf[ParquetScan].pushedAggregate |
There was a problem hiding this comment.
Hi @gaoyangxiaozhu, can we directly use the below code to get the pushed aggregate? It seems there is no need to get file format to do the matching. Thanks!
@transient lazy val pushedAggregate: Option[Aggregation] = {
scan match {
case s: ParquetScan =>
s.pushedAggregate
case o: OrcScan =>
o.pushedAggregate
case _ =>
None
}
}
|
@gaoyangxiaozhu thanks for enabling those unit tests! |
philo-he
left a comment
There was a problem hiding this comment.
Looks great! I will do some small refinement in a follow-up PR. Thanks for the contribution!
fix some spark 33 uts #2169
new supported suites
GlutenParquetDeltaByteArrayEncodingSuite
GlutenParquetDeltaLengthByteArrayEncodingSuite
GlutenParquetFieldIdIOSuite
GlutenParquetVectorizedSuite
GlutenFileMetadataStructSuite
GlutenParquetV2AggregatePushDownSuite exclude "aggregate push down - different data types"
GlutenOrcV1AggregatePushDownSuite exclude "aggregate push down - different data types"
GlutenOrcV2AggregatePushDownSuite exclude "aggregate push down - different data types"
GlutenReplaceHashWithSortAggSuite
GlutenBroadcastJoinSuite
Here the basic principal of this PR is to fallback all unsupported scenario/op to JVM to quick quick the UTs issue.
Open 3 new issues to let refactor code in future do native support insteads fallback for maybe better perf gain.
pushaggregate support in native scan - #2617
medascan column support in native scan - #2618
match column by filedIds in native scan #2619