I suppose this scenario is similar to the ANSI mode, and should be categorised as the limitation to the spark 4.0 support.

To use df.stat.bloomFilter, user should explicitly set spark.gluten.sql.native.bloomFilter=false

Perhaps we should document it somewhere. @zhouyuan @jinchengchenghh Any suggestions?

Can we offload df.stat.bloomFilter?

Dit you meet Not yet implemented errors?

Can we offload df.stat.bloomFilter?

@jinchengchenghh It's being offloaded in spark 4.0 after SPARK-45565. The test failure is caused by the hard-coded BloomFilter.readFrom in Spark's code https://github.com/apache/spark/blob/v4.0.1/sql/api/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L583

Besides, this test case also tests unsupported API expectedFpp and bitSize in VeloxBloomFilter https://github.com/apache/spark/blob/v4.0.1/sql/core/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala#L529

Dit you meet Not yet implemented errors?

@zhztheplayer Haven't run into this error but there are indeed some unimplemented API in VeloxBloomFilter, meaning we cannot use VeloxBloomFilter for the stat function.

I don't know the path call BloomFilter.readFrom, can we update the plan or other ways to offload the df.stat.bloomFilter to native, then it can call VeloxBloomFilter, we can implement the unimplemented functions in it

@jinchengchenghh The bloom filter agg here is being offloaded in spark 4.0, but not in spark 3.x. BloomFilter.readFrom is not part of the plan.

I think there are two options to solve this issue:

1. Extend the CallerInfo utility to selectively offload the bloom filter expression.
2. (Perhaps eventually) Implement the unimplemented APIs in VeloxBloomFilter.

Thanks. @zhztheplayer @jinchengchenghh

Looks like 1 is the most feasible solution before we implement 2. Let me try to address 1 in this PR.

And we still need to figure out how to address the hardcoded BloomFilter.readFrom in Spark.

Maybe all the data frame stats function has similar issue, the information is native information, we could supply a utility class to enhance current df.stat.xxx

✅ 方案 1（推荐）：使用隐式类扩展 df.stat 功能

不用修改 Spark 源码，也不会破坏 API。

class GlutenDataFrameStatFunctions(df: DataFrame)
  extends DataFrameStatFunctions(df) {

  def glutenApproxQuantile(cols: Seq[String]): DataFrame = {
    // your gluten implementation
    df // return something
  }
}

object GlutenStatImplicits {
  implicit class GlutenStatOps(df: DataFrame) {
    def glutenStat: GlutenDataFrameStatFunctions =
      new GlutenDataFrameStatFunctions(df)
  }
}


使用：

import GlutenStatImplicits._

df.glutenStat.glutenApproxQuantile(Seq("col1"))


✔ 不修改 Spark
✔ 和 Spark 版本兼容
✔ 可以加入任意 Gluten 优化

@jinchengchenghh If I understand it correctly, it requires users to modify the code to use Gluten's implementation? In this case directly call df.stat.xxx in Gluten still fails.

yes

Run Gluten Clickhouse CI on x86