Skip to content

array_contains falls back to Spark in case of Array literal #1929

@comphead

Description

@comphead

Describe the bug

After reviewing apache/datafusion#16529

I added empty array test to Comet

test("array_contains") {
    withSQLConf(CometConf.COMET_EXPR_ALLOW_INCOMPATIBLE.key -> "true") {
      withTempDir { dir =>
        val path = new Path(dir.toURI.toString, "test.parquet")
        makeParquetFileAllTypes(path, dictionaryEnabled = false, n = 10000)
        spark.read.parquet(path.toString).createOrReplaceTempView("t1");
        checkSparkAnswerAndOperator(
          spark.sql("SELECT array_contains(array(_2, _3, _4), _2) FROM t1"))
        checkSparkAnswerAndOperator(
          spark.sql("SELECT array_contains((CASE WHEN _2 =_3 THEN array(_4) END), _4) FROM t1"));
        checkSparkAnswerAndOperator(spark.sql("SELECT array_contains(cast(array() as array<int>), _2) FROM t1"))
      }
    }
  }

and it fails with fallback

Expected only Comet native operators, but found Project.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions