fix: Improve testing for array_remove and fallback to Spark for unsupported types #1308
Conversation
| Range(0, num).map(_ => generateRow(schema, stringGen)) | ||
| } | ||
|
|
||
| private val primitiveTypes = Seq( |
There was a problem hiding this comment.
This code is copied moved from the fuzz testing tool
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1308 +/- ##
============================================
+ Coverage 34.69% 34.91% +0.22%
+ Complexity 991 989 -2
============================================
Files 116 119 +3
Lines 44891 45204 +313
Branches 9864 9959 +95
============================================
+ Hits 15574 15785 +211
- Misses 26168 26253 +85
- Partials 3149 3166 +17 ☔ View full report in Codecov by Sentry. |
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
The PR title says array_remove but the test is for array-intersect?
Or other places that we already have tests?
fuzz-testing/src/main/scala/org/apache/comet/fuzz/QueryRunner.scala
Outdated
Show resolved
Hide resolved
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
| r, | ||
| spark, | ||
| s"test$i.parquet", | ||
| numRows = conf.generateData.numRows(), |
There was a problem hiding this comment.
numRows is for all rows from all generated files or for one single file? Do we have definition for it?
There was a problem hiding this comment.
This is the number of rows per file. I pushed a commit to add descriptions for all command-line options.
-e, --exclude-negative-zero Whether to exclude negative zero
-g, --generate-arrays Whether to generate arrays
--generate-maps Whether to generate maps
--generate-structs Whether to generate structs
-n, --num-files <arg> Number of files to generate
--num-rows <arg> Number of rows per file
-h, --help Show help message
…usion-comet into array-remove-type-checks
|
Thanks for the reviews @viirya and @kazuyukitanimura |
…ported types (apache#1308) * Fallback to Spark for unsupported types for ArrayRemove * save progress * improve tests * revert debug logging * prepare for review * fix * format * remove test failure * fix * more testing * refactor * update readme * fix * format * fix * fix * fix * fix inverted options * Update QueryRunner.scala Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> * Add descriptions to CLI options --------- Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Which issue does this PR close?
Closes #1307
Rationale for this change
This PR fixes a specific bug and provides an approach for solving a general problem where we are not checking for supported types when translating expressions to native.
What changes are included in this PR?
array_removeand related testsHow are these changes tested?