[GLUTEN-11088][VL] Add config in GlutenCastSuite to cast to char/varchar#11181
[GLUTEN-11088][VL] Add config in GlutenCastSuite to cast to char/varchar#11181zhouyuan merged 5 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
| } | ||
|
|
||
| testGluten("Casting to char/varchar") { | ||
| withSQLConf(SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true") { |
There was a problem hiding this comment.
Do we not support PRESERVE_CHAR_VARCHAR_TYPE_INFO false? What's the failure? If we always need to set PRESERVE_CHAR_VARCHAR_TYPE_INFO to true, may set in the plugin as default value
There was a problem hiding this comment.
If false, then exception will be thrown in Spark. We need to add this into the Gluten test because it overrides the checkEvaluation that invokes Spark's RowEncoder, which requires this configuration to be set. But in Vanilla spark, the checkEvaluation method doesn't invoke RowEncoder.
There was a problem hiding this comment.
Thanks for you explaination, could you add it to PR description?
There was a problem hiding this comment.
What if we set the config to all the tests in this suite? Looks like if add the new test, we still needs this config
There was a problem hiding this comment.
Then we can set it in beforeClass
There was a problem hiding this comment.
Make sense. I will make this change.
2763656 to
6c7e492
Compare
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
@jinchengchenghh Do you have further comments? cc: @zhouyuan |
Need to explicitly set
spark.sql.preserveCharVarcharTypeInfo=truefor gluten's test framework. We need to add this into the Gluten test because it overrides the checkEvaluation that invokes Spark's RowEncoder, which requires this configuration to be set. But in Vanilla spark, the checkEvaluation method doesn't invoke RowEncoder.Related issue: #11088