[VL] Enable Spark legacy date formatter if spark.sql.legacy.timeParserPolicy is set to 'LEGACY'#7375
Conversation
…cy is set to 'LEGACY'.
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
|
@rui-mo Could you help to take a look please? Thanks. |
|
|
||
| // Enable Spark legacy date formatter if spark.sql.legacy.timeParserPolicy is set to 'LEGACY'. | ||
| if (veloxCfg_->get<std::string>(kSparkLegacyTimeParserPolicy, "") == "LEGACY") { | ||
| configs[velox::core::QueryConfig::kSparkLegacyDateFormatter] = "true"; |
There was a problem hiding this comment.
Do we need to set this config as false if 'kSparkLegacyTimeParserPolicy' is not 'LEGACY'?
There was a problem hiding this comment.
Thanks for your feedback. It is necessary to do this. Updated.
|
Run Gluten Clickhouse CI |
|
@rui-mo Could you help to merge this PR? Thanks. |
|
@NEUpanning does this work for spark properties |
|
@surnaik No, this PR only works for |
What changes were proposed in this pull request?
Velox#10966 introduced simple date time formatter that aligns with Spark legacy date parsing/formatting behavior. This PR enables Velox simple date time formatter if
spark.sql.legacy.timeParserPolicyis set toLEGACY.Relates Velox issue: #10354
How was this patch tested?
Integration tests with Spark