[feat](jdbc) support custom function rules in catalog properties #51471
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33856 ms |
TPC-DS: Total hot run time: 188523 ms |
ClickBench: Total hot run time: 29.46 s |
Contributor
Author
|
run buildall |
This was referenced Jun 6, 2025
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33963 ms |
TPC-DS: Total hot run time: 193058 ms |
ClickBench: Total hot run time: 29.16 s |
ddc3281 to
34c78f1
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33633 ms |
TPC-DS: Total hot run time: 192566 ms |
ClickBench: Total hot run time: 29.17 s |
34c78f1 to
c99653d
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33892 ms |
TPC-DS: Total hot run time: 193228 ms |
ClickBench: Total hot run time: 29.21 s |
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jun 19, 2025
…che#51471) Add new jdbc catalog property: `function_rules`. This property will enable user to customize which function can be pushed down to jdbc datasource, and which function should be rewrote to function in jdbc datasource: eg: ``` create catalog jdbc properties ( ... 'function_rules' = '{"pushdown" : {"supported": ["to_date"], "unsupported" : ["abs"]}, "rewrite" : {"to_date" : "date2"}}' ) ``` The value is a json format config: - `pushdown`: the value of `pushdown` is a map, the `supported` array indicate the function that can be pushed down to datasource. the `unsupported` array indicate the function that can not. - `rewrite`: the value of `rewrite` is map, if the function can be pushed down to datasource, it will be renamed to the value in rewrite map. eg: ``` select * from jdbc_table where abs(k1) > 0 and to_date("k2") = "2022-10-10" ``` will generate sql: ``` select * from jdbc_table where date2("k2") = "2022-10-10" ``` You can see the `abs()` is not pushed down, and `to_date()` is rewrote to `date2()`. ------- This change is compatible with previous version. Previously, there are some hard coded function rules, and also a FE config `jdbc_mysql_unsupported_pushdown_functions`. Doris will merge the user specified `function_rules` and the above rules.
dataroaring
pushed a commit
that referenced
this pull request
Jun 20, 2025
yiguolei
pushed a commit
that referenced
this pull request
Jun 20, 2025
…#51541) This may cause some statistic collection task fail: ``` 2025-06-05 15:02:32,277 WARN (Statistics Job Appender|182) [ExternalCatalog.buildDbForInit():902] Failed to check db DORIS_TEST exist in remote system, ignore it. java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.jdbc.client.JdbcClient.getDatabaseNameList()" because "this.jdbcClient" is null at org.apache.doris.datasource.jdbc.JdbcExternalCatalog.listDatabaseNames(JdbcExternalCatalog.java:267) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.ExternalCatalog.getFilteredDatabaseNames(ExternalCatalog.java:476) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.ExternalCatalog.lambda$makeSureInitialized$0(ExternalCatalog.java:318) ~[doris-fe.jar:1.2-SNAPSHOT] at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?] ``` Only for branch-2.1. Fix master in #51471
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jun 29, 2025
…che#51471) Add new jdbc catalog property: `function_rules`. This property will enable user to customize which function can be pushed down to jdbc datasource, and which function should be rewrote to function in jdbc datasource: eg: ``` create catalog jdbc properties ( ... 'function_rules' = '{"pushdown" : {"supported": ["to_date"], "unsupported" : ["abs"]}, "rewrite" : {"to_date" : "date2"}}' ) ``` The value is a json format config: - `pushdown`: the value of `pushdown` is a map, the `supported` array indicate the function that can be pushed down to datasource. the `unsupported` array indicate the function that can not. - `rewrite`: the value of `rewrite` is map, if the function can be pushed down to datasource, it will be renamed to the value in rewrite map. eg: ``` select * from jdbc_table where abs(k1) > 0 and to_date("k2") = "2022-10-10" ``` will generate sql: ``` select * from jdbc_table where date2("k2") = "2022-10-10" ``` You can see the `abs()` is not pushed down, and `to_date()` is rewrote to `date2()`. ------- This change is compatible with previous version. Previously, there are some hard coded function rules, and also a FE config `jdbc_mysql_unsupported_pushdown_functions`. Doris will merge the user specified `function_rules` and the above rules.
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Jun 29, 2025
…e Lombok annotations (apache#51787) Related PR: apache#51471
morrySnow
pushed a commit
that referenced
this pull request
Jun 30, 2025
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jul 10, 2025
apache#52817) Followup apache#52561 Also fix a bug introduced by apache#51471, which lost the function rules in jdbc external table.
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jul 10, 2025
apache#52817) Followup apache#52561 Also fix a bug introduced by apache#51471, which lost the function rules in jdbc external table.
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jul 10, 2025
apache#52817) Followup apache#52561 Also fix a bug introduced by apache#51471, which lost the function rules in jdbc external table.
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jul 11, 2025
apache#52817) Followup apache#52561 Also fix a bug introduced by apache#51471, which lost the function rules in jdbc external table.
zy-kkk
added a commit
to zy-kkk/doris
that referenced
this pull request
Jul 13, 2025
…e Lombok annotations (apache#51787) Related PR: apache#51471
zy-kkk
added a commit
to zy-kkk/doris
that referenced
this pull request
Jul 13, 2025
…e Lombok annotations (apache#51787) Related PR: apache#51471
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Add new jdbc catalog property:
function_rules.This property will enable user to customize which function can be pushed down to jdbc datasource,
and which function should be rewrote to function in jdbc datasource:
eg:
The value is a json format config:
pushdown: the value ofpushdownis a map, thesupportedarray indicate the function that can be pushed down to datasource. theunsupportedarray indicate the function that can not.rewrite: the value ofrewriteis map, if the function can be pushed down to datasource, it will be renamed to the value in rewrite map.eg:
will generate sql:
You can see the
abs()is not pushed down, andto_date()is rewrote todate2().This change is compatible with previous version.
Previously, there are some hard coded function rules, and also a FE config
jdbc_mysql_unsupported_pushdown_functions.Doris will merge the user specified
function_rulesand the above rules.Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)