[enhancement](UDF)The user defined functions support the global operation #17608

lexluo09 · 2023-03-09T08:55:22Z

Proposed changes

Issue Number: close #16973

Support create/drop global function.
When you create a custom function, it can only be used within in one database. It cannot be used in other database/catalog. When there are many databases/catalog, it needs to create function one by one.

Problem summary

Describe your changes.
1、 When a function is created or deleted, add the global keyword.

CREATE [GLOBAL] [AGGREGATE] [ALIAS] FUNCTION function_name (arg_type [, ...]) [RETURNS ret_type] [INTERMEDIATE inter_type] [WITH PARAMETER(param [,...]) AS origin_function] [PROPERTIES ("key" = "value" [, ...]) ]

DROP [GLOBAL] FUNCTION function_name (arg_type [, ...])

2、A completely global global function is set, and the global function metadata is stored in the image. The function lookup strategy is to look in the database first, and if it can't be found, it looks in the global function.

Checklist(Required)

[ no] Does it affect the original behavior
[ yes] Has unit tests been added
[ no] Has document been added or modified
[ no] Does it need to update dependencies
[yes] Is this PR support rollback (If NO, please explain WHY)

Further comments

lexluo09 · 2023-03-09T09:29:22Z

run buildall

lexluo09 · 2023-03-10T05:18:42Z

run buildall

morningman · 2023-03-12T13:47:47Z

fe/fe-core/src/main/java/org/apache/doris/analysis/ShowFunctionsStmt.java

+
+    public ShowFunctionsStmt(String dbName, boolean isBuiltin, boolean isVerbose, String wild,
+            Expr expr) {
+        this.type = type;


missing type param?

Show functions not fully implemented. I plan to fully implement next PR.

I will remove the relevant code in this PR.

morningman · 2023-03-12T13:51:58Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java

+     **/
+    public long loadGlobalFunction(DataInputStream in, long checksum) throws IOException {
+        GlobalFunctionMgr mgr = GlobalFunctionMgr.read(in);
+        if (mgr != null) {


Under what circumstances, mgr is null?

Yes, there is no case where mgr is null.

morningman · 2023-03-12T13:52:27Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java

    }

+    public long saveGlobalFunction(CountingDataOutputStream out, long checksum) throws IOException {
+        Env.getCurrentEnv().getGlobalFunctionMgr().write(out);


Suggested change

Env.getCurrentEnv().getGlobalFunctionMgr().write(out);

this.globalFunctionMgr.write(out);

Yes, thank you for your advice.

morningman · 2023-03-12T13:52:56Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java

        FunctionName name = stmt.getFunctionName();
-        Database db = getInternalCatalog().getDbOrDdlException(name.getDb());
-        db.addFunction(stmt.getFunction(), stmt.isIfNotExists());
+        InternalCatalog internalCatalog = getInternalCatalog();


Move this line to else block

Yes, thank you for your advice

fe/fe-core/src/main/java/org/apache/doris/catalog/GlobalFunctionMgr.java

morningman · 2023-03-13T16:10:09Z

run buildall

lexluo09 · 2023-03-14T09:21:07Z

run compile feut

lexluo09 · 2023-03-14T11:18:18Z

run compile feut

lexluo09 · 2023-03-18T03:52:38Z

run buildall

…che#16973)

lexluo09 · 2023-03-18T04:05:02Z

run buildall

hello-stephen · 2023-03-18T04:22:31Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.85 seconds
stream load tsv: 470 seconds loaded 74807831229 Bytes, about 151 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 75 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230318080951_clickbench_pr_116385.html

lexluo09 · 2023-03-18T07:46:53Z

run buildall

morningman

LGTM

…che#16973) (apache#17608) Support create/drop global function. When you create a custom function, it can only be used within in one database. It cannot be used in other database/catalog. When there are many databases/catalog, it needs to create function one by one. ## Problem summary Describe your changes. 1、 When a function is created or deleted, add the global keyword. CREATE [GLOBAL] [AGGREGATE] [ALIAS] FUNCTION function_name (arg_type [, ...]) [RETURNS ret_type] [INTERMEDIATE inter_type] [WITH PARAMETER(param [,...]) AS origin_function] [PROPERTIES ("key" = "value" [, ...]) ] DROP [GLOBAL] FUNCTION function_name (arg_type [, ...]) 2、A completely global global function is set, and the global function metadata is stored in the image. The function lookup strategy is to look in the database first, and if it can't be found, it looks in the global function. Co-authored-by: lexluo <lexluo@tencent.com>

github-actions bot added the area/planner Issues or PRs related to the query planner label Mar 9, 2023

lexluo09 changed the title ~~[enhancement] Function(create/drop) support the global operation~~ [enhancement]The user defined functions support the global operation Mar 9, 2023

lexluo09 force-pushed the add_global_function branch 2 times, most recently from 5d3840a to 808be93 Compare March 9, 2023 09:17

lexluo09 force-pushed the add_global_function branch from 3ba2195 to 6f06578 Compare March 9, 2023 10:04

morningman reviewed Mar 12, 2023

View reviewed changes

lexluo09 force-pushed the add_global_function branch 2 times, most recently from fe7686e to 1395308 Compare March 13, 2023 02:41

morrySnow changed the title ~~[enhancement]The user defined functions support the global operation~~ [enhancement](UDF)The user defined functions support the global operation Mar 13, 2023

lexluo09 force-pushed the add_global_function branch 2 times, most recently from 0a60a69 to a1df463 Compare March 18, 2023 03:51

github-actions bot added the kind/docs Categorizes issue or PR as related to documentation. label Mar 18, 2023

[enhancement] Function(create/drop) support the global operation (apa…

5261fc8

…che#16973)

lexluo09 force-pushed the add_global_function branch from a1df463 to 5261fc8 Compare March 18, 2023 03:58

morningman approved these changes Mar 18, 2023

View reviewed changes

morningman merged commit c95eb8a into apache:master Mar 18, 2023

	Env.getCurrentEnv().getGlobalFunctionMgr().write(out);
	this.globalFunctionMgr.write(out);

[enhancement](UDF)The user defined functions support the global operation #17608

[enhancement](UDF)The user defined functions support the global operation #17608

Uh oh!

Conversation

lexluo09 commented Mar 9, 2023

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

lexluo09 commented Mar 9, 2023

Uh oh!

lexluo09 commented Mar 10, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

morningman commented Mar 13, 2023

Uh oh!

lexluo09 commented Mar 14, 2023

Uh oh!

lexluo09 commented Mar 14, 2023

Uh oh!

lexluo09 commented Mar 18, 2023

Uh oh!

lexluo09 commented Mar 18, 2023

Uh oh!

hello-stephen commented Mar 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lexluo09 commented Mar 18, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hello-stephen commented Mar 18, 2023 •

edited

Loading