Add documents on how to develop a UDF / UDAF#4094
Add documents on how to develop a UDF / UDAF#4094LantaoJin merged 8 commits intoopensearch-project:mainfrom
Conversation
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
DEVELOPER_GUIDE.rst
Outdated
|
|
||
| New PPL Command Checklist | ||
| ========================= | ||
| Developing PPL Commands and Functions |
There was a problem hiding this comment.
This will break the the link.
I think we you can add a new section out of New PPL Command Checklist
There was a problem hiding this comment.
I thought it was too heavy to have two sections for developing functions and commands, considering most of other sections are high level instructions.
But somehow I cannot add lower level sections to this document. I updated them to separate sections.
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
DEVELOPER_GUIDE.rst
Outdated
| - Consult repository maintainers if PM is unavailable | ||
| - Be prepared for meetings to discuss syntax and usage details |
There was a problem hiding this comment.
Modified by LLM unintentionally, should be ok though
DEVELOPER_GUIDE.rst
Outdated
| - Adapt existing static methods: Convert Java static methods to UDFs using utility functions like ``UserDefinedFunctionUtils.adaptExprMethodToUDF`` | ||
| - Implement from scratch | ||
|
|
||
| * Implement the ``ImplementorUDF`` interface |
There was a problem hiding this comment.
Please call out that developers should ensure their UDF calculating logic is efficient by implementing data-irrelevant logic during the compilation phase rather than at runtime. In other words, that part of logic should preferably use ling4j's expression instead of internal static method calls.
There was a problem hiding this comment.
Thanks. I added such comments. I'm thinking about adding an example so that it will be easier to comprehend. Does the following example fit the case?
@Override
public Expression implement(RexToLixTranslator translator, RexCall call, List<Expression> translatedOperands) {
// Don't
SqlTypeName type = call.getOperands().getFirst().getType().getTypeName();
List<Expression> operands = Lists.concat(translatedOperands, List.of(Expressions.constant(type));
return Expressions.call(Implementor.class, "allInOneOp", operands)
// Do
String opName = type == FLOAT ? "floatOp" : "integerOp";
return Expressions.call(Implementor.class, , translatedOperands);
}
public static int allInOneOp(Number n, SqlTypeFamily t){
if (t == FLOAT) { ... }
else if (t == INTEGER) { .... }
}
public static int floatOp(float n){
...
}
public static int integerOp(integer n) {
...
}Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
DEVELOPER_GUIDE.rst
Outdated
| - Include at least syntax definition, usage and examples | ||
| - Implementation options are welcome if you have multiple ways to implement it | ||
| | ✅ Open an RFC issue describing the command: | ||
| - Specify the purpose and need for the new command |
There was a problem hiding this comment.
Is the new description really better than the original?
The LLM generated content changed the level of requirements. Some info I tried to callout is missing. For example, offline meeting is not necessary, syntax, usage and examples are minimal requirements of RFC, implementation details are not necessary in RFC, etc.
Can you revert the LLM generated suggestions except grammar fixing?
There was a problem hiding this comment.
Sorry for the unnecessary change. I have reverted them.
|
This should go in the dev docs at Dev guide should be more high level on how to build the project and maybe what the core entrypoints are. Making this its own doc will also give more room to add detail in e.g. lines like "Creating UDFs: A user-defined function is an instance of |
…GUIDE.rst Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
…cs/dev Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
That makes sense. I have moved both the guidance for commands and functions from DEVELOPER_GUIDE to docs/dev |
|
The backport to To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4094-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1ae4f288a3d6ec59fb5234b551e5cd6293cc3c8f
# Push it to GitHub
git push --set-upstream origin backport/backport-4094-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-devThen, create a pull request where the |
* Refactor: simplify registration of user-defined aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Draft guidance on developing user-defined functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Improve the developer guide with the help of LLM Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update DEVELOPER_GUIDE.rst Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update DEVELOPER_GUIDE.rst Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert changes to the section New PPL Command Checklist in DEVELOPER_GUIDE.rst Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Move ppl command and function dev guidance from DEVELOEPR_GUIDE to docs/dev Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
* Refactor: simplify registration of user-defined aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Draft guidance on developing user-defined functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Improve the developer guide with the help of LLM Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update DEVELOPER_GUIDE.rst Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update DEVELOPER_GUIDE.rst Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert changes to the section New PPL Command Checklist in DEVELOPER_GUIDE.rst Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Move ppl command and function dev guidance from DEVELOEPR_GUIDE to docs/dev Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
* Refactor: simplify registration of user-defined aggregation functions * Draft guidance on developing user-defined functions * Improve the developer guide with the help of LLM * Update DEVELOPER_GUIDE.rst * Update DEVELOPER_GUIDE.rst * Revert changes to the section New PPL Command Checklist in DEVELOPER_GUIDE.rst * Move ppl command and function dev guidance from DEVELOEPR_GUIDE to docs/dev --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com> Co-authored-by: Yuanchun Shen <yuanchu@amazon.com>
Description
This PR creates a dev doc Developing PPL Functions under folder docs/dev.
Additionally, it refactors the registration of aggregation functions to simplify its logic, making it similar to the creation and registration of non-aggregation functions to reduce learning barriers.
Related Issues
Resolves #4043
Check List
--signoffor-s.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.