test: Add native expression optimizer for e2e tests by pdabre12 · Pull Request #26903 · prestodb/presto

pdabre12 · 2026-01-05T20:41:20Z

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.
If adding new dependencies, verified they have an OpenSSF Scorecard score of 5.0 or higher (or obtained explicit TSC approval for lower scores).

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* ... 
* ... 

Hive Connector Changes
* ... 
* ...

If release note is NOT required, use:

== NO RELEASE NOTE ==

sourcery-ai · 2026-01-05T20:41:27Z

Reviewer's Guide

Introduces a configurable expression optimizer path for native e2e tests by wiring a native expression optimizer factory into the testing stack, allowing tests to exercise either the built-in or native optimizer via a serving reference, while updating native expression function handle generation and relaxing or disabling a few flaky tests.

Sequence diagram for test-mode expression optimizer selection in e2e tests

sequenceDiagram
    actor Test
    participant TestingPrestoServer
    participant ExpressionOptimizerManager
    participant ExpressionOptimizerFactory
    participant RowExpressionOptimizer as BuiltInOptimizer
    participant NativeExpressionOptimizer as NativeOptimizer

    Test->>TestingPrestoServer: new TestingPrestoServer(...)
    TestingPrestoServer->>ExpressionOptimizerManager: new ExpressionOptimizerManager(...)
    activate ExpressionOptimizerManager
    ExpressionOptimizerManager->>BuiltInOptimizer: new RowExpressionOptimizer(FunctionAndTypeManager)
    ExpressionOptimizerManager->>ExpressionOptimizerManager: put(DEFAULT_EXPRESSION_OPTIMIZER_NAME, BuiltInOptimizer)
    ExpressionOptimizerManager->>ExpressionOptimizerManager: servingExpressionOptimizer.set(BuiltInOptimizer)
    deactivate ExpressionOptimizerManager

    TestingPrestoServer->>ExpressionOptimizerManager: loadExpressionOptimizerFactories()
    activate ExpressionOptimizerManager
    ExpressionOptimizerManager->>ExpressionOptimizerFactory: createExpressionOptimizer(context)
    ExpressionOptimizerFactory-->>ExpressionOptimizerManager: NativeOptimizer
    ExpressionOptimizerManager->>ExpressionOptimizerManager: put(nativeName, NativeOptimizer)
    ExpressionOptimizerManager->>ExpressionOptimizerManager: servingExpressionOptimizer.compareAndSet(current, NativeOptimizer)
    deactivate ExpressionOptimizerManager

    TestingPrestoServer->>ExpressionOptimizerManager: enableTestMode()
    ExpressionOptimizerManager->>ExpressionOptimizerManager: testMode = true

    Test->>ExpressionOptimizerManager: getExpressionOptimizer(connectorSession)
    activate ExpressionOptimizerManager
    ExpressionOptimizerManager->>ExpressionOptimizerManager: if testMode
    ExpressionOptimizerManager-->>Test: servingExpressionOptimizer.get() (NativeOptimizer)
    deactivate ExpressionOptimizerManager

Sequence diagram for updated native function handle generation

sequenceDiagram
    participant VeloxToPrestoExprConverter as Converter
    participant CallTypedExpr as VeloxCall
    participant Util as PrestoUtil
    participant BuiltInFunctionHandle
    participant SqlFunctionHandle

    Converter->>VeloxCall: getName()
    VeloxCall-->>Converter: exprName
    Converter->>VeloxCall: inputs()
    VeloxCall-->>Converter: exprInputs
    Converter->>Converter: build argumentTypes and signature

    Converter->>PrestoUtil: getFunctionNameParts(exprName)
    PrestoUtil-->>Converter: parts

    alt presto default namespace
        Converter->>BuiltInFunctionHandle: new BuiltInFunctionHandle()
        BuiltInFunctionHandle-->>Converter: builtInFunctionHandle
        Converter->>Converter: functionHandle = builtInFunctionHandle
    else sql function
        Converter->>Converter: functionId = toSqlFunctionId(exprName, signature.argumentTypes)
        Converter->>SqlFunctionHandle: new SqlFunctionHandle()
        SqlFunctionHandle-->>Converter: sqlFunctionHandle
        Converter->>Converter: sqlFunctionHandle.functionId = functionId
        Converter->>Converter: functionHandle = sqlFunctionHandle
    end

    Converter->>Converter: attach functionHandle to CallExpression JSON

Class diagram for updated ExpressionOptimizerManager test-mode behavior

classDiagram
    class ExpressionOptimizerManager {
        - PluginNodeManager nodeManager
        - FunctionAndTypeManager functionAndTypeManager
        - RowExpressionSerde rowExpressionSerde
        - FunctionResolution functionResolution
        - File configurationDirectory
        - Map~String,ExpressionOptimizerFactory~ expressionOptimizerFactories
        - Map~String,ExpressionOptimizer~ expressionOptimizers
        - AtomicReference~ExpressionOptimizer~ servingExpressionOptimizer
        - boolean testMode
        + ExpressionOptimizerManager(PluginNodeManager nodeManager, FunctionAndTypeManager functionAndTypeManager, RowExpressionSerde rowExpressionSerde, File configurationDirectory)
        + void loadExpressionOptimizerFactories()
        + void loadExpressionOptimizerFactory(String factoryName, String optimizerName)
        + void addExpressionOptimizerFactory(ExpressionOptimizerFactory expressionOptimizerFactory)
        + ExpressionOptimizer getExpressionOptimizer(ConnectorSession connectorSession)
        + void enableTestMode()
    }

    class ExpressionOptimizerFactory {
        + ExpressionOptimizer createExpressionOptimizer(ExpressionOptimizerContext context)
    }

    class ExpressionOptimizer {
        <<interface>>
        + RowExpression optimize(RowExpression expression)
    }

    class RowExpressionOptimizer {
        + RowExpressionOptimizer(FunctionAndTypeManager functionAndTypeManager)
        + RowExpression optimize(RowExpression expression)
    }

    class TestingPrestoServer {
        - ExpressionOptimizerManager expressionManager
        + TestingPrestoServer(...)
    }

    ExpressionOptimizerManager o--> ExpressionOptimizerFactory : uses
    ExpressionOptimizerManager o--> ExpressionOptimizer : manages
    ExpressionOptimizer <|.. RowExpressionOptimizer
    TestingPrestoServer --> ExpressionOptimizerManager : configures

    class BuiltInFunctionHandle {
        + String _type
        + Signature signature
    }

    class SqlFunctionHandle {
        + String _type
        + String functionId
        + String version
    }

    class Signature {
        + String name
        + List~String~ argumentTypes
        + String returnType
        + boolean variableArity
    }

    class VeloxToPrestoExprConverter {
        + CallExpressionPtr getCallExpression(CallTypedExpr* expr)
    }

    VeloxToPrestoExprConverter --> BuiltInFunctionHandle : may create
    VeloxToPrestoExprConverter --> SqlFunctionHandle : may create

File-Level Changes

Change	Details	Files
Add function handle abstraction in Velox-to-Presto expression conversion to support both built-in and SQL-invoked functions.	Include common utils header used for parsing function name parts. Introduce helper to build SQL function IDs from function name and argument types. Add getFunctionHandle helper that selects BuiltInFunctionHandle for presto.default functions and SqlFunctionHandle otherwise, using parsed function name parts. Change call expression serialization to use the new getFunctionHandle helper instead of always emitting a BuiltInFunctionHandle.	`presto-native-execution/presto_cpp/main/types/VeloxToPrestoExpr.cpp`
Make ExpressionOptimizerManager choose a single serving optimizer (default or native) and add a test mode bypass for session-based selection.	Add AtomicReference to track the currently serving ExpressionOptimizer and a testMode flag. Initialize servingExpressionOptimizer to the default RowExpressionOptimizer in the constructor. When loading an expression optimizer factory, construct the optimizer and update the servingExpressionOptimizer reference to point to the new optimizer. Short-circuit getExpressionOptimizer to return servingExpressionOptimizer when testMode is enabled, bypassing session-based selection. Expose enableTestMode to switch manager into this deterministic test behavior.	`presto-main-base/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java`
Wire the native expression optimizer into the native sidecar plugin tests and adjust tests to account for behavior differences and current limitations.	In native sidecar plugin setup, load the NativeExpressionOptimizerFactory as the "native" optimizer into the ExpressionManager. Align expected error message for division by zero to "division by zero" to match new behavior. Comment out or disable assertions that are currently incompatible with the native optimizer, including CASE expression and ARRAY literal session-variance checks, count(*) aggregation, and array_sort_by_key tests (mark the latter as disabled via Test(enabled = false)).	`presto-native-sidecar-plugin/src/test/java/com/facebook/presto/sidecar/TestNativeSidecarPlugin.java`
Enable test-mode expression optimizer behavior on TestingPrestoServer to make test optimizer selection deterministic.	After loading expression optimizer factories in TestingPrestoServer when not in coordinator-only or resource manager mode, enable test mode on the ExpressionOptimizerManager so tests always use the serving optimizer.	`presto-main/src/main/java/com/facebook/presto/server/testing/TestingPrestoServer.java`

Possibly linked issues

feat: Make SSD cache maxEntries limit configurable #26795: PR adds proper native expression optimizer and function handle resolution, which should fix the unknown native functions in CTE tests.

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

) ## Description Enables Velox to Presto expression conversion for lambda expressions. ## Motivation and Context Velox to Presto expression conversion currently returns the input `RowExpression` for Velox expressions of types `kConcat`, `kInput`, and `kLambda`. This is incorrect for Lambda expressions because Velox `ExprOptimizer` optimizes the body of `core::LambdaTypedExpr` and the result isn't translated back to Presto. This PR adds the missing wiring for Velox to Presto lambda expression conversion required to complete the constant folding. `getRowExpression` API in `VeloxToPrestoExprConverter` takes an optional `RowExpression` for the default result as and argument. This was being returned for lambda, concat and input Velox expressions. With this change lambda expressions are converted and the Velox ExprOptimizer never returns concat or input expressions. So the default result parameter is unused and can be removed. ## Impact Fixes errors uncovered by `array_sort` in #26903. ## Test Plan Added e2e tests. ``` == NO RELEASE NOTE == ```

…stodb#26913) ## Description Enables Velox to Presto expression conversion for lambda expressions. ## Motivation and Context Velox to Presto expression conversion currently returns the input `RowExpression` for Velox expressions of types `kConcat`, `kInput`, and `kLambda`. This is incorrect for Lambda expressions because Velox `ExprOptimizer` optimizes the body of `core::LambdaTypedExpr` and the result isn't translated back to Presto. This PR adds the missing wiring for Velox to Presto lambda expression conversion required to complete the constant folding. `getRowExpression` API in `VeloxToPrestoExprConverter` takes an optional `RowExpression` for the default result as and argument. This was being returned for lambda, concat and input Velox expressions. With this change lambda expressions are converted and the Velox ExprOptimizer never returns concat or input expressions. So the default result parameter is unused and can be removed. ## Impact Fixes errors uncovered by `array_sort` in prestodb#26903. ## Test Plan Added e2e tests. ``` == NO RELEASE NOTE == ```

## Description Add parsing logic for `NativeFunctionHandle`. All functions under `NativeFunctionNamespaceManager` now return and parse `NativeFunctionHandle` instead of `SqlFunctionHandle`. ## Motivation and Context This change is required for `VeloxToPrestoExpr` when the native expression optimizer is enabled. During translation from a Velox row expression back to a Presto row expression, we must construct a function handle that is supported by `NativeFunctionNamespaceManager` when the underlying function is native. To avoid parsing different function handle types, all functions under `NativeFunctionNamespaceManager` now consistently return and parse `NativeFunctionHandle`. Issues discovered by : #26903 ## Impact There is no direct user-facing impact. This change only affects internal function handle construction and parsing. ## Test Plan CI, Unit tests. ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [x] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [x] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [x] Adequate tests were added if applicable. - [x] CI passed. - [x] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == Prestissimo (Native Execution) Changes * Add support for `NativeFunctionHandle` parsing. ```

…ersion

This reverts commit 366a7fa.

prestodb-ci added the from:IBM PR from IBM label Jan 5, 2026

pdabre12 force-pushed the add-constant-folding-tests branch from 5f23456 to 988995d Compare January 5, 2026 20:54

pdabre12 changed the title ~~[DNR] Add native expression optimizer for e2e tests~~ [DNR] test : Add native expression optimizer for e2e tests Jan 5, 2026

pdabre12 changed the title ~~[DNR] test : Add native expression optimizer for e2e tests~~ [DNR] test: Add native expression optimizer for e2e tests Jan 5, 2026

pdabre12 changed the title ~~[DNR] test: Add native expression optimizer for e2e tests~~ test: Add native expression optimizer for e2e tests Jan 5, 2026

pramodsatya self-requested a review January 6, 2026 21:33

pramodsatya mentioned this pull request Jan 7, 2026

fix(native): Enable Velox to Presto lambda expression conversion #26913

Merged

pramodsatya mentioned this pull request Jan 13, 2026

fix(native): Fix Velox to Presto IN expression conversion #26951

Merged

pdabre12 mentioned this pull request Jan 13, 2026

feat(native): Add support for NativeFunctionHandle parsing #26948

Merged

7 tasks

pdabre12 force-pushed the add-constant-folding-tests branch from fa865f4 to 37b8aad Compare January 22, 2026 08:30

pdabre12 and others added 9 commits January 29, 2026 11:11

fix(native): Fix Velox to Presto IN expression conversion

08cc26e

fix(native): Account for searched form of CASE during expression conv…

9791731

…ersion

test: Add native expression optimizer for e2e tests

24795be

Aggregate test failures

baa8051

Revert "Convert cast to substr when target is capped varchar"

e0f32c6

This reverts commit 366a7fa.

Add varcharn signature generation for sidecar

09966c2

Fix varchar return type if no parameter needed

b9c6a8c

Update submodule to user repo and advance Velox

0315283

Disable more test cases after cherry-picking varcharN changes

186914d

pdabre12 force-pushed the add-constant-folding-tests branch from 0db6a9e to 186914d Compare January 30, 2026 21:11

Advance velox

440f5cc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: Add native expression optimizer for e2e tests#26903

test: Add native expression optimizer for e2e tests#26903
pdabre12 wants to merge 10 commits into
prestodb:masterfrom
pdabre12:add-constant-folding-tests

pdabre12 commented Jan 5, 2026

Uh oh!

sourcery-ai Bot commented Jan 5, 2026 •

edited

Loading

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pdabre12 commented Jan 5, 2026

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Uh oh!

sourcery-ai Bot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviewer's Guide

Sequence diagram for test-mode expression optimizer selection in e2e tests

Sequence diagram for updated native function handle generation

Class diagram for updated ExpressionOptimizerManager test-mode behavior

File-Level Changes

Possibly linked issues

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sourcery-ai Bot commented Jan 5, 2026 •

edited

Loading