feat(native): Add an Arrow federation connector to run federated queries by pdabre12 · Pull Request #26404 · prestodb/presto

pdabre12 · 2025-10-22T18:20:14Z

Description

Enable C++ workers to run queries through existing single-node Java connector implementations by introducing a dedicated Arrow Federation connector. This connector forwards requests to an Arrow Flight server shim, creating a practical migration path for connectors that have not yet been ported to C++, allowing users to run federated queries in a native cluster.

Motivation and Context

RFC: prestodb/rfcs#46

Impact

No impact

Test Plan

Includes e2e testing between the coordinator - native FlightShim connector - FlightShim server - multiple data sources. We have also tested internally in a full platform environment.

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.
If adding new dependencies, verified they have an OpenSSF Scorecard score of 5.0 or higher (or obtained explicit TSC approval for lower scores).

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

Prestissimo (Native Execution) Changes
* Add an Arrow federation connector to run federated queries.

sourcery-ai

Sorry @pdabre12, your pull request is larger than the review limit of 150000 diff characters

pdabre12 · 2025-10-28T21:12:41Z

@sourcery-ai review

sourcery-ai · 2025-10-28T21:12:48Z

Sorry @pdabre12, your pull request is larger than the review limit of 150000 diff characters

SourceryAI · 2025-10-28T21:13:07Z

Sorry @pdabre12, your pull request is larger than the review limit of 150000 diff characters

linux-foundation-easycla · 2025-11-03T21:59:26Z

The committers listed above are authorized under a signed CLA.

✅ login: pdabre12 / name: Pratik Joseph Dabre (2f59030, 48eaf8a, 5b955d1)

elbinpallimalilibm · 2025-12-11T17:06:06Z

Is the proposed design to run the Flight server as a plugin in the single node Java coordinator itself. Does all the data fetch happen through the Java coordinator itself then?

pdabre12 · 2025-12-13T00:37:15Z

Is the proposed design to run the Flight server as a plugin in the single node Java coordinator itself. Does all the data fetch happen through the Java coordinator itself then?

@elbinpallimalilibm The proposed design is to have a separate Flight server shim—a lightweight Java process—running independently. The data fetching would occur through this process. We use a similar connector plugin loading logic as in a Java coordinator.
Please check out the RFC for the details ofthe design : https://github.com/prestodb/rfcs/blob/main/RFC-0018-java-connector-federation.md

elbinpallimalilibm · 2025-12-18T05:14:08Z

When would having a Presto CPP worker + Arrow Flight connector + Java Flight server shim be more advantageous than Presto Java worker with JDBC connector.

Arrow Flight server that can load Presto Java connectors, and provide a record batch stream from the given connector split. The FlightShim server is designed to work with a native arrow connector, that will use a Flight client to forward the connector split to the server and process the record batch stream. See related PR at prestodb#26404 RFC: prestodb/rfcs#46 Co-authored-by: Pratik Joseph Dabre <pdabre12@gmail.com>

…workers (#26369) ## Description This adds a FlightShim module for connector federation. This includes an Apache Arrow Flight server that can load Presto Java connectors, and provide a record batch stream from the given connector split. The FlightShim server is designed to work with a native arrow connector, that will use a Flight client to forward the connector split to the server and process the record batch stream. See related PR at #26404 RFC: prestodb/rfcs#46 ## Motivation and Context   This provides a path to connector federation with native workers ## Impact  No API changes required. ## Test Plan  Included unit tests for the FlightShim server functionality. The followup native connector #26404 includes e2e testing between the coordinator - native FlightShim connector - FlightShim server - multiple data sources. We have also tested internally in a full platform environment. ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [x] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [x] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [x] Adequate tests were added if applicable. - [x] CI passed. - [x] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == General Changes * Adding presto-flight-shim server module for connector federation. ``` Co-authored-by: Pratik Joseph Dabre <pdabre12@gmail.com>

…workers (prestodb#26369) ## Description This adds a FlightShim module for connector federation. This includes an Apache Arrow Flight server that can load Presto Java connectors, and provide a record batch stream from the given connector split. The FlightShim server is designed to work with a native arrow connector, that will use a Flight client to forward the connector split to the server and process the record batch stream. See related PR at prestodb#26404 RFC: prestodb/rfcs#46 ## Motivation and Context   This provides a path to connector federation with native workers ## Impact  No API changes required. ## Test Plan  Included unit tests for the FlightShim server functionality. The followup native connector prestodb#26404 includes e2e testing between the coordinator - native FlightShim connector - FlightShim server - multiple data sources. We have also tested internally in a full platform environment. ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [x] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [x] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [x] Adequate tests were added if applicable. - [x] CI passed. - [x] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == General Changes * Adding presto-flight-shim server module for connector federation. ``` Co-authored-by: Pratik Joseph Dabre <pdabre12@gmail.com>

pdabre12 · 2026-06-11T23:22:22Z

@SourceryAI review

sourcery-ai · 2026-06-11T23:22:27Z

Sorry @pdabre12, your pull request is larger than the review limit of 150000 diff characters

sourcery-ai

Sorry @pdabre12, your pull request is larger than the review limit of 150000 diff characters

prestodb-ci added the from:IBM PR from IBM label Oct 22, 2025

sourcery-ai Bot reviewed Oct 22, 2025

View reviewed changes

pdabre12 force-pushed the oss-fed-flight-server branch 2 times, most recently from af94842 to 906eb52 Compare October 28, 2025 21:12

pdabre12 changed the title ~~[native] Add Arrow Federation connector to run federated queries on a native cluster~~ feat (arrow-flight) : Add Arrow Federation connector to run federated queries on a native cluster Oct 28, 2025

pdabre12 force-pushed the oss-fed-flight-server branch from 906eb52 to ef1126d Compare October 31, 2025 20:14

pdabre12 changed the title ~~feat (arrow-flight) : Add Arrow Federation connector to run federated queries on a native cluster~~ feat(arrow-flight) : Add Arrow Federation connector to run federated queries on a native cluster Nov 3, 2025

pdabre12 changed the title ~~feat(arrow-flight) : Add Arrow Federation connector to run federated queries on a native cluster~~ feat(plugin-flightshim) : Add Arrow Federation connector to run federated queries on a native cluster Nov 3, 2025

pdabre12 changed the title ~~feat(plugin-flightshim) : Add Arrow Federation connector to run federated queries on a native cluster~~ feat (plugin-flightshim) : Add Arrow Federation connector to run federated queries on a native cluster Nov 3, 2025

pdabre12 changed the title ~~feat (plugin-flightshim) : Add Arrow Federation connector to run federated queries on a native cluster~~ feat(plugin-flightshim): Add Arrow Federation connector to run federated queries on a native cluster Nov 3, 2025

pdabre12 force-pushed the oss-fed-flight-server branch 3 times, most recently from a4b4cfa to 65e8e43 Compare November 5, 2025 20:20

pdabre12 force-pushed the oss-fed-flight-server branch 6 times, most recently from 743fb7d to 5b1d093 Compare November 20, 2025 20:00

pdabre12 force-pushed the oss-fed-flight-server branch 4 times, most recently from 1c6c870 to 21bb5c5 Compare December 5, 2025 18:27

pdabre12 force-pushed the oss-fed-flight-server branch from e81c9b4 to 0a064ca Compare March 31, 2026 01:59

pdabre12 force-pushed the oss-fed-flight-server branch from 112b354 to 7d1042d Compare April 23, 2026 20:04

pdabre12 force-pushed the oss-fed-flight-server branch from 61cb308 to 9679ddc Compare May 4, 2026 18:26

pdabre12 force-pushed the oss-fed-flight-server branch from b795a39 to 5150d67 Compare May 5, 2026 23:17

pdabre12 force-pushed the oss-fed-flight-server branch 2 times, most recently from ad83804 to f5a9535 Compare May 13, 2026 01:07

pdabre12 force-pushed the oss-fed-flight-server branch from b6a250e to 426740b Compare May 15, 2026 17:50

pdabre12 force-pushed the oss-fed-flight-server branch from 756d79c to 2fe59b3 Compare May 26, 2026 20:15

pdabre12 force-pushed the oss-fed-flight-server branch 2 times, most recently from b3690a0 to e44b433 Compare June 10, 2026 00:04

pdabre12 force-pushed the oss-fed-flight-server branch from b214976 to e7662b7 Compare June 11, 2026 22:52

pdabre12 changed the title ~~feat(plugin-flightshim): Add Arrow Federation connector to run federated queries on a native cluster~~ feat(native): Add arrow federation connector to run federated queries Jun 11, 2026

pdabre12 changed the title ~~feat(native): Add arrow federation connector to run federated queries~~ feat(native): Add Arrow federation connector to run federated queries Jun 11, 2026

feat(native): Add an Arrow federation connector to run federated queries

7453c32

sourcery-ai Bot reviewed Jun 12, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(native): Add an Arrow federation connector to run federated queries#26404

feat(native): Add an Arrow federation connector to run federated queries#26404
pdabre12 wants to merge 1 commit into
prestodb:masterfrom
pdabre12:oss-fed-flight-server

pdabre12 commented Oct 22, 2025 •

edited

Loading

Uh oh!

sourcery-ai Bot left a comment

Uh oh!

pdabre12 commented Oct 28, 2025

Uh oh!

sourcery-ai Bot commented Oct 28, 2025

Uh oh!

SourceryAI commented Oct 28, 2025

Uh oh!

linux-foundation-easycla Bot commented Nov 3, 2025 •

edited

Loading

Uh oh!

elbinpallimalilibm commented Dec 11, 2025

Uh oh!

pdabre12 commented Dec 13, 2025

Uh oh!

elbinpallimalilibm commented Dec 18, 2025

Uh oh!

pdabre12 commented Jun 11, 2026

Uh oh!

sourcery-ai Bot commented Jun 11, 2026

Uh oh!

sourcery-ai Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

pdabre12 commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Uh oh!

sourcery-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pdabre12 commented Oct 28, 2025

Uh oh!

sourcery-ai Bot commented Oct 28, 2025

Uh oh!

SourceryAI commented Oct 28, 2025

Uh oh!

linux-foundation-easycla Bot commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elbinpallimalilibm commented Dec 11, 2025

Uh oh!

pdabre12 commented Dec 13, 2025

Uh oh!

elbinpallimalilibm commented Dec 18, 2025

Uh oh!

pdabre12 commented Jun 11, 2026

Uh oh!

sourcery-ai Bot commented Jun 11, 2026

Uh oh!

sourcery-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pdabre12 commented Oct 22, 2025 •

edited

Loading

linux-foundation-easycla Bot commented Nov 3, 2025 •

edited

Loading