Add RFC-0018 for Java connector federation in C++ workers by tdcmeehan · Pull Request #46 · prestodb/rfcs

tdcmeehan · 2025-08-29T20:36:51Z

No description provided.

BryanCutler

LGTM, thanks @tdcmeehan. I mostly just had some questions about auth with the flight server

aditi-pandit

RFC looks good overall. Have only a minor comment.

pdabre12

Thanks @tdcmeehan , LTGM.

Should we add some documentation on how to load up the new ArrowFederationConnector?

BryanCutler

LGTM, just a minor clarification to the split scheduling section

BryanCutler · 2025-09-17T17:28:52Z

+
+#### Split Scheduling Phase
+1. Coordinator's single-node connector (e.g., JDBC, MongoDB) creates its connector-specific split
+2. Split is scheduled to C++ worker as `ArrowFederationSplit` containing serialized split bytes


Just to clarify, the coordinator will send a connector-specific split to the C++ worker, but it will be handled by the ArrowFederation protocol where it will be read as bytes only.

Thanks, made that a bit more clear.

czentgr · 2025-09-18T18:52:56Z

FYI RFC-0013 exists already. Next one would be RFC-0016.

Arrow Flight server that can load Presto Java connectors, and provide a record batch stream from the given connector split. The FlightShim server is designed to work with a native arrow connector, that will use a Flight client to forward the connector split to the server and process the record batch stream. See related PR at prestodb#26404 RFC: prestodb/rfcs#46 Co-authored-by: Pratik Joseph Dabre <pdabre12@gmail.com>

…workers (#26369) ## Description This adds a FlightShim module for connector federation. This includes an Apache Arrow Flight server that can load Presto Java connectors, and provide a record batch stream from the given connector split. The FlightShim server is designed to work with a native arrow connector, that will use a Flight client to forward the connector split to the server and process the record batch stream. See related PR at #26404 RFC: prestodb/rfcs#46 ## Motivation and Context   This provides a path to connector federation with native workers ## Impact  No API changes required. ## Test Plan  Included unit tests for the FlightShim server functionality. The followup native connector #26404 includes e2e testing between the coordinator - native FlightShim connector - FlightShim server - multiple data sources. We have also tested internally in a full platform environment. ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [x] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [x] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [x] Adequate tests were added if applicable. - [x] CI passed. - [x] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == General Changes * Adding presto-flight-shim server module for connector federation. ``` Co-authored-by: Pratik Joseph Dabre <pdabre12@gmail.com>

…workers (prestodb#26369) ## Description This adds a FlightShim module for connector federation. This includes an Apache Arrow Flight server that can load Presto Java connectors, and provide a record batch stream from the given connector split. The FlightShim server is designed to work with a native arrow connector, that will use a Flight client to forward the connector split to the server and process the record batch stream. See related PR at prestodb#26404 RFC: prestodb/rfcs#46 ## Motivation and Context   This provides a path to connector federation with native workers ## Impact  No API changes required. ## Test Plan  Included unit tests for the FlightShim server functionality. The followup native connector prestodb#26404 includes e2e testing between the coordinator - native FlightShim connector - FlightShim server - multiple data sources. We have also tested internally in a full platform environment. ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [x] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [x] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [x] Adequate tests were added if applicable. - [x] CI passed. - [x] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == General Changes * Adding presto-flight-shim server module for connector federation. ``` Co-authored-by: Pratik Joseph Dabre <pdabre12@gmail.com>

prestodb-ci added the from:IBM PRs from IBM label Aug 29, 2025

prestodb-ci requested review from a team, ShahimSharafudeen and nishithakbhaskaran and removed request for a team August 29, 2025 20:36

tdcmeehan force-pushed the conn branch 9 times, most recently from eced569 to 826a238 Compare September 4, 2025 16:11

BryanCutler approved these changes Sep 4, 2025

View reviewed changes

Comment thread RFC-0013-java-connector-federaton.md Outdated

Comment thread RFC-0013-java-connector-federaton.md Outdated

Comment thread RFC-0013-java-connector-federaton.md Outdated

Comment thread RFC-0013-java-connector-federaton.md Outdated

tdcmeehan force-pushed the conn branch from 826a238 to 9842622 Compare September 5, 2025 20:45

infvg reviewed Sep 9, 2025

View reviewed changes

Comment thread RFC-0013-java-connector-federaton.md Outdated

tdcmeehan force-pushed the conn branch 3 times, most recently from bdfb6b2 to 79aed22 Compare September 9, 2025 15:12

aditi-pandit reviewed Sep 13, 2025

View reviewed changes

Comment thread RFC-0013-java-connector-federaton.md Outdated

tdcmeehan force-pushed the conn branch 2 times, most recently from a4ddced to 7196f6d Compare September 16, 2025 19:40

pdabre12 approved these changes Sep 16, 2025

View reviewed changes

BryanCutler approved these changes Sep 17, 2025

View reviewed changes

Add RFC-0013 for Java connector federation in C++ workers

783ccab

tdcmeehan force-pushed the conn branch from 7196f6d to 783ccab Compare September 19, 2025 02:07

BryanCutler mentioned this pull request Oct 20, 2025

feat: Adding Apache Arrow FlightShim for connector federation in C++ workers prestodb/presto#26369

Merged

7 tasks

16 -> 18

25870b7

aditi-pandit merged commit 05783f4 into prestodb:main Oct 22, 2025

pdabre12 mentioned this pull request Jun 11, 2026

feat(native): Add an Arrow federation connector to run federated queries prestodb/presto#26404

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add RFC-0018 for Java connector federation in C++ workers#46

Add RFC-0018 for Java connector federation in C++ workers#46
aditi-pandit merged 3 commits into
prestodb:mainfrom
tdcmeehan:conn

tdcmeehan commented Aug 29, 2025

Uh oh!

BryanCutler left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aditi-pandit left a comment

Uh oh!

Uh oh!

pdabre12 left a comment

Uh oh!

BryanCutler left a comment

Uh oh!

BryanCutler Sep 17, 2025

Uh oh!

tdcmeehan Sep 19, 2025

Uh oh!

czentgr commented Sep 18, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

tdcmeehan commented Aug 29, 2025

Uh oh!

BryanCutler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aditi-pandit left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pdabre12 left a comment

Choose a reason for hiding this comment

Uh oh!

BryanCutler left a comment

Choose a reason for hiding this comment

Uh oh!

BryanCutler Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

tdcmeehan Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

czentgr commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

czentgr commented Sep 18, 2025 •

edited

Loading