Connector-based data sources (Arrow Flight/gRPC, future JDBC) always run as a single stream on the coordinator, regardless of how much parallelism the remote system can offer. This work extends the Connector SPI to support split-based execution and wires Arrow Flight as the first consumer.
Connector SPI split bridge — Allow connectors to opt into receiving split information from the execution engine, enabling multi-driver parallelism without breaking existing implementations.
Arrow Flight parallel execution — Map each Flight endpoint (from getFlightInfo) to an independent split so multiple endpoints are read in parallel across separate drivers.
Arrow Flight distributed execution — Serialize Flight splits across the transport layer so they can be assigned to data nodes through the existing exchange infrastructure.
Connector-based data sources (Arrow Flight/gRPC, future JDBC) always run as a single stream on the coordinator, regardless of how much parallelism the remote system can offer. This work extends the Connector SPI to support split-based execution and wires Arrow Flight as the first consumer.
Connector SPI split bridge — Allow connectors to opt into receiving split information from the execution engine, enabling multi-driver parallelism without breaking existing implementations.
Arrow Flight parallel execution — Map each Flight endpoint (from
getFlightInfo) to an independent split so multiple endpoints are read in parallel across separate drivers.Arrow Flight distributed execution — Serialize Flight splits across the transport layer so they can be assigned to data nodes through the existing exchange infrastructure.