I noticed that we never filled in the more details, so here is a note. Merging the two SQL execution engine is a large project, and so requires some intermediate milestones.

First, an overview of SQL execution: queries are parsed into a planNode tree. This planNode tree is optimized/rewritten, but eventually there is a final tree that is "ready for execution. At this point, there is a fork in the road:

We can execute the plan node tree as-is, as each plan node has a Next() implementation, which can be called, triggering volcano-style execution. This is known as "local SQL execution", since we do not have any functionality for the Next() function to trigger an RPC to execute a Next() on a remote machine.
We can take this plan node tree, and go through 'distributed SQL physical planning'. This generates a dataflow graph, which is designed to allow for vertices in the dataflow graph to be placed all across
the cluster, and communicate with each other using RPCs. Each vertex has a corresponding 'processor' implementation. (known as 'operators' in the wider literature).

The DistSQL processors are thus redundant with the Next() functionality provided by the planNode trees.

We eventually want only the latter path to exist; local SQL execution should eventually be sunsetted. However, this is a complex task because:

There is not a 1:1 relationship between planNodes and processors due to engineering effort. Some processors have simply not yet been implemented. These are tracked in distsql: tracking issue for unsupported query subexpressions #15793.
DistSQL does not currently support writes, as the transaction coordination mechanism (which I do not understand) assumes that all the information about all the writes is on one node. While I apologize for the lack of detail on this point, I hope it achieves the meta-point that the entire scope of this project is beyond the comprehension of any single team member.
DistSQL processors have behavior which is simply not present in Local SQL. For instance, Local SQL does not have the capability to spill to disk when its working-set is larger than the available memory. This creates unintuitive user behavior (I add an INSERT to my transaction, and then the SELECT OOMs). While there exists two execution engines, this behavior should be harmonized.

The plan to merge the execution engines is as follows:

Create a shim so that calling Next() on a planNode instantiates and calls the underlying processor. This achieves a feature parity, and lets us test for any surprises. Once all the planNodes no longer have a parallel execution path, we know that the only remaining divergences in behavior are because DistSQL doesn't implement something.
Close those feature parity gaps. This work is tracked in distsql: tracking issue for unsupported query subexpressions #15793.
Create a reverse shim that allows us to wrap arbitrary planNodes inside a virtual processor, so that we can plan a single "wrapper" distsql processor that does planNode execution under the hood, and thus there is a simple non-distributed plan we can create to encapsulate all plan nodes.
We remove the Next() function and all glue code that we added in 1.

Thus, 1. is mostly to proactively reveal feature incompatibilities. This effort already revealed the lack of tuple support in DistSQL , which was something we had all forgotten about.

sql: implement local-distsql shim for distinct #23901

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions