-
Notifications
You must be signed in to change notification settings - Fork 270
Description
Describe the bug
CometNativeExec builds a tree of CometMetric nodes containing SQLMetrics for the Spark plan operators. This structure is passed into the Native.createPlan method.
During native execution, calls to update_metrics will recurse through the native ExecutionPlan tree and through the CometMetric tree simultaneously and copy native metrics to Spark metrics.
This approach assumes a 1:1 mapping between the Spark plan and the native plan. In some cases, this assumption is not correct and we therefore drop metrics and the Spark UI does not show metrics for some nodes.
One example is for aggregate queries, where the native planner sometimes inserts an additional projection.
Another example is that we insert CopyExec nodes around the inputs to aggregate, sort, and join operators.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response