Skip to content

ScanExec metrics do not get reported in Spark UI for aggregate, join, sort, etc #1110

@andygrove

Description

@andygrove

Describe the bug

Here is the native plan for a join. The join metrics of build_time and join_time get reported in the Spark UI but we do not report the metrics for fetching the input batches from the JVM or for unpacking dictionaries and performing deep copies where needed.

For this example it means we are reporting a time of ~410ms when the actual time is closer to ~600ms, and this is just for one partition.

HashJoinExec: metrics=[build_time=400.827077ms, join_time=8.557039ms]
  CopyExec [UnpackOrDeepCopy], metrics=[elapsed_compute=18.643737ms]
    ScanExec: source=[ShuffleQueryStage], metrics=[elapsed_compute=186.719525ms]
  CopyExec [UnpackOrDeepCopy], metrics=[..., elapsed_compute=293.113µs]
    ScanExec: source=[ShuffleQueryStage ...], metrics=[elapsed_compute=5.906924ms]

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions