-
Notifications
You must be signed in to change notification settings - Fork 7.4k
Closed
Copy link
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn'tdataRay Data-related issuesRay Data-related issues
Description
The "Task Completion Time Without Backpressure" metrics chart is broken.
I think it's because #57788 renamed task_completion_time_without_backpressure without updating the associated code in data_dashboard_panels.py:
ray/python/ray/dashboard/modules/metrics/dashboards/data_dashboard_panels.py
Lines 680 to 694 in 395e60e
| TASK_COMPLETION_TIME_WITHOUT_BACKPRESSURE_PANEL = Panel( | |
| id=40, | |
| title="Task Completion Time Without Backpressure", | |
| description="Average time (in seconds) tasks spend executing their core logic, excluding backpressure, over a recent 5-minute window. This metric helps isolate actual computation time from delays caused by data flow bottlenecks, aiding in differentiating between computation-bound and data-flow-bound performance issues.", | |
| unit="s", | |
| targets=[ | |
| Target( | |
| expr='increase(ray_data_task_completion_time_without_backpressure{{{global_filters}, operator=~"$Operator"}}[5m]) / increase(ray_data_num_tasks_finished{{{global_filters}, operator=~"$Operator"}}[5m])', | |
| legend="Task Completion Time w/o Backpressure: {{dataset}}, {{operator}}", | |
| ), | |
| ], | |
| fill=0, | |
| stack=False, | |
| ) | |
What you need to do:
- Update the Grafana panel definition to use the correct name in the
expr - Run a simple Ray Data job and ensure the panel works correctly.
- You'll need to setup Grafana and Prometheus to do this. Anecdotally, this can be difficult to set up for the first time. See https://docs.ray.io/en/latest/cluster/metrics.html for instructions.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn'tdataRay Data-related issuesRay Data-related issues