[train] add proper filtering to metrics#53788
Merged
matthewdeng merged 7 commits intoray-project:masterfrom Jun 18, 2025
Merged
[train] add proper filtering to metrics#53788matthewdeng merged 7 commits intoray-project:masterfrom
matthewdeng merged 7 commits intoray-project:masterfrom
Conversation
Signed-off-by: Matthew Deng <matt@anyscale.com>
Signed-off-by: Matthew Deng <matt@anyscale.com>
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR enhances dashboard filtering for training metrics and streamlines the Train Run view.
- Added GPU filters (
GpuIndex,GpuDeviceName) to the base JSON dashboard - Updated all Train panels to use explicit
SessionName,TrainRunName, andTrainRunIdfilters - Simplified the Train Run view by removing system resource panels and standardizing variable quoting
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| python/ray/dashboard/modules/metrics/dashboards/train_grafana_dashboard_base.json | Added GPU filter variables (GpuIndex, GpuDeviceName) |
| python/ray/dashboard/modules/metrics/dashboards/train_dashboard_panels.py | Scoped queries by explicit train-run filters, removed system panels, and adjusted filter quoting |
Comments suppressed due to low confidence (1)
python/ray/dashboard/modules/metrics/dashboards/train_grafana_dashboard_base.json:191
- The GPU variable queries still use
{global_filters}and may include data from all sessions; consider addingSessionName=~\"$SessionName\"to scope GPU values to the current session.
"definition": "label_values(ray_node_gpus_utilization{{{global_filters}}}, GpuIndex)",
| default_uid="rayTrainDashboard", | ||
| rows=TRAIN_GRAFANA_ROWS, | ||
| standard_global_filters=['SessionName=~"$SessionName"'], | ||
| standard_global_filters=["SessionName=~'$SessionName'"], |
There was a problem hiding this comment.
The filter string uses single quotes around $SessionName, which may break the regex match in Grafana; switch to double quotes: SessionName=~\"$SessionName\".
Suggested change
| standard_global_filters=["SessionName=~'$SessionName'"], | |
| standard_global_filters=["SessionName=~\"$SessionName\""], |
Signed-off-by: Matthew Deng <matt@anyscale.com>
matthewdeng
commented
Jun 13, 2025
| targets=[ | ||
| Target( | ||
| expr="sum(ray_train_controller_state{{{global_filters}}}) by (ray_train_run_name, ray_train_controller_state)", | ||
| expr='sum(ray_train_controller_state{{SessionName=~"$SessionName", ray_train_run_name=~"$TrainRunName", ray_train_run_id=~"$TrainRunId"}}) by (ray_train_run_name, ray_train_controller_state)', |
Contributor
Author
There was a problem hiding this comment.
@alanwguo I made these all explicit, but is it better to have {{global_filters}} instead?
Contributor
There was a problem hiding this comment.
you need to keep global_filters on every single panel because there are env vars that lets users add additional global filters.
Signed-off-by: Matthew Deng <matt@anyscale.com>
alanwguo
approved these changes
Jun 18, 2025
Contributor
alanwguo
left a comment
There was a problem hiding this comment.
worked e2e! just need one change
python/ray/dashboard/modules/metrics/dashboards/train_grafana_dashboard_base.json
Outdated
Show resolved
Hide resolved
python/ray/dashboard/modules/metrics/dashboards/train_grafana_dashboard_base.json
Outdated
Show resolved
Hide resolved
minerharry
pushed a commit
to minerharry/ray
that referenced
this pull request
Jun 27, 2025
1. Added GPU filtering:
- Added `GpuIndex` and `GpuDeviceName` variables to dashboard
- Updated GPU panels to use these variables
- Variables hidden by default
2. Improved Train metrics:
- Added `TrainRunName` and `TrainRunId` to all Train queries
- Standardized quote usage
- Removed `{global_filters}` in favor of explicit variables
3. Simplified Train Run view:
- Reduced to only Train-specific metrics
- Removed system resource panels
---------
Signed-off-by: Matthew Deng <matt@anyscale.com>
elliot-barn
pushed a commit
that referenced
this pull request
Jul 2, 2025
1. Added GPU filtering:
- Added `GpuIndex` and `GpuDeviceName` variables to dashboard
- Updated GPU panels to use these variables
- Variables hidden by default
2. Improved Train metrics:
- Added `TrainRunName` and `TrainRunId` to all Train queries
- Standardized quote usage
- Removed `{global_filters}` in favor of explicit variables
3. Simplified Train Run view:
- Reduced to only Train-specific metrics
- Removed system resource panels
---------
Signed-off-by: Matthew Deng <matt@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added GPU filtering:
GpuIndexandGpuDeviceNamevariables to dashboardImproved Train metrics:
TrainRunNameandTrainRunIdto all Train queries{global_filters}in favor of explicit variablesSimplified Train Run view: