[Docs] Add ServiceMonitor section and make some step optional in Grafana & Promethus page by owenowenisme · Pull Request #53474 · ray-project/ray

owenowenisme · 2025-06-02T08:15:35Z

Why are these changes needed?

This PR update docs to show latest change of KubeRay.
Changes including:

A page listing existing KubeRay Metrics https://anyscale-ray--53474.com.readthedocs.build/en/53474/cluster/kubernetes/k8s-ecosystem/metrics-references.html#kuberay-metrics-references
Instruction of ServiceMonitor in https://anyscale-ray--53474.com.readthedocs.build/en/53474/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html#step-7-collect-kuberay-metrics-with-servicemonitor

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

owenowenisme · 2025-06-02T08:17:06Z

@troychiu @win5923 This PR added ServiceMonitor section, would you mind checking if I'm missing anything?

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

win5923 · 2025-06-04T02:04:13Z

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

We can add a new step to display the KubeRay Operator Dashboard once this PR is merged. Thanks!

win5923

Overall LGTM. Thanks Owen!

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

troychiu

Can we also mention that KubeRay provides the service monitor in helm chart https://github.com/ray-project/kuberay/blob/master/helm-chart/kuberay-operator/templates/servicemonitor.yaml? Maybe in Step 3: Install a KubeRay operator

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

troychiu · 2025-06-06T08:38:47Z

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

Maybe simply KubeRay Metrics?

I think KubeRay Metrics should include Controller Runtime Metrics and KubeRay Custom Metrics emphasize "custom" so we can differentiate with natively provided metrics (Controller Runtime), does it make sense to you?

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

owenowenisme · 2025-06-06T18:27:18Z

Can we also mention that KubeRay provides the service monitor in helm chart https://github.com/ray-project/kuberay/blob/master/helm-chart/kuberay-operator/templates/servicemonitor.yaml? Maybe in Step 3: Install a KubeRay operator

Is this for production purpose?
Cause we already deploy ServiceMonitor with install.sh as mentioned in Step 7.

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

troychiu · 2025-06-07T08:58:30Z

Can we also mention that KubeRay provides the service monitor in helm chart https://github.com/ray-project/kuberay/blob/master/helm-chart/kuberay-operator/templates/servicemonitor.yaml? Maybe in Step 3: Install a KubeRay operator

Is this for production purpose? Cause we already deploy ServiceMonitor with install.sh as mentioned in Step 7.

Yes, they serve different purposes. The install.sh script is mainly for users to quickly try out the Grafana dashboard, while the ServiceMonitor in the Helm chart is intended for production use.
I just want to make sure users are aware of it, so it doesn’t need to be a long explanation.

owenowenisme · 2025-06-07T13:04:34Z

@troychiu Then I suppose I should also make helm chart for other components in install.sh?
e.g. podMonitor

Signed-off-by: owenowenisme <mses010108@gmail.com>

Co-authored-by: Jun-Hao Wan <ken89@kimo.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

Signed-off-by: owenowenisme <mses010108@gmail.com>

Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

Co-authored-by: Troy Chiu <114708546+troychiu@users.noreply.github.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

owenowenisme · 2025-06-15T03:04:02Z

@kevin85421 PTAL

kevin85421 · 2025-06-15T06:29:31Z

I’ve requested that the docs team review this PR. @owenowenisme, if you haven’t installed Vale yet, please do so and ensure its rules are being followed.

owenowenisme · 2025-06-15T07:24:33Z

Thanks for reminding, I already ran vale, and make sure there is no new error.

angelinalg

Just some style and grammatical fixes. Our style guide says to use sentence case for titles. Unfortunately, the titles you haven't changed in this PR aren't all sentence case. If it's not too much work for you to fix them in this PR, please do. Otherwise, I can follow up with a PR after yours merges to fix them. Thanks for updating the docs!

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

owenowenisme · 2025-06-16T19:21:46Z

Thanks @angelinalg I just applied your suggestions.

angelinalg

I made a typo in one of my suggestions. Could you accept it? Also, are the corrections accurate? I want to verify that you meant RayService and not Ray Service for those metrics titles. @owenowenisme

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

kevin85421 · 2025-06-16T20:15:56Z

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

 ## Step 3: Install a KubeRay operator

 * Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository.
+* You can enable the ServiceMonitor when installing the KubeRay operator with helm. See [Step 7](#step-7-collect-kuberay-metrics-with-servicemonitor) for more details.


Remove this and provide the instruction to install KubeRay operator with ServiceMonitor instead.

helm install kuberay-operator kuberay/kuberay-operator --version 1.4.0 --set metrics.serviceMonitor.enabled=true

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

kevin85421 · 2025-06-16T20:16:56Z

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

 * KubeRay provides an [install.sh script](https://github.com/ray-project/kuberay/blob/master/install/prometheus/install.sh) to:
-  * Install the [kube-prometheus-stack v48.2.1](https://github.com/prometheus-community/helm-charts/tree/kube-prometheus-stack-48.2.1/charts/kube-prometheus-stack) chart and related custom resources, including **PodMonitor** and **PrometheusRule**, in the namespace `prometheus-system` automatically. 
-  * Import Ray Dashboard’s [Grafana JSON files](https://github.com/ray-project/kuberay/tree/master/config/grafana) into Grafana using the `--auto-load-dashboard true` flag. If the flag isn't set, the following step also provides instructions for manual import.
+  * Install the [kube-prometheus-stack v48.2.1](https://github.com/prometheus-community/helm-charts/tree/kube-prometheus-stack-48.2.1/charts/kube-prometheus-stack) chart and related custom resources, including **PodMonitor**, **ServiceMonitor**, and **PrometheusRule**, in the namespace `prometheus-system` automatically. 


Please remove ServiceMonitor for KubeRay operator from the install.sh.

kevin85421 · 2025-06-16T20:19:16Z

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

-## Step 7: Collect custom metrics with Recording Rules
+## Step 7: Collect KubeRay metrics with ServiceMonitor
+
+Starting with KubeRay 1.4.0, KubeRay provides a [ServiceMonitor](https://github.com/ray-project/kuberay/blob/master/config/prometheus/serviceMonitor.yaml) to help Prometheus discover and scrape KubeRay metrics.


We just need to tell users that the ServiceMonitor will be created when install KubeRay operator is installed and then provide a instruction to help users to verify it (something like kubectl get servicemonitor).

kevin85421 · 2025-06-16T20:21:30Z

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

-## Step 13: Embed Grafana panels in Ray Dashboard
+## Step 14: View the KubeRay operator dashboard
+
+Once the KubeRay Operator dashboard is imported into Grafana, you can monitor metrics from the KubeRay operator. The dashboard provides a dropdown menu to filter and view controller runtime metrics for specific Ray CRs:`RayCluster`, `RayJob`, `RayService`.


We only need to include one picture here; four pictures are too many.

Left only one.

kevin85421 · 2025-06-16T20:25:31Z

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

+# KubeRay metrics references
+
+## Controller runtime metrics
+KubeRay is built with Controller Runtime, which natively exposes metrics that KubeRay includes in its metrics. These metrics include:


❯ vale doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md 1:1 suggestion Use parentheses judiciously. Google.Parens 1:2 error Use 'KubeRay' instead of Google.WordList 'kuberay'. 6:38 error Use 'Kubernetes' instead of Vale.Terms 'kubernetes'. 8:57 error Use 'Kubernetes' instead of Vale.Terms 'kubernetes'. 32:95 suggestion In general, use active voice Google.Passive instead of passive voice ('is provisioned'). 33:158 suggestion Use parentheses judiciously. Google.Parens 49:313 suggestion In general, use active voice Google.Passive instead of passive voice ('is enabled').

I suppose that we can ignore these.

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

kevin85421 · 2025-06-16T20:35:59Z

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

+# KubeRay metrics references
+
+## Controller runtime metrics
+KubeRay is built with Controller Runtime, which natively exposes metrics that KubeRay includes in its metrics. These metrics include:


It's too verbose.

Please remove:

KubeRay is built with Controller Runtime ... ... - Go runtime stats like Goroutines and GC duration

and use the following line instead:

"KubeRay exposes metrics provided by kubernetes-sigs/controller-runtime, including information about reconciliation, work queues, and more, to help users operate the KubeRay operator in production environments."

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md

Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

kevin85421 · 2025-06-18T00:44:24Z

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

+  ```sh
+  helm install kuberay-operator kuberay/kuberay-operator --version 1.4.0 \
+    --set metrics.serviceMonitor.enabled=true
+  ```


Add an instruction to verify whether the ServiceMonitor is created correctly.

kevin85421 · 2025-06-18T00:47:32Z

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

-## Step 7: Collect custom metrics with Recording Rules
+## Step 7: Collect KubeRay metrics with ServiceMonitor
+
+Installing the KubeRay operator automatically creates a ServiceMonitor to help Prometheus discover and scrape KubeRay metrics. You can verify the ServiceMonitor creation with:


kevin85421 · 2025-06-18T00:52:09Z

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

+      honorLabels: true
+
+```
+* Same as PodMonitor, the **install.sh** script also creates the [serviceMonitor.yaml](https://github.com/ray-project/kuberay/blob/master/config/prometheus/serviceMonitor.yaml) shown above, so you don't need to create it manually.


Delete L271 - L276

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

kevin85421

Nice

kevin85421 · 2025-06-18T04:26:48Z

@owenowenisme please ping me when all CI tests pass.

owenowenisme · 2025-06-18T07:25:55Z

@kevin85421 CI passed.

kevin85421 · 2025-06-18T08:01:43Z

cc @jjyao @edoakes would you mind merging this PR? Thanks!

…ana & Promethus page (ray-project#53474) This PR update docs to show latest change of KubeRay. Changes including: - A page listing existing KubeRay Metrics https://anyscale-ray--53474.com.readthedocs.build/en/53474/cluster/kubernetes/k8s-ecosystem/metrics-references.html#kuberay-metrics-references - Instruction of ServiceMonitor in https://anyscale-ray--53474.com.readthedocs.build/en/53474/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html#step-7-collect-kuberay-metrics-with-servicemonitor  ## Related issue number  ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: owenowenisme <mses010108@gmail.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com> Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Co-authored-by: Jun-Hao Wan <ken89@kimo.com> Co-authored-by: Troy Chiu <114708546+troychiu@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org>

…ana & Promethus page (#53474) This PR update docs to show latest change of KubeRay. Changes including: - A page listing existing KubeRay Metrics https://anyscale-ray--53474.com.readthedocs.build/en/53474/cluster/kubernetes/k8s-ecosystem/metrics-references.html#kuberay-metrics-references - Instruction of ServiceMonitor in https://anyscale-ray--53474.com.readthedocs.build/en/53474/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html#step-7-collect-kuberay-metrics-with-servicemonitor  ## Related issue number  ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: owenowenisme <mses010108@gmail.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com> Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Co-authored-by: Jun-Hao Wan <ken89@kimo.com> Co-authored-by: Troy Chiu <114708546+troychiu@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

owenowenisme changed the title ~~add ServiceMonitor section and make some step optional~~ [Docs] Add ServiceMonitor section and make some step optional in Grafana & Promethus page Jun 2, 2025

win5923 reviewed Jun 2, 2025

View reviewed changes

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md Outdated Show resolved Hide resolved

win5923 reviewed Jun 2, 2025

View reviewed changes

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md Outdated Show resolved Hide resolved

win5923 reviewed Jun 2, 2025

View reviewed changes

doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md Outdated Show resolved Hide resolved

owenowenisme force-pushed the kuberay/update-grafana-prom-docs branch from 75e443e to 66b002c Compare June 3, 2025 02:47

owenowenisme marked this pull request as ready for review June 3, 2025 05:12

owenowenisme requested review from a team, kevin85421 and pcmoritz as code owners June 3, 2025 05:12

owenowenisme requested a review from win5923 June 3, 2025 05:12

win5923 reviewed Jun 4, 2025

View reviewed changes

win5923 approved these changes Jun 4, 2025

View reviewed changes

troychiu reviewed Jun 6, 2025

View reviewed changes

troychiu reviewed Jun 7, 2025

View reviewed changes

owenowenisme requested review from troychiu and win5923 June 11, 2025 07:32

owenowenisme and others added 9 commits June 12, 2025 21:49

add ServiceMonitor section and make some step optional

c3d712b

Signed-off-by: owenowenisme <mses010108@gmail.com>

add docs for KubeRay Metrics References

17631e4

Signed-off-by: owenowenisme <mses010108@gmail.com>

update ServiceMonitor section

17fab42

Signed-off-by: owenowenisme <mses010108@gmail.com>

add reference to k8s-eco-md

904623a

Signed-off-by: owenowenisme <mses010108@gmail.com>

Apply suggestions from code review

9bf3939

Co-authored-by: Jun-Hao Wan <ken89@kimo.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

update

2f536ba

Signed-off-by: owenowenisme <mses010108@gmail.com>

update

3f364a0

Signed-off-by: owenowenisme <mses010108@gmail.com>

Update prometheus-grafana.md

3ae78f7

Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

Apply suggestions from code review

c60a7c2

Co-authored-by: Troy Chiu <114708546+troychiu@users.noreply.github.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

angelinalg reviewed Jun 16, 2025

View reviewed changes

angelinalg approved these changes Jun 16, 2025

View reviewed changes

Apply suggestions from code review

181ab2e

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

owenowenisme force-pushed the kuberay/update-grafana-prom-docs branch from ca7c630 to 181ab2e Compare June 16, 2025 19:20

angelinalg reviewed Jun 16, 2025

View reviewed changes

doc/source/cluster/kubernetes/k8s-ecosystem/metrics-references.md Outdated Show resolved Hide resolved

kevin85421 reviewed Jun 16, 2025

View reviewed changes

owenowenisme and others added 2 commits June 18, 2025 01:08

Apply suggestions from code review

7b286e7

Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

update

6ce320a

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

owenowenisme requested review from angelinalg and kevin85421 June 17, 2025 18:35

owenowenisme and others added 2 commits June 18, 2025 02:36

Merge branch 'master' into kuberay/update-grafana-prom-docs

cf71cb4

remove image

89b4769

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

kevin85421 reviewed Jun 18, 2025

View reviewed changes

owenowenisme and others added 3 commits June 18, 2025 10:15

Apply suggestions from code review

61ce8d3

Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

update docs

b35f5c6

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>

Update prometheus-grafana.md

d2d799f

Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>

kevin85421 approved these changes Jun 18, 2025

View reviewed changes

kevin85421 added the go add ONLY when ready to merge, run all tests label Jun 18, 2025

edoakes merged commit cf5eeb8 into ray-project:master Jun 18, 2025
6 checks passed

Conversation

owenowenisme commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

owenowenisme commented Jun 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

troychiu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

owenowenisme commented Jun 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

troychiu commented Jun 7, 2025

Uh oh!

owenowenisme commented Jun 7, 2025

Uh oh!

owenowenisme commented Jun 15, 2025

Uh oh!

kevin85421 commented Jun 15, 2025

Uh oh!

owenowenisme commented Jun 15, 2025

Uh oh!

angelinalg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

owenowenisme commented Jun 16, 2025

Uh oh!

angelinalg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

owenowenisme commented Jun 2, 2025 •

edited

Loading