[core][telemetry/01] migrate python telemetry recorder to opentelemetry by can-anyscale · Pull Request #53098 · ray-project/ray

can-anyscale · 2025-05-16T19:24:35Z

This is a series of PR to migrate metric collection from opencencus to openlemetry. For context, ray has two main parts that instrument and record metric through opencencus: c++ and python part.

The python part consists of three components:

A singleton python object (let's call it the TelemetryMetricRecorder) that stores all telemetry that got sent to it in memories.
A PrometheusServer that holds this TelemetryMetricRecorder object. The server takes a snapshot of this object, convert telemetries into prometheus metrics and persist the metrics every few seconds.
A DashboardAgent, which runs as a cronjob on every ray node, that collects node-level telemetries and send them to the TelemetryMetricRecorder

The c++ part consists of one component (to be deep dived more):

A cronjob runs on every ray worker and send telemetries to TelemetryMetricRecorder via grpc messages.

This PR migrates the TelemetryMetricRecorder from using opencencus to opentelemetry. The migration is also control via a flag (which turns on the new stuff by default). Fairly small PR (most files are auto-generated).

Test:

CI + new unit test
test_metrics_agent.py + test_reporter.py has a lot of e2e tests and they still work
e2e test in anyscale (go/cantel01)

aslonnie

approve for dependency change.

edoakes · 2025-05-16T21:16:00Z

🙌

edoakes · 2025-05-16T21:16:06Z

@MengjinYan can you review this pls?

python/ray/_private/telemetry/open_telemetry_metric_recorder.py

can-anyscale · 2025-05-22T23:10:19Z

@MengjinYan's comments

can-anyscale · 2025-05-23T01:57:17Z

@MengjinYan: I changed _observations_by_gauge_name back to the version that deduplicates tags instead of simply appending data points. Previously, I reset _observations_by_gauge_name at collection time (triggered by the /metrics endpoint), but this broke test_reporter in a way that is updated recently by @dayshah: the test calls /metrics twice in a row and the second call returns no data (see here). To keep the behavior as close as it was, I removed the reset of _observations_by_gauge_name. It now needs to be a map rather than a vector of data points to prevent memory leaks.

We can discuss more in person next week perhaps if this makes no sense.

This is a series of PR to migrate metric collection from opencencus to openlemetry. For context on the existing components, see #53098. --------------- This PR - Use open telemetry to export gauge metric to grpc endpoint - Grpc endpoint also export received metrics to prometheus server - Add unit tests + e2e tests. Note that I create a test job separately for open telemetry, the grpc server needs to use the network host directly (it doesn't work in a docker-in-docker environment from CI for reasons I don't 100% understand yet). Overall, when `RAY_experimental_enable_open_telemetry_on_core` is True, gauge metric uses open telemetry and everything else keeps using opencensus. Otherwise, everything will use opencensus. This ensure correctness for both options. Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com> Co-authored-by: Dhyey Shah <dhyey2019@gmail.com>

This is a series of PR to migrate metric collection from opencencus to openlemetry. For context on the existing components, see #53098. --------------- This PR - Support Counter metric for `open_telemetry_metric_recorder.cc` - Add a unittest to make sure it works - This feature is not used anywhere yet, since the counter part `open_telemetry_metric_recorder.py` needs to be able to consume the counter metric as well, if this part is used. I'll add that in the next PR Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com> Co-authored-by: Dhyey Shah <dhyey2019@gmail.com>

This is a series of PR to migrate metric collection from opencencus to openlemetry. For context on the existing components, see #53098. --------------- This PR - Support Counter metric for `open_telemetry_metric_recorder.cc` - Add a unittest to make sure it works - This feature is not used anywhere yet, since the counter part `open_telemetry_metric_recorder.py` needs to be able to consume the counter metric as well, if this part is used. I'll add that in the next PR Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com> Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

This is a series of PR to migrate metric collection from opencencus to openlemetry. For context on the existing components, see #53098. ------------ This PR - Support Counter metric on dashboard agent side - Support Counter metric e2e (from worker to dashboard agent) Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Cuong Nguyen <can@anyscale.com>

This is a series of PR to migrate metric collection from opencencus to openlemetry. For context on the existing components, see ray-project#53098. ------------ This PR - Support Counter metric on dashboard agent side - Support Counter metric e2e (from worker to dashboard agent) Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Cuong Nguyen <can@anyscale.com>

This is a series of PR to migrate metric collection from opencencus to openlemetry. For context on the existing components, see #53098. --------------- This PR - Support Counter metric for `open_telemetry_metric_recorder.cc` - Add a unittest to make sure it works - This feature is not used anywhere yet, since the counter part `open_telemetry_metric_recorder.py` needs to be able to consume the counter metric as well, if this part is used. I'll add that in the next PR Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com> Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

This is a series of PR to migrate metric collection from opencencus to openlemetry. For context on the existing components, see #53098. ------------ This PR - Support Counter metric on dashboard agent side - Support Counter metric e2e (from worker to dashboard agent) Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Cuong Nguyen <can@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

can-anyscale requested review from aslonnie, edoakes and richardliaw as code owners May 16, 2025 19:24

can-anyscale requested a review from a team May 16, 2025 19:24

can-anyscale added the go add ONLY when ready to merge, run all tests label May 16, 2025

aslonnie approved these changes May 16, 2025

View reviewed changes

can-anyscale force-pushed the can-tel01 branch from 143f3f6 to 89154c5 Compare May 16, 2025 20:37

edoakes assigned MengjinYan May 16, 2025

can-anyscale changed the title ~~[can][telemetry/01] migrate python telemetry recorder to opentelemetry~~ [core][telemetry/01] migrate python telemetry recorder to opentelemetry May 17, 2025

can-anyscale force-pushed the can-tel01 branch 2 times, most recently from 23327c7 to 3c4f326 Compare May 20, 2025 13:30

hainesmichaelc added the community-backlog label May 22, 2025

can-anyscale force-pushed the can-tel01 branch 3 times, most recently from 3345501 to 1e0aa1e Compare May 22, 2025 02:26

MengjinYan reviewed May 22, 2025

View reviewed changes

python/ray/_private/telemetry/open_telemetry_metric_recorder.py Show resolved Hide resolved

python/ray/_private/telemetry/open_telemetry_metric_recorder.py Show resolved Hide resolved

hainesmichaelc removed the community-backlog label May 22, 2025

can-anyscale force-pushed the can-tel01 branch from 1e0aa1e to 4f88256 Compare May 22, 2025 23:07

can-anyscale requested a review from MengjinYan May 22, 2025 23:10

can-anyscale force-pushed the can-tel01 branch 2 times, most recently from a744a44 to 4ff61cc Compare May 23, 2025 01:48

can-anyscale force-pushed the can-tel01 branch from 4ff61cc to f5269f9 Compare May 23, 2025 15:53

This was referenced May 23, 2025

[core][telemetry/02] opentelemetry recorder grpc service #53158

Merged

[core][telemetry/03] add OpenTelemetryMetricRecorder c++ client #53159

Merged

can-anyscale force-pushed the can-tel01 branch from f5269f9 to b3bc265 Compare May 23, 2025 21:29

can-anyscale mentioned this pull request May 27, 2025

[core][telemetry/04] register and record metrics on worker side #53209

Merged

can-anyscale mentioned this pull request Jun 5, 2025

[core][telemetry/10] record histogram metric e2e #53523

Closed

This was referenced Jun 11, 2025

[core][telemetry/10] support custom gauge+counter+sum metrics #53734

Merged

[core][telemetry/11] record histogram metric e2e #53740

Merged

can-anyscale mentioned this pull request Jun 18, 2025

[core][telemetry/12] record histogram metric e2e #53927

Merged

snorkelopstesting1-a11y mentioned this pull request Oct 22, 2025

[core][telemetry/11] record histogram metric e2e snorkel-marlin-repos/ray-project_ray_pr_53740_d99c81f7-14c3-4998-95b2-a7d3ec51e01c#1

Merged

snorkelopsstgtesting1-spec mentioned this pull request Oct 22, 2025

[core][telemetry/12] record histogram metric e2e snorkel-marlin-repos/ray-project_ray_pr_53927_182c0673-1f27-43ea-9c0e-a5b09940d9ac#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core][telemetry/01] migrate python telemetry recorder to opentelemetry#53098

[core][telemetry/01] migrate python telemetry recorder to opentelemetry#53098
can-anyscale merged 2 commits intomasterfrom
can-tel01

can-anyscale commented May 16, 2025 •

edited

Loading

Uh oh!

aslonnie left a comment

Uh oh!

edoakes commented May 16, 2025

Uh oh!

edoakes commented May 16, 2025

Uh oh!

Uh oh!

Uh oh!

can-anyscale commented May 22, 2025

Uh oh!

can-anyscale commented May 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

can-anyscale commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aslonnie left a comment

Choose a reason for hiding this comment

Uh oh!

edoakes commented May 16, 2025

Uh oh!

edoakes commented May 16, 2025

Uh oh!

Uh oh!

Uh oh!

can-anyscale commented May 22, 2025

Uh oh!

can-anyscale commented May 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

can-anyscale commented May 16, 2025 •

edited

Loading