Skip test_metrics_agent_with_open_telemetry on mac#53917
Conversation
ddbfaf5 to
e267244
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR reverts the previous move of the OpenTelemetry tests into a pytest module and instead isolates them to Linux-only runs to avoid macOS failures.
- Adds a new
py_test_module_listentry for OpenTelemetry tests with the required feature flags - Introduces a dedicated Buildkite step to run the OpenTelemetry tests
- Ensures the feature remains behind a flag so production remains unaffected
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| python/ray/tests/BUILD | Added a pytest module list for OpenTelemetry tests with env flags, tags, and dependencies |
| .buildkite/core.rayci.yml | Added a new Buildkite step to run OpenTelemetry tests in Docker with feature flags |
Comments suppressed due to low confidence (2)
.buildkite/core.rayci.yml:116
- This step should be restricted to Linux agents only to avoid running on macOS. Consider adding an
os: linuxtag or agent queue filter under this step.
- label: ":ray: core: open telemetry tests"
.buildkite/core.rayci.yml:122
- The Bazel invocation references
test_metrics_agentand includes a straycoreargument, but the new test target is namedtest_metrics_agent_open_telemetry. Update the target to//python/ray/tests:test_metrics_agent_open_telemetryand remove the extra argument.
- bazel run //ci/ray_ci:test_in_docker -- //python/ray/tests:test_metrics_agent core
e267244 to
1de393b
Compare
1de393b to
d77ba7c
Compare
|
do we know why it's failing on mac and if it works locally on your mac / apple silicon macs? would also just prefer doing pytest.skip, based on os which makes it more clear that it's not running on a certain os |
|
@dayshah ah nice skipping at pytest level is way better I know why it fails on mac yes; in cases where a metric is exported with a missing tag (e.g., the metric is defined with tags A and B but only exported with tag A), OpenTelemetry on macOS CI can misalign tag key-value pairs. You can see an example here: https://buildkite.com/ray-project/postmerge-macos/builds/6168/steps/canvas?jid=01976d5d-0669-485b-8a2d-e8288d8dedcf#01976d5d-0669-485b-8a2d-e8288d8dedcf/6-5824. This might be a bug in the opentelemetry-prometheus-exporter package. We're upgrading that package in another PR, and if the issue persists afterward, I'll look into a different fix. I haven’t tested it on my local Mac yet. |
d77ba7c to
50d429a
Compare
|
@dayshah's comments |
This test is failing on mac (ray-project#53828). Remove it on mac so it doesn't pollute go/flaky. This feature is behind a flag so it doesn't affect production. Closes ray-project#53828 Test: - CI Signed-off-by: can <can@anyscale.com>
This test is failing on mac (#53828). Remove it on mac so it doesn't pollute go/flaky. This feature is behind a flag so it doesn't affect production.
Closes #53828
Test: