Skip to content

[Metrics] Custom metrics don't work after calling ray.shutdown() followed by ray.init() #13532

@architkulkarni

Description

@architkulkarni

What is the problem?

Ray version and other system information (Python version, TensorFlow version, OS):
Ray master, Python 3.6, Mac OS

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

import ray
from ray.util.metrics import Count

print(ray.init())
ray.shutdown()

print(ray.init())
counter = Count("TEST_COUNTER", description="desc")                                                                                                
counter.record(1)

This metric never appears at localhost:<metrics-export-port>. However, if you remove the initial ray.init() ray.shutdown(), it works.

Adding debug print statements shows that Ray::Stats::Shutdown() is correctly being called in ray.shutdown(), and Ray::Stats::Init() is being correctly being called in the second (and first) ray.init(), so I'm not sure what the problem is.

  • I have verified my script runs in a clean environment and reproduces the issue.
  • I have verified the issue also occurs with the latest wheels.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Issue moderate in impact or severitybugSomething that is supposed to be working; but isn'tcoreIssues that should be addressed in Ray CoreobservabilityIssues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profilingpending-cleanupThis issue is pending cleanup. It will be removed in 2 weeks after being assigned.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions