-
Notifications
You must be signed in to change notification settings - Fork 7.4k
Closed
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn't
Description
As of #3892, the stdout/stderr of all drivers is streamed to all other drivers. An individual driver can opt out by calling ray.init(..., log_to_driver=False).
To reproduce this, do the following.
-
Start Ray.
ray start --head --redis-port=6379 -
Start a driver that prints stuff.
import ray import sys import time ray.init(redis_address='localhost:6379') @ray.remote def f(): while True: print('hi') sys.stdout.flush() time.sleep(0.1) f.remote()
-
Start another driver, and see that it receives the stdout from the first driver.
import ray ray.init(redis_address='localhost:6379')
Some options to address this.
- Workers can belong to specific drivers (this means logs will be complicated for cross driver applications, e.g., named actors).
- Workers can redirect there stdout/stderr on a per-task basis.
- Workers can print extra annotations in their stdout/stderr which the log monitor can parse and use to choose which drivers to publish to.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn't