Skip to content

Stdout/stderr from all drivers are streamed to all other drivers. #4029

@robertnishihara

Description

@robertnishihara

As of #3892, the stdout/stderr of all drivers is streamed to all other drivers. An individual driver can opt out by calling ray.init(..., log_to_driver=False).

To reproduce this, do the following.

  1. Start Ray.

    ray start --head --redis-port=6379
    
  2. Start a driver that prints stuff.

    import ray
    import sys
    import time
    
    ray.init(redis_address='localhost:6379')
    
    @ray.remote
    def f():
        while True:
            print('hi')
            sys.stdout.flush()
            time.sleep(0.1)
    
    f.remote()
  3. Start another driver, and see that it receives the stdout from the first driver.

    import ray
    
    ray.init(redis_address='localhost:6379')

Some options to address this.

  1. Workers can belong to specific drivers (this means logs will be complicated for cross driver applications, e.g., named actors).
  2. Workers can redirect there stdout/stderr on a per-task basis.
  3. Workers can print extra annotations in their stdout/stderr which the log monitor can parse and use to choose which drivers to publish to.

Metadata

Metadata

Assignees

Labels

P1Issue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn't

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions