Skip to content

Stream logs to driver by default.#3892

Merged
pcmoritz merged 23 commits intoray-project:masterfrom
robertnishihara:logtodriver
Feb 8, 2019
Merged

Stream logs to driver by default.#3892
pcmoritz merged 23 commits intoray-project:masterfrom
robertnishihara:logtodriver

Conversation

@robertnishihara
Copy link
Copy Markdown
Collaborator

@robertnishihara robertnishihara commented Jan 29, 2019

This fixes #2173.

In this PR:

  • By default, worker stdout/stderr on all machines (including the single machine case) are written to files and streamed to the driver via file on disk -> log_monitor -> redis -> driver.
  • If the log monitor dies, an exception is pushed to the driver.
  • We now shutdown driver threads when we call ray.shutdown(). This is most relevant for our tests.

Possible TODO:

  • Make sure we don't ignore files once a large number of workers have been created.
  • Think through relevant thread safety issues.
  • Format print statements to include node and process ID of process that logged the statement, maybe function name as well.
  • Code cleanups (hard coded strings).
  • Isolation between drivers, ideally only print logging statements for the relevant driver.
  • Think through the API (naming, and in particular, the interaction with redirect_output and redirect_worker_output).
  • Tests
  • Push error to all drivers if log monitor fails.
  • Test log monitor failure.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provide option to print task stdout to driver on a cluster in real time

5 participants