Skip to content

[core] recover startup logs#12876

Merged
richardliaw merged 3 commits intoray-project:masterfrom
richardliaw:fix-p0
Dec 15, 2020
Merged

[core] recover startup logs#12876
richardliaw merged 3 commits intoray-project:masterfrom
richardliaw:fix-p0

Conversation

@richardliaw
Copy link
Copy Markdown
Contributor

@richardliaw richardliaw commented Dec 15, 2020 •
edited
Loading

Why are these changes needed?

Startup logs were going to devnull :) This change reverts the regression, and it now works for me on my cluster.

Note that this also means that monitor will have 3 files -- monitor.log, monitor.err, and monitor.out.

This is only a temporary measure to unblock the release

Closes #12771

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@richardliaw
try-fix
73d1418 
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
@richardliaw
Copy link
Copy Markdown
Contributor Author

richardliaw commented Dec 15, 2020

cc @rkooo567, i haven't tried this out yet but i think this is the culprit.

I'll run this on a cluster tomorrow.

@rkooo567 rkooo567 self-assigned this Dec 15, 2020
@rkooo567
Copy link
Copy Markdown
Contributor

rkooo567 commented Dec 15, 2020 •
edited
Loading

We don't redirect stdout and stderr directly to log file anymore btw (except worker logs). We should use logger.info instead in this case

@richardliaw
push
df779d2 
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
@richardliaw richardliaw marked this pull request as ready for review December 15, 2020 18:11
@rkooo567 rkooo567 mentioned this pull request Dec 15, 2020
Closed
@richardliaw
update-comment
1089a43 
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
@richardliaw richardliaw added the release-blocker P0 Issue that blocks the release label Dec 15, 2020
edoakes
edoakes approved these changes Dec 15, 2020
View reviewed changes
Copy link
Copy Markdown
Collaborator

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for now. @rkooo567 can you make sure the proper long-term fix gets merged?

@richardliaw richardliaw merged commit 87cf1a9 into ray-project:master Dec 15, 2020
@richardliaw richardliaw deleted the fix-p0 branch December 15, 2020 21:49
mfitton pushed a commit that referenced this pull request Dec 21, 2020
@richardliaw @mfitton
[core] recover startup logs (#12876)
d9fc24a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@edoakes edoakes edoakes approved these changes

@ericl ericl Awaiting requested review from ericl

@wuisawesome wuisawesome Awaiting requested review from wuisawesome

Assignees

@rkooo567 rkooo567

Labels

release-blocker P0 Issue that blocks the release

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Autoscaler] Startup logs are missing

3 participants

@richardliaw @rkooo567 @edoakes