fix: WandbLogger crashing silently on a FileNotFoundError#50308
fix: WandbLogger crashing silently on a FileNotFoundError#50308matthewdeng merged 6 commits intoray-project:masterfrom
Conversation
Signed-off-by: Daniel Sperber <github.blurry@9ox.net>
7e42185 to
139cc83
Compare
This comment was marked as outdated.
This comment was marked as outdated.
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
|
| except FileNotFoundError as e: | ||
| logger.error( | ||
| "FileNotFoundError: Possible reason: relative instead of absolute path: %s", | ||
| e, | ||
| ) |
There was a problem hiding this comment.
Were you able verify locally if this resolves the issue?
There was a problem hiding this comment.
Thanks for picking this up. Yes, If I include this snippet I get the error logged in my terminal:
(_WandbLoggingActor pid=1189334) FileNotFoundError: Possible reason: relative instead of absolute path: [Errno 2] No such file or directory: 'videos/vid.mp4' And as I do not re-raise the error the logger stays alive. The data for that iteration is lost though (no entry for step 10).
Currently logs will be truncated and lost & without any error:
I think I could extend the warning message to make this clear and forward this issue wandb as well.
Signed-off-by: Daraan <github.blurry@9ox.net>
|
I just remembered the reason why relative paths fail. The working directory is inside the session folder and not directory where I executed |
matthewdeng
left a comment
There was a problem hiding this comment.
Looks good, just one minor adjustment to the error message.
| except FileNotFoundError as e: | ||
| logger.error( | ||
| "FileNotFoundError: This result will not be logged to WandB. " | ||
| "Possible reason: relative instead of absolute path for file %s", | ||
| e, | ||
| ) |
There was a problem hiding this comment.
| except FileNotFoundError as e: | |
| logger.error( | |
| "FileNotFoundError: This result will not be logged to WandB. " | |
| "Possible reason: relative instead of absolute path for file %s", | |
| e, | |
| ) | |
| except FileNotFoundError as e: | |
| logger.error( | |
| f"FileNotFoundError: Did not log result to Weights & Biases. " | |
| f"Possible cause: relative file path used instead of an absolute path. " | |
| f"Error: {e}" | |
| ) |
There was a problem hiding this comment.
Thank you. I incorporated you wording your wording. Concerning the f-string I am bit reluctant to using them as they go against flake8 G004 so I left them out for now.
b7223cf to
cb6507e
Compare
Signed-off-by: Daniel Sperber <github.blurry@9ox.net> Signed-off-by: Daraan <github.blurry@9ox.net> Co-authored-by: matthewdeng <matt@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: Daniel Sperber <github.blurry@9ox.net> Signed-off-by: Daraan <github.blurry@9ox.net> Co-authored-by: matthewdeng <matt@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Fixes: #50307