Skip to content

Fix hang when wandb.init is interrupted mid setup using wandb-service#3569

Merged
raubitsj merged 4 commits intomasterfrom
WB-9386
Apr 28, 2022
Merged

Fix hang when wandb.init is interrupted mid setup using wandb-service#3569
raubitsj merged 4 commits intomasterfrom
WB-9386

Conversation

@kptkin
Copy link
Copy Markdown
Collaborator

@kptkin kptkin commented Apr 25, 2022

Fixes WB-9386

Description

What does the PR do?

when service enabled and code fails during wandb.init before this PR it would hang because we would clean up the backend hence unable to properly terminate the program with service this PR fixes this issue, by gating cleaning backend only in the non-service case

Testing

How was this PR tested?

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 25, 2022

Codecov Report

Merging #3569 (64e771a) into master (81a9cb7) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3569      +/-   ##
==========================================
- Coverage   81.55%   81.51%   -0.05%     
==========================================
  Files         240      240              
  Lines       29506    29508       +2     
==========================================
- Hits        24064    24052      -12     
- Misses       5442     5456      +14     
Flag Coverage Δ
functest 57.61% <50.00%> (+0.02%) ⬆️
unittest 71.75% <100.00%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
wandb/sdk/wandb_init.py 85.45% <100.00%> (ø)
wandb/sdk/wandb_run.py 89.85% <100.00%> (+0.01%) ⬆️
wandb/sdk/internal/artifacts.py 80.00% <0.00%> (-6.32%) ⬇️
wandb/sdk/internal/internal_api.py 82.47% <0.00%> (-1.50%) ⬇️
wandb/filesync/step_prepare.py 93.50% <0.00%> (-1.30%) ⬇️
wandb/sdk/launch/agent/agent.py 92.70% <0.00%> (-0.73%) ⬇️
wandb/sdk/lib/git.py 76.19% <0.00%> (ø)
wandb/sdk/internal/meta.py 90.68% <0.00%> (+3.10%) ⬆️

@kptkin kptkin requested a review from raubitsj April 25, 2022 17:03
@kptkin kptkin changed the title avoid hang fix(sdk-service) avoid hang Apr 26, 2022
@kptkin kptkin marked this pull request as ready for review April 26, 2022 04:26
@kptkin kptkin modified the milestones: sdk-2022-05.2, sdk-2022-05.1 Apr 26, 2022
@raubitsj raubitsj self-assigned this Apr 26, 2022
@kptkin kptkin changed the title fix(sdk-service) avoid hang [WB-9386] fix hang when wandb.init is interrupted mid setup with service on Apr 26, 2022
Comment thread wandb/sdk/wandb_run.py Outdated
if log_dir:
# printer = printer or get_printer(settings._jupyter)
log_dir = os.path.dirname(log_dir.replace(os.getcwd(), "."))
log_dir = os.path.dirname(log_dir)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the old behavior of showing a relative path was nicer:

OLD:
wandb: Find logs at: ./wandb/run-20220427_161225-3tz4bz6z/logs
NEW:
wandb: Find logs at: /Users/jeff/work/wb/client/functional_tests/core/wandb/run-20220427_161208-2893jyzh/logs

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure I can undo this change...

@raubitsj raubitsj merged commit 4f21ae3 into master Apr 28, 2022
@raubitsj raubitsj deleted the WB-9386 branch April 28, 2022 00:49
@raubitsj raubitsj changed the title [WB-9386] fix hang when wandb.init is interrupted mid setup with service on Fix hang when wandb.init is interrupted mid setup using wandb-service May 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants