Skip to content

Fix handling keyboard interrupt to avoid hangs with wandb-service enabled#3566

Merged
raubitsj merged 6 commits intomasterfrom
WB-9364
Apr 28, 2022
Merged

Fix handling keyboard interrupt to avoid hangs with wandb-service enabled#3566
raubitsj merged 6 commits intomasterfrom
WB-9364

Conversation

@kptkin
Copy link
Copy Markdown
Collaborator

@kptkin kptkin commented Apr 24, 2022

Fixes WB-9364
Fixes #3542

Description

What does the PR do?

Fixes a hang issues that is caused by the fact that the same keyboardInterrupt is sent both to the service as well as the user process.

This fix is only valid for posix based OS (a fix for windows will be done in a separate PR)

Testing

How was this PR tested?

Manually test and verified that it actually terminates properly

@kptkin kptkin changed the title start new session for service to avoid start new session for service to avoid hangs Apr 24, 2022
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 24, 2022

Codecov Report

Merging #3566 (1c67472) into master (4f21ae3) will increase coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3566      +/-   ##
==========================================
+ Coverage   81.54%   81.55%   +0.01%     
==========================================
  Files         240      240              
  Lines       29508    29508              
==========================================
+ Hits        24061    24064       +3     
+ Misses       5447     5444       -3     
Flag Coverage Δ
functest 57.62% <100.00%> (+0.04%) ⬆️
unittest 71.78% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
wandb/sdk/service/service.py 89.74% <100.00%> (ø)
wandb/sdk/internal/artifacts.py 83.15% <0.00%> (-3.16%) ⬇️
wandb/sdk/internal/internal_api.py 83.55% <0.00%> (-0.41%) ⬇️
wandb/sdk/lib/git.py 76.19% <0.00%> (ø)
wandb/sdk/service/server_sock.py 92.89% <0.00%> (+1.01%) ⬆️
wandb/sdk/lib/sock_client.py 94.16% <0.00%> (+1.66%) ⬆️
wandb/sdk/internal/meta.py 90.68% <0.00%> (+3.10%) ⬆️

@raubitsj raubitsj added this to the sdk-2022-05.1 milestone Apr 26, 2022
@kptkin kptkin marked this pull request as ready for review April 26, 2022 20:29
@kptkin kptkin changed the title start new session for service to avoid hangs [WB-9364] Fixes handling keyboard interrupt to avoid hangs with service enabled Apr 26, 2022
@raubitsj raubitsj merged commit ee010d4 into master Apr 28, 2022
@raubitsj raubitsj deleted the WB-9364 branch April 28, 2022 01:31
@raubitsj raubitsj changed the title [WB-9364] Fixes handling keyboard interrupt to avoid hangs with service enabled Fix handling keyboard interrupt to avoid hangs with wandb-service enabled May 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CLI]: W&B service deadlock on KeyboardInterrupt with PyTorch-Lightning and Hydra

3 participants