Skip to content

[data loader] Graceful data loader threads exit on KeyboardInterrupt #22924

Open
Open
[data loader] Graceful data loader threads exit on KeyboardInterrupt#22924
Labels
module: dataloaderRelated to torch.utils.data.DataLoader and Samplerneeds reproductionEnsure you have actionable steps to reproduce the issue. Someone else needs to confirm the repro.triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module
@vadimkantorov

Description

@vadimkantorov
vadimkantorov
opened on Jul 16, 2019

During training with PyTorch 1.2.0.dev20190607 I pressed Ctrl+C and got the following:

KeyboardInterrupt
Traceback (most recent call last):
  File "/miniconda/lib/python3.7/multiprocessing/queues.py", line 242, in _feed
    send_bytes(obj)
Fatal Python error: could not acquire lock for <_io.BufferedWriter name='<stderr>'> at interpreter shutdown, possibly due to daemon threads

Thread 0x00007f49347f8700 (most recent call first):

Thread 0x00007f49357fa700 (most recent call first):

Thread 0x00007f4934ff9700 (most recent call first):

Thread 0x00007f494f7fe700 (most recent call first):
  File "/miniconda/lib/python3.7/traceback.py", line 105 in print_exception
  File "/miniconda/lib/python3.7/traceback.py", line 163 in print_exc
  File "/miniconda/lib/python3.7/multiprocessing/queues.py", line 273 in _on_queue_feeder_error
  File "/miniconda/lib/python3.7/multiprocessing/queues.py", line 264 in _feed
  File "/miniconda/lib/python3.7/threading.py", line 865 in run
  File "/miniconda/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/miniconda/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f49c70e5700 (most recent call first):
  File "/miniconda/lib/python3.7/threading.py", line 300 in wait
  File "/miniconda/lib/python3.7/queue.py", line 179 in get
  File "/miniconda/lib/python3.7/site-packages/tensorboard/summary/writer/event_file_writer.py", line 204 in run
  File "/miniconda/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/miniconda/lib/python3.7/threading.py", line 885 in _bootstrap

Current thread 0x00007f49df943700 (most recent call first):
train.sh: line 1:  4022 Aborted                 (core dumped) python3 train.py

It would be great if data loader threads could die less verbosely (and mysteriously) on KeyboardInterrupt.

Metadata

Metadata

Assignees

No one assigned

    Labels

    module: dataloaderRelated to torch.utils.data.DataLoader and Samplerneeds reproductionEnsure you have actionable steps to reproduce the issue. Someone else needs to confirm the repro.triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions