Skip to content

Remove handling workers waiting to put in dataloader.__del__#7140

Closed
ssnl wants to merge 2 commits intopytorch:masterfrom
ssnl:dl
Closed

Remove handling workers waiting to put in dataloader.__del__#7140
ssnl wants to merge 2 commits intopytorch:masterfrom
ssnl:dl

Conversation

@ssnl
Copy link
Collaborator

@ssnl ssnl commented May 1, 2018

Now that each worker has its own queue, each worker has at most 2 batches, which should be far below SimpleQueue's limit.

Also fixes #6932

@onnxbot-worker-3 onnxbot-worker-3 mentioned this pull request May 1, 2018
Open
zou3519
zou3519 reviewed May 1, 2018
View reviewed changes
torch/utils/data/dataloader.py Outdated

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

apaszke
apaszke previously requested changes May 2, 2018
View reviewed changes
torch/utils/data/dataloader.py Outdated

This comment was marked as off-topic.

Sign in to view

@ssnl ssnl force-pushed the dl branch 2 times, most recently from c516864 to b28c549 Compare May 3, 2018 20:47
@ezyang ezyang dismissed apaszke’s stale review May 4, 2018 21:02

CR comment fixed

ssnl
ssnl commented May 4, 2018
View reviewed changes
Copy link
Collaborator Author

@ssnl ssnl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to leave zombie process alive in py2 (see CI). I'll figure it out.

@ssnl
Copy link
Collaborator Author

ssnl commented May 25, 2018

@pytorchbot retest this please

@zou3519 zou3519 mentioned this pull request May 29, 2018
Closed
@ssnl
Copy link
Collaborator Author

ssnl commented May 29, 2018

@pytorchbot retest this please

@ssnl
Copy link
Collaborator Author

ssnl commented May 29, 2018

@pytorchbot retest this please

@ssnl
Copy link
Collaborator Author

ssnl commented May 29, 2018

@pytorchbot retest this please

3 similar comments
@ssnl
Copy link
Collaborator Author

ssnl commented May 30, 2018

@pytorchbot retest this please

@ezyang
Copy link
Contributor

ezyang commented May 30, 2018

@pytorchbot retest this please

@ssnl
Copy link
Collaborator Author

ssnl commented May 30, 2018

@pytorchbot retest this please

@ssnl
Copy link
Collaborator Author

ssnl commented Jun 1, 2018 •
edited
Loading

will reopen later after figuring out the mp issues

@ssnl ssnl closed this Jun 1, 2018
@ssnl ssnl mentioned this pull request Jul 20, 2018
Closed
facebook-github-bot pushed a commit that referenced this pull request Jul 23, 2018
@ssnl @facebook-github-bot
Fix dataloader hang when it is not completely iterated (#9655)
9ee5133 
Summary:
second trial of #7140

cc csarofeen Let's see if this works. It passes everything locally.
Pull Request resolved: #9655

Differential Revision: D8940177

Pulled By: SsnL

fbshipit-source-id: 8d6340fc9f7355c71e1e26b262da166402faa158
ssnl added a commit that referenced this pull request Jul 23, 2018
@ssnl
Fix dataloader hang when it is not completely iterated (#9655)
2edf053 
Summary:
second trial of #7140

cc csarofeen Let's see if this works. It passes everything locally.
Pull Request resolved: #9655

Differential Revision: D8940177

Pulled By: SsnL

fbshipit-source-id: 8d6340fc9f7355c71e1e26b262da166402faa158
jramseyer pushed a commit to jramseyer/pytorch that referenced this pull request Jul 30, 2018
@ssnl @jramseyer
Fix dataloader hang when it is not completely iterated (pytorch#9655)
045a39e 
Summary:
second trial of pytorch#7140

cc csarofeen Let's see if this works. It passes everything locally.
Pull Request resolved: pytorch#9655

Differential Revision: D8940177

Pulled By: SsnL

fbshipit-source-id: 8d6340fc9f7355c71e1e26b262da166402faa158
goodlux pushed a commit to goodlux/pytorch that referenced this pull request Aug 15, 2018
@ssnl
Fix dataloader hang when it is not completely iterated (pytorch#9655)
ef13ba8 
Summary:
second trial of pytorch#7140

cc csarofeen Let's see if this works. It passes everything locally.
Pull Request resolved: pytorch#9655

Differential Revision: D8940177

Pulled By: SsnL

fbshipit-source-id: 8d6340fc9f7355c71e1e26b262da166402faa158
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zou3519 zou3519 zou3519 left review comments

@ezyang ezyang ezyang approved these changes

@colesbury colesbury Awaiting requested review from colesbury

@gchanan gchanan Awaiting requested review from gchanan

@soumith soumith Awaiting requested review from soumith

@zdevito zdevito Awaiting requested review from zdevito

+1 more reviewer

@apaszke apaszke apaszke left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

open source

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Exception NameError: "global name 'FileNotFoundError' is not defined"

4 participants

@ssnl @ezyang @apaszke @zou3519