Skip to content

[WIP][DataLoader] Implement CallableIterableDataset#50045

Closed
ejguan wants to merge 9 commits intogh/ejguan/17/basefrom
gh/ejguan/17/head
Closed

[WIP][DataLoader] Implement CallableIterableDataset#50045
ejguan wants to merge 9 commits intogh/ejguan/17/basefrom
gh/ejguan/17/head

Conversation

@ejguan
Copy link
Copy Markdown
Contributor

@ejguan ejguan commented Jan 4, 2021 •
edited
Loading

Stack from ghstack:

Add CallableIterableDataset
Modify CollateIterableDataset as another callable

Differential Revision: D26209889

ejguan added a commit that referenced this pull request Jan 4, 2021
@ejguan
[WIP][DataLoader] Implement CallableIterableDataset
f143bc9 
ghstack-source-id: 916de11
Pull Request resolved: #50045
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Jan 4, 2021 •
edited
Loading

💊 CI failures summary and remediations

As of commit 64a8664 (more details on the Dr. CI page):

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

See CircleCI build pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2 (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Feb 03 02:34:07 RuntimeError: Process 0 terminated or timed out after 100.03775095939636 seconds 
Feb 03 02:34:07 ERROR: test_pytorch_parity (__main__.TestZeroRedundancyOptimizerDistributed)
Feb 03 02:34:07 When combined with DDP, check that ZeroRedundancyOptimizer(optimizer) and the same monolithic optimizer
Feb 03 02:34:07 ----------------------------------------------------------------------
Feb 03 02:34:07 Traceback (most recent call last):
Feb 03 02:34:07   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 282, in wrapper
Feb 03 02:34:07     self._join_processes(fn)
Feb 03 02:34:07   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 399, in _join_processes
Feb 03 02:34:07     self._check_return_codes(elapsed_time)
Feb 03 02:34:07   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 440, in _check_return_codes
Feb 03 02:34:07     raise RuntimeError('Process {} terminated or timed out after {} seconds'.format(i, elapsed_time))
Feb 03 02:34:07 RuntimeError: Process 0 terminated or timed out after 100.03775095939636 seconds
Feb 03 02:34:07 
Feb 03 02:34:07 ----------------------------------------------------------------------
Feb 03 02:34:07 Ran 12 tests in 131.462s
Feb 03 02:34:07 
Feb 03 02:34:07 FAILED (errors=1)
Feb 03 02:34:07 Traceback (most recent call last):
Feb 03 02:34:07   File "test/run_test.py", line 925, in <module>
Feb 03 02:34:07     main()
Feb 03 02:34:07   File "test/run_test.py", line 904, in main
Feb 03 02:34:07     raise RuntimeError(err_message)

ci.pytorch.org: 1 failed

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

@ejguan
Update on "[WIP][DataLoader] Implement CallableIterableDataset"
e641a37 
Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable

Differential Revision: [D25757521](https://our.internmc.facebook.com/intern/diff/D25757521)

[ghstack-poisoned]
ejguan added a commit that referenced this pull request Jan 6, 2021
@ejguan
[WIP][DataLoader] Implement CallableIterableDataset
fa32942 
ghstack-source-id: 083eaff
Pull Request resolved: #50045
@ejguan
Update on "[WIP][DataLoader] Implement CallableIterableDataset"
6a60e55 
Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable

Differential Revision: [D25757521](https://our.internmc.facebook.com/intern/diff/D25757521)

[ghstack-poisoned]
ejguan added a commit that referenced this pull request Jan 8, 2021
@ejguan
[WIP][DataLoader] Implement CallableIterableDataset
7e576fe 
ghstack-source-id: 98974f6
Pull Request resolved: #50045
@ejguan
Update on "[WIP][DataLoader] Implement CallableIterableDataset"
692d975 
Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable

Differential Revision: [D25757521](https://our.internmc.facebook.com/intern/diff/D25757521)

[ghstack-poisoned]
ejguan added a commit that referenced this pull request Jan 8, 2021
@ejguan
[WIP][DataLoader] Implement CallableIterableDataset
03b44f6 
ghstack-source-id: 9d4846b
Pull Request resolved: #50045
@ejguan
Update on "[WIP][DataLoader] Implement CallableIterableDataset"
30c9a2d 
Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable


[ghstack-poisoned]
ejguan added a commit that referenced this pull request Jan 11, 2021
@ejguan
[WIP][DataLoader] Implement CallableIterableDataset
f5a0df1 
ghstack-source-id: 7398ec7
Pull Request resolved: #50045
@ejguan
Update on "[WIP][DataLoader] Implement CallableIterableDataset"
67d7372 
Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable

[ghstack-poisoned]
@ejguan ejguan mentioned this pull request Jan 21, 2021
Closed
8 tasks
@ejguan
Update on "[WIP][DataLoader] Implement CallableIterableDataset"
2684b8d 
Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable

[ghstack-poisoned]
ejguan added a commit that referenced this pull request Jan 25, 2021
@ejguan
[WIP][DataLoader] Implement CallableIterableDataset
9c1da7d 
ghstack-source-id: 2ce9789
Pull Request resolved: #50045
@ejguan ejguan mentioned this pull request Jan 26, 2021
Closed
@ejguan
Update on "[WIP][DataLoader] Implement CallableIterableDataset"
0c45d7c 
Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable

[ghstack-poisoned]
@ejguan ejguan mentioned this pull request Feb 1, 2021
Closed
glaringlee
glaringlee approved these changes Feb 2, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@glaringlee glaringlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ejguan
Update on "[WIP][DataLoader] Implement CallableIterableDataset"
64a8664 
Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable

Differential Revision: [D26209889](https://our.internmc.facebook.com/intern/diff/D26209889)

[ghstack-poisoned]
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Feb 3, 2021

@ejguan merged this pull request in 14ee63f.

@facebook-github-bot facebook-github-bot deleted the gh/ejguan/17/head branch February 6, 2021 15:15
laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 24, 2026
@ejguan @facebook-github-bot
[WIP][DataLoader] Implement CallableIterableDataset (pytorch#50045)
959789e 
Summary:
Pull Request resolved: pytorch#50045

Add **CallableIterableDataset**
Modify **CollateIterableDataset** as another callable

Test Plan: Imported from OSS

Reviewed By: H-Huang

Differential Revision: D26209889

Pulled By: ejguan

fbshipit-source-id: d4773026c1269e43b29a3efb16e36e1865fdd024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@glaringlee glaringlee glaringlee approved these changes

@VitalyFedyunin VitalyFedyunin Awaiting requested review from VitalyFedyunin

Assignees

No one assigned

Labels

cla signed Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ejguan @facebook-github-bot @glaringlee