Skip to content

[WIP][DataLoader] Implement BucketBatchIterableDataset#51126

Closed
ejguan wants to merge 4 commits intogh/ejguan/22/basefrom
gh/ejguan/22/head
Closed

[WIP][DataLoader] Implement BucketBatchIterableDataset#51126
ejguan wants to merge 4 commits intogh/ejguan/22/basefrom
gh/ejguan/22/head

Conversation

@ejguan
Copy link
Copy Markdown
Contributor

@ejguan ejguan commented Jan 26, 2021 •
edited
Loading

Stack from ghstack:

BucketBatch:
Get a chunk of data as a bucket, and sort the bucket by the specified key, then batching.
If sort key is not specified, directly use batchIterableDS..

  1. Implement BucketBatch for bucket sampler
  2. Improve BatchDS tests

Differential Revision: D26209890

@ejguan ejguan mentioned this pull request Jan 26, 2021
Closed
ejguan added a commit that referenced this pull request Jan 26, 2021
@ejguan
[WIP][DataLoader] Implement BucketBatchIterableDataset
fe282a5 
ghstack-source-id: 448a35e
Pull Request resolved: #51126
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Jan 26, 2021 •
edited
Loading

💊 CI failures summary and remediations

As of commit 9fcada9 (more details on the Dr. CI page):

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

See CircleCI build pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2 (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Feb 03 02:30:02 RuntimeError: Process 0 terminated or timed out after 100.03852462768555 seconds 
Feb 03 02:30:02 ERROR: test_pytorch_parity (__main__.TestZeroRedundancyOptimizerDistributed)
Feb 03 02:30:02 When combined with DDP, check that ZeroRedundancyOptimizer(optimizer) and the same monolithic optimizer
Feb 03 02:30:02 ----------------------------------------------------------------------
Feb 03 02:30:02 Traceback (most recent call last):
Feb 03 02:30:02   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 282, in wrapper
Feb 03 02:30:02     self._join_processes(fn)
Feb 03 02:30:02   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 399, in _join_processes
Feb 03 02:30:02     self._check_return_codes(elapsed_time)
Feb 03 02:30:02   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 440, in _check_return_codes
Feb 03 02:30:02     raise RuntimeError('Process {} terminated or timed out after {} seconds'.format(i, elapsed_time))
Feb 03 02:30:02 RuntimeError: Process 0 terminated or timed out after 100.03852462768555 seconds
Feb 03 02:30:02 
Feb 03 02:30:02 ----------------------------------------------------------------------
Feb 03 02:30:02 Ran 12 tests in 131.526s
Feb 03 02:30:02 
Feb 03 02:30:02 FAILED (errors=1)
Feb 03 02:30:02 Traceback (most recent call last):
Feb 03 02:30:02   File "test/run_test.py", line 925, in <module>
Feb 03 02:30:02     main()
Feb 03 02:30:02   File "test/run_test.py", line 904, in main
Feb 03 02:30:02     raise RuntimeError(err_message)

ci.pytorch.org: 1 failed

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

@ejguan
Update on "[WIP][DataLoader] Implement BucketBatchIterableDataset"
4e27ffd 
BucketBatch:
Get a chunk of data as a bucket, and sort the bucket by the specified key, then batching.
If sort key is not specified, directly use batchIterableDS..

1. Implement BucketBatch for bucket sampler
2. Improve BatchDS tests

[ghstack-poisoned]
@ejguan ejguan mentioned this pull request Feb 1, 2021
Closed
@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 2, 2021

Codecov Report

Merging #51126 (4e27ffd) into gh/ejguan/22/base (0c45d7c) will increase coverage by 0.00%.
The diff coverage is 86.66%.

@@                Coverage Diff                 @@
##           gh/ejguan/22/base   #51126   +/-   ##
==================================================
  Coverage              80.84%   80.85%           
==================================================
  Files                   1938     1938           
  Lines                 211251   211290   +39     
==================================================
+ Hits                  170791   170833   +42     
+ Misses                 40460    40457    -3

glaringlee
glaringlee approved these changes Feb 2, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@glaringlee glaringlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@ejguan
Update on "[WIP][DataLoader] Implement BucketBatchIterableDataset"
9fcada9 
BucketBatch:
Get a chunk of data as a bucket, and sort the bucket by the specified key, then batching.
If sort key is not specified, directly use batchIterableDS..

1. Implement BucketBatch for bucket sampler
2. Improve BatchDS tests

Differential Revision: [D26209890](https://our.internmc.facebook.com/intern/diff/D26209890)

[ghstack-poisoned]
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Feb 3, 2021

@ejguan merged this pull request in bea0519.

@facebook-github-bot facebook-github-bot deleted the gh/ejguan/22/head branch February 6, 2021 15:15
laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 24, 2026
@ejguan @facebook-github-bot
[WIP][DataLoader] Implement BucketBatchIterableDataset (pytorch#51126)
7e3f442 
Summary:
Pull Request resolved: pytorch#51126

BucketBatch:
Get a chunk of data as a bucket, and sort the bucket by the specified key, then batching.
If sort key is not specified, directly use batchIterableDS..

1. Implement BucketBatch for bucket sampler
2. Improve BatchDS tests

Test Plan: Imported from OSS

Reviewed By: H-Huang

Differential Revision: D26209890

Pulled By: ejguan

fbshipit-source-id: 8519e2e49da158b3fe32913c8f3cadfa6f3ff1fc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@glaringlee glaringlee glaringlee approved these changes

@VitalyFedyunin VitalyFedyunin Awaiting requested review from VitalyFedyunin

Assignees

No one assigned

Labels

cla signed Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ejguan @facebook-github-bot @glaringlee