Make HeaderIterDataPipe with limit=None a no-op#908
Make HeaderIterDataPipe with limit=None a no-op#908felix-newman wants to merge 2 commits intometa-pytorch:mainfrom
Conversation
|
Hi @felix-newman! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
| break | ||
| self.length = min(i, self.limit) # We know length with certainty when we reach here | ||
| # We know length with certainty when we reach here | ||
| self.length = i if self.limit is None else min(i, self.limit) |
There was a problem hiding this comment.
This can probably be simplified to using just i without a min. As incrementing is done before yielding or breaking, the length is i-1.
The following test case should be added: use a generator (has no len function) with some yields, drain it and assert self.length afterwards.
There was a problem hiding this comment.
TBH, I think self.length should even be eliminated here for the reason that the DataPipe graph for each epoch might have different lengths as users can always alter the graph after one epoch ends.
We can simply rely on limit and len(self.source_datapipe) IMHO
| break | ||
| self.length = min(i, self.limit) # We know length with certainty when we reach here | ||
| # We know length with certainty when we reach here | ||
| self.length = i if self.limit is None else min(i, self.limit) |
There was a problem hiding this comment.
TBH, I think self.length should even be eliminated here for the reason that the DataPipe graph for each epoch might have different lengths as users can always alter the graph after one epoch ends.
We can simply rely on limit and len(self.source_datapipe) IMHO
* Reraise TypeError with additional warning * Remove self.length as we can instead rely on limit and len(self.sourcedatapipe) * Use Optional instead of Union
NivekT
left a comment
There was a problem hiding this comment.
LGTM! Thanks for contributing
|
@NivekT has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Fixes #762
Changes