[DataPipe] Add container template for _Fork and _Demux#89216
[DataPipe] Add container template for _Fork and _Demux#89216ejguan wants to merge 2 commits intopytorch:masterfrom
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/89216
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 2 PendingAs of commit 30e5d2e: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
| @abstractmethod | ||
| def get_length_by_instance(self, instance_id: int): | ||
| r""" | ||
| Raise TypeError if it's not supposed to be implemented to support `list(datapipe)` | ||
| """ |
There was a problem hiding this comment.
Add another method to add the functionality to provide different length per Child.
cc: @NivekT
I need it for demux have a deterministic classifier_fn.
There was a problem hiding this comment.
meta-pytorch/data#903 is an example of the usage of get_length_by_instance.
|
@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Summary: The only reason I added this DataPipe without directly using `enumereate().demux().drop_index()` is this `RoundRobinDemux` should provide a valid length. So, this PR needs pytorch/pytorch#89216 landed And, this DataPipe will be used for `Proto2RS`. Side change: Move `Unzip` to `combining.py` as well. Pull Request resolved: #903 Reviewed By: wenleix Differential Revision: D41387813 Pulled By: ejguan fbshipit-source-id: b3a5939763b53875dd223809041ad8d0337b1bda
- This would remove the hard-coded check within `_ChildDataPipe`. - Add `get_length_by_instance` to parent class to make sure there is a chance that child DataPipe can have different lengths - Prevent Error when `__del__` executed when the object has already been removed Pull Request resolved: pytorch#89216 Approved by: https://github.com/NivekT
_ChildDataPipe.get_length_by_instanceto parent class to make sure there is a chance that child DataPipe can have different lengths__del__executed when the object has already been removed