[DataLoader2] Adding support for naive snapshotting#915
[DataLoader2] Adding support for naive snapshotting#915NivekT wants to merge 39 commits intogh/NivekT/102/basefrom
Conversation
[ghstack-poisoned]
| self.datapipe = self.reading_service.initialize(self.datapipe) | ||
| self._adapted = True | ||
| self.reading_service._restore_naive_datapipe_snapshot(n_samples_yielded, initial_seed) | ||
| # TODO: I might want to skip `initialize_iteration` after this???? |
There was a problem hiding this comment.
Note: long term we probably want to have a seed generator in DataLoader2
| shared_seed_int: int = shared_seed.item() # type: ignore[assignment] | ||
| _seed_generator = torch.Generator() | ||
| _seed_generator.manual_seed(shared_seed_int) | ||
| self._initial_seed = shared_seed_int |
There was a problem hiding this comment.
The way I should get original state, prior to generate_random_scalar_tensor I should get PyTorch global random state.
When I restore, my restoration function should restore that state prior to calling initialize_iteration (probably by iter(dp2))
There was a problem hiding this comment.
Think about what user will call (probably iter(dl2)), and how that will trigger various initialization methods
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
|
@NivekT has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Differential Revision: [D43644875](https://our.internmc.facebook.com/intern/diff/D43644875) [ghstack-poisoned]
Differential Revision: [D43644875](https://our.internmc.facebook.com/intern/diff/D43644875) [ghstack-poisoned]
Differential Revision: [D43644875](https://our.internmc.facebook.com/intern/diff/D43644875) [ghstack-poisoned]
Differential Revision: [D43644875](https://our.internmc.facebook.com/intern/diff/D43644875) [ghstack-poisoned]
Differential Revision: [D43644875](https://our.internmc.facebook.com/intern/diff/D43644875) [ghstack-poisoned]
|
Hi @NivekT! Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours needs attention. You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
Closing. Feel free to re-open if someone else would like to work on this. |
Stack from ghstack:
restore_iterationto ReadingService method for arbitrary checkpointing #1056Differential Revision: D43644875