Adding ability to redefine cache timeout#571
Adding ability to redefine cache timeout#571VitalyFedyunin wants to merge 7 commits intogh/VitalyFedyunin/12/basefrom
Conversation
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
ejguan
left a comment
There was a problem hiding this comment.
LGTM. Thank you.
I just realize one general question for Adapter. Since adapters would modify the DataPipe graph in-place, do we want to revert the modification for the DataPipe graph after one?epoch?
|
IMO DLv2 should apply adapter modifications once on the local copy of the graph it gets from the args. |
[ghstack-poisoned]
[ghstack-poisoned]
|
@VitalyFedyunin has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Differential Revision: [D37723437](https://our.internmc.facebook.com/intern/diff/D37723437) [ghstack-poisoned]
|
@VitalyFedyunin has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
parmeet
left a comment
There was a problem hiding this comment.
LGTM! Thanks @VitalyFedyunin for adding the ability to manually set time for caching files. Just to be sure, the users can add this adapter on top of datapipe we return for datasets, in other words, we do not have to add this adapter internally in the dataset implementation right?
Correct |
Stack from ghstack (oldest at bottom):
Differential Revision: D37723437