[DataPipe] Simple graph snapshotting by NivekT · Pull Request #79479 · pytorch/pytorch

NivekT · 2022-06-13T23:57:05Z

Stack from ghstack (oldest at bottom):

-> [DataPipe] Simple graph snapshotting #79479

This mostly completes the "poor man's snapshotting" implementation (named "simple snapshotting"). This is the most basic version of snapshotting but it should work for all DataPipes. I will be adding more efficient implementation for different types of DataPipes in future PRs.

Implementation

The general idea of the simple snapshot is that we will:

Create a new iterator
Move that iterator forward by n_iterations
Save that as the _fast_forward_iterator of the DataPipe
The next time iter is called on the DataPipe, use the _fast_forward_iterator

Usage

As of this implementation, the usage will something like:

rng = torch.Generator()
initial_rng_state = rng.get_state()
datapipe: IterDataPipe = ...
# Some usage of the DataPipe, here maybe yielding the first 5 values
n_iter = 5
it = iter(datapipe)
for _ in range(n_iter):
    next(it)
serialized_graph = pickle.dumps(datapipe)

# The serialized object has most of the sufficient information for simple snapshot (except for initial RNG state)
# It can be deserialized at a later point in time or by a different process
deserialized_graph = pickle.loads(serialized_graph)
# I think `DataLoader2` or `ReadingService` should store `initial_rng_state` that can be saved by the API that we later use
rng_for_deserialized = torch.Generator()
rng_for_deserialized.set_state(initial_rng_state)
n_iterations = deserialized_graph._number_of_samples_yielded

_simple_snapshot_graph(deserialized_graph, n_iterations, rng=rng_for_deserialized)
# The while DataPipe graph should have the same state as before serialization, such that:
self.assertEqual(list(it), list(deserialized_graph))  # True

Next Steps

If this looks acceptable, the next step is I will modify DataLoader2's prototype ReadingService (the one with queues) to remember things like initial_rng_state and to have methods save_snapshot that will return the (serialized graph, initial_rng) and restore_snapshot. This should work for single worker data loading.

Note that, in the long term, initial_rng_state may not be necessary if we are able to directly save/restore the buffer and RNG state of Shuffler (that is work in progress). However, initial_rng_state and simple snapshot is still a good fall-back option for some edge cases where the buffer can't be stored.

Differential Revision: D37943406

[ghstack-poisoned]

facebook-github-bot · 2022-06-13T23:57:11Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/79479
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓Need help or want to give feedback on the CI? Visit our office hours

✅ No Failures (0 Pending)

As of commit 7604dfe (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[ghstack-poisoned]

ejguan

This is the question I asked before. Do we need to make DataPipe itself supporting fast forward.
If we directly use DataLoader2 to save/restore the state, we can simply fast forward the iterator of DataPipe graph directly in DataLoader2.

NivekT · 2022-06-27T17:10:58Z

This is the question I asked before. Do we need to make DataPipe itself supporting fast forward. If we directly use DataLoader2 to save/restore the state, we can simply fast forward the iterator of DataPipe graph directly in DataLoader2.

I discuss part of the design here.

For the most basic implementation like the one here, DataLoader2 can directly call the fast forward function in this PR and and get the iterator to the desired state. However, this requires re-doing all the computations.
For more efficient implementations (in a future PR), DataLoader2 will still call a fast forward API, but individual DataPipes will need custom fast-forward methods that allow efficient restoration state.
- Notably, 1) stateless DataPipes do not need to do anything, 2) stateful ones may be able to restore from a serialized buffer, 3) source DataPipes (e.g. IterableWrapper) may still do a simple fast forward

ejguan · 2022-06-27T17:44:04Z

I hesitate to expose fast-forwarding as the API at DataPipe for two reasons:

It makes us to maintain two different APIs on DataPipe (one is for snapshotting, the another one is for fast forwarding). Besides, based on my understand, the last DataPipe of the graph is used to determine the number of iteration to be fast-forwarded. It's kind the same approach to use DataLoader2 to determine the number of iterations. I don't see the benefit to do so as we will remove fast forwarding ultimately.
It's the DataLoader2 to control the determinism aka creating RNG per process and save initial_random_state. Given your current design, it requires an RNG saved by DataLoader2. But, exposing fast-forwarding at DataPipe would allow users to do fast-fowrading directly.

NivekT · 2022-06-27T17:47:04Z

I hesitate to expose fast-forwarding as the API at DataPipe for two reasons:

That is fine with me. We can definitely make this private and only have DataLoader2 call it.

maintain two different APIs on DataPipe (one is for snapshotting, the another one is for fast forwarding)

They serve the same purpose. I can rename fast-forward as "poor man's snapshotting" if that is better.

I also agree that the API should be controlled by DataLoader2 (especially to get the randomness related states), unless there is an user demand for this API to be called outside of it.

This mostly completes the poor man's snapshotting implementation (named simple snapshot). This is the most basic version of snapshotting but it should work for all DataPipes. I will be adding more efficient implementation for different types of DataPipes in future PRs. ### Implementation The general idea of the simple snapshot is that we will: 1. Create a new iterator 2. Move that iterator forward by `n_iterations` 3. Save that as the `_fast_forward_iterator` pf the DataPipe 4. The next time `iter` is called on the DataPipe, use the `_fast_forward_iterator` ### Usage As of this implementation, the usage will something like: ```python rng = torch.Generator() initial_rng_state = rng.get_state() datapipe: IterDataPipe = ... # Some usage of the DataPipe, here maybe yielding the first 5 values n_iter = 5 it = iter(datapipe) for _ in range(n_iter): next(it) serialized_graph = pickle.dumps(datapipe) # The serialized object has most of the sufficient information for simple snapshot (except for initial RNG state) # It can be deserialized at a later point in time or by a different process deserialized_graph = pickle.loads(serialized_graph) # I think `DataLoader2` or `ReadingService` should store `initial_rng_state` that can be saved by the API that we later use rng_for_deserialized = torch.Generator() rng_for_deserialized.set_state(initial_rng_state) n_iterations = deserialized_graph._number_of_samples_yielded _simple_snapshot_graph(deserialized_graph, n_iterations, rng=rng_for_deserialized) # The while DataPipe graph should have the same state as before serialization, such that: self.assertEqual(list(it), list(deserialized_graph)) # True ``` ### Next Steps If this looks acceptable, the next step is I will modify `DataLoader2`'s prototype ReadingService (the one with queues) to remember things like `initial_rng_state` and to have methods `save_snapshot` that will return the `(serialized graph, initial_rng)` and `restore_snapshot`. This should work for single worker data loading. Note that, in the long term, `initial_rng_state` may not be necessary if we are able to directly save/restore the buffer and RNG state of `Shuffler` (that is work in progress). However, `initial_rng_state` and simple snapshot is still a good fall-back option for some edge cases where the buffer can't be stored. [ghstack-poisoned]

torch/utils/data/datapipes/_hook_iterator.py

torch/utils/data/datapipes/_typing.py

test/test_datapipe.py

This mostly completes the "poor man's snapshotting" implementation (named "simple snapshotting"). This is the most basic version of snapshotting but it should work for all DataPipes. I will be adding more efficient implementation for different types of DataPipes in future PRs. ### Implementation The general idea of the simple snapshot is that we will: 1. Create a new iterator 2. Move that iterator forward by `n_iterations` 3. Save that as the `_fast_forward_iterator` of the DataPipe 4. The next time `iter` is called on the DataPipe, use the `_fast_forward_iterator` ### Usage As of this implementation, the usage will something like: ```python rng = torch.Generator() initial_rng_state = rng.get_state() datapipe: IterDataPipe = ... # Some usage of the DataPipe, here maybe yielding the first 5 values n_iter = 5 it = iter(datapipe) for _ in range(n_iter): next(it) serialized_graph = pickle.dumps(datapipe) # The serialized object has most of the sufficient information for simple snapshot (except for initial RNG state) # It can be deserialized at a later point in time or by a different process deserialized_graph = pickle.loads(serialized_graph) # I think `DataLoader2` or `ReadingService` should store `initial_rng_state` that can be saved by the API that we later use rng_for_deserialized = torch.Generator() rng_for_deserialized.set_state(initial_rng_state) n_iterations = deserialized_graph._number_of_samples_yielded _simple_snapshot_graph(deserialized_graph, n_iterations, rng=rng_for_deserialized) # The while DataPipe graph should have the same state as before serialization, such that: self.assertEqual(list(it), list(deserialized_graph)) # True ``` ### Next Steps If this looks acceptable, the next step is I will modify `DataLoader2`'s prototype ReadingService (the one with queues) to remember things like `initial_rng_state` and to have methods `save_snapshot` that will return the `(serialized graph, initial_rng)` and `restore_snapshot`. This should work for single worker data loading. Note that, in the long term, `initial_rng_state` may not be necessary if we are able to directly save/restore the buffer and RNG state of `Shuffler` (that is work in progress). However, `initial_rng_state` and simple snapshot is still a good fall-back option for some edge cases where the buffer can't be stored. [ghstack-poisoned]

NivekT · 2022-07-18T23:12:14Z

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

This mostly completes the "poor man's snapshotting" implementation (named "simple snapshotting"). This is the most basic version of snapshotting but it should work for all DataPipes. I will be adding more efficient implementation for different types of DataPipes in future PRs. ### Implementation The general idea of the simple snapshot is that we will: 1. Create a new iterator 2. Move that iterator forward by `n_iterations` 3. Save that as the `_fast_forward_iterator` of the DataPipe 4. The next time `iter` is called on the DataPipe, use the `_fast_forward_iterator` ### Usage As of this implementation, the usage will something like: ```python rng = torch.Generator() initial_rng_state = rng.get_state() datapipe: IterDataPipe = ... # Some usage of the DataPipe, here maybe yielding the first 5 values n_iter = 5 it = iter(datapipe) for _ in range(n_iter): next(it) serialized_graph = pickle.dumps(datapipe) # The serialized object has most of the sufficient information for simple snapshot (except for initial RNG state) # It can be deserialized at a later point in time or by a different process deserialized_graph = pickle.loads(serialized_graph) # I think `DataLoader2` or `ReadingService` should store `initial_rng_state` that can be saved by the API that we later use rng_for_deserialized = torch.Generator() rng_for_deserialized.set_state(initial_rng_state) n_iterations = deserialized_graph._number_of_samples_yielded _simple_snapshot_graph(deserialized_graph, n_iterations, rng=rng_for_deserialized) # The while DataPipe graph should have the same state as before serialization, such that: self.assertEqual(list(it), list(deserialized_graph)) # True ``` ### Next Steps If this looks acceptable, the next step is I will modify `DataLoader2`'s prototype ReadingService (the one with queues) to remember things like `initial_rng_state` and to have methods `save_snapshot` that will return the `(serialized graph, initial_rng)` and `restore_snapshot`. This should work for single worker data loading. Note that, in the long term, `initial_rng_state` may not be necessary if we are able to directly save/restore the buffer and RNG state of `Shuffler` (that is work in progress). However, `initial_rng_state` and simple snapshot is still a good fall-back option for some edge cases where the buffer can't be stored. Differential Revision: [D37943406](https://our.internmc.facebook.com/intern/diff/D37943406) [ghstack-poisoned]

NivekT · 2022-07-19T21:22:44Z

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

NivekT

Updated this PR. I am expecting this to be final.

torch/utils/data/datapipes/_typing.py

test/test_datapipe.py

This mostly completes the "poor man's snapshotting" implementation (named "simple snapshotting"). This is the most basic version of snapshotting but it should work for all DataPipes. I will be adding more efficient implementation for different types of DataPipes in future PRs. ### Implementation The general idea of the simple snapshot is that we will: 1. Create a new iterator 2. Move that iterator forward by `n_iterations` 3. Save that as the `_fast_forward_iterator` of the DataPipe 4. The next time `iter` is called on the DataPipe, use the `_fast_forward_iterator` ### Usage As of this implementation, the usage will something like: ```python rng = torch.Generator() initial_rng_state = rng.get_state() datapipe: IterDataPipe = ... # Some usage of the DataPipe, here maybe yielding the first 5 values n_iter = 5 it = iter(datapipe) for _ in range(n_iter): next(it) serialized_graph = pickle.dumps(datapipe) # The serialized object has most of the sufficient information for simple snapshot (except for initial RNG state) # It can be deserialized at a later point in time or by a different process deserialized_graph = pickle.loads(serialized_graph) # I think `DataLoader2` or `ReadingService` should store `initial_rng_state` that can be saved by the API that we later use rng_for_deserialized = torch.Generator() rng_for_deserialized.set_state(initial_rng_state) n_iterations = deserialized_graph._number_of_samples_yielded _simple_snapshot_graph(deserialized_graph, n_iterations, rng=rng_for_deserialized) # The while DataPipe graph should have the same state as before serialization, such that: self.assertEqual(list(it), list(deserialized_graph)) # True ``` ### Next Steps If this looks acceptable, the next step is I will modify `DataLoader2`'s prototype ReadingService (the one with queues) to remember things like `initial_rng_state` and to have methods `save_snapshot` that will return the `(serialized graph, initial_rng)` and `restore_snapshot`. This should work for single worker data loading. Note that, in the long term, `initial_rng_state` may not be necessary if we are able to directly save/restore the buffer and RNG state of `Shuffler` (that is work in progress). However, `initial_rng_state` and simple snapshot is still a good fall-back option for some edge cases where the buffer can't be stored. Differential Revision: [D37943406](https://our.internmc.facebook.com/intern/diff/D37943406) [ghstack-poisoned]

NivekT · 2022-07-20T22:00:29Z

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ejguan

LGTM based on the test cases

torch/utils/data/datapipes/iter/combining.py

NivekT · 2022-07-22T20:09:44Z

@pytorchbot rebase

pytorchmergebot · 2022-07-22T20:11:11Z

@pytorchbot successfully started a rebase job. Check the current status here

This mostly completes the "poor man's snapshotting" implementation (named "simple snapshotting"). This is the most basic version of snapshotting but it should work for all DataPipes. I will be adding more efficient implementation for different types of DataPipes in future PRs. ### Implementation The general idea of the simple snapshot is that we will: 1. Create a new iterator 2. Move that iterator forward by `n_iterations` 3. Save that as the `_fast_forward_iterator` of the DataPipe 4. The next time `iter` is called on the DataPipe, use the `_fast_forward_iterator` ### Usage As of this implementation, the usage will something like: ```python rng = torch.Generator() initial_rng_state = rng.get_state() datapipe: IterDataPipe = ... # Some usage of the DataPipe, here maybe yielding the first 5 values n_iter = 5 it = iter(datapipe) for _ in range(n_iter): next(it) serialized_graph = pickle.dumps(datapipe) # The serialized object has most of the sufficient information for simple snapshot (except for initial RNG state) # It can be deserialized at a later point in time or by a different process deserialized_graph = pickle.loads(serialized_graph) # I think `DataLoader2` or `ReadingService` should store `initial_rng_state` that can be saved by the API that we later use rng_for_deserialized = torch.Generator() rng_for_deserialized.set_state(initial_rng_state) n_iterations = deserialized_graph._number_of_samples_yielded _simple_snapshot_graph(deserialized_graph, n_iterations, rng=rng_for_deserialized) # The while DataPipe graph should have the same state as before serialization, such that: self.assertEqual(list(it), list(deserialized_graph)) # True ``` ### Next Steps If this looks acceptable, the next step is I will modify `DataLoader2`'s prototype ReadingService (the one with queues) to remember things like `initial_rng_state` and to have methods `save_snapshot` that will return the `(serialized graph, initial_rng)` and `restore_snapshot`. This should work for single worker data loading. Note that, in the long term, `initial_rng_state` may not be necessary if we are able to directly save/restore the buffer and RNG state of `Shuffler` (that is work in progress). However, `initial_rng_state` and simple snapshot is still a good fall-back option for some edge cases where the buffer can't be stored. Differential Revision: [D37943406](https://our.internmc.facebook.com/intern/diff/D37943406) [ghstack-poisoned]

pytorchmergebot · 2022-07-22T20:11:24Z

Successfully rebased gh/nivekt/49/orig onto refs/remotes/origin/master, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/79479)

ghstack-source-id: 6d3120b Pull Request resolved: #79479 [DataPipe] Snapshotting with simple fast-forwarding ghstack-source-id: 6d3120b Pull Request resolved: #80250

NivekT · 2022-07-22T20:28:29Z

@pytorchbot merge

pytorchmergebot · 2022-07-22T20:31:09Z

@pytorchbot successfully started a merge job. Check the current status here

pytorchmergebot · 2022-07-22T23:46:54Z

Merge failed due to This PR has internal changes and must be landed via Phabricator
Raised by https://github.com/pytorch/pytorch/actions/runs/2721018437

NivekT · 2022-07-23T02:51:12Z

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

NivekT · 2022-07-23T02:51:58Z

@pytorchbot merge

pytorchmergebot · 2022-07-23T02:53:10Z

@pytorchbot successfully started a merge job. Check the current status here

Summary: This mostly completes the "poor man's snapshotting" implementation (named "simple snapshotting"). This is the most basic version of snapshotting but it should work for all DataPipes. I will be adding more efficient implementation for different types of DataPipes in future PRs. ### Implementation The general idea of the simple snapshot is that we will: 1. Create a new iterator 2. Move that iterator forward by `n_iterations` 3. Save that as the `_fast_forward_iterator` of the DataPipe 4. The next time `iter` is called on the DataPipe, use the `_fast_forward_iterator` ### Usage As of this implementation, the usage will something like: ```python rng = torch.Generator() initial_rng_state = rng.get_state() datapipe: IterDataPipe = ... # Some usage of the DataPipe, here maybe yielding the first 5 values n_iter = 5 it = iter(datapipe) for _ in range(n_iter): next(it) serialized_graph = pickle.dumps(datapipe) # The serialized object has most of the sufficient information for simple snapshot (except for initial RNG state) # It can be deserialized at a later point in time or by a different process deserialized_graph = pickle.loads(serialized_graph) # I think `DataLoader2` or `ReadingService` should store `initial_rng_state` that can be saved by the API that we later use rng_for_deserialized = torch.Generator() rng_for_deserialized.set_state(initial_rng_state) n_iterations = deserialized_graph._number_of_samples_yielded _simple_snapshot_graph(deserialized_graph, n_iterations, rng=rng_for_deserialized) # The while DataPipe graph should have the same state as before serialization, such that: self.assertEqual(list(it), list(deserialized_graph)) # True ``` ### Next Steps If this looks acceptable, the next step is I will modify `DataLoader2`'s prototype ReadingService (the one with queues) to remember things like `initial_rng_state` and to have methods `save_snapshot` that will return the `(serialized graph, initial_rng)` and `restore_snapshot`. This should work for single worker data loading. Note that, in the long term, `initial_rng_state` may not be necessary if we are able to directly save/restore the buffer and RNG state of `Shuffler` (that is work in progress). However, `initial_rng_state` and simple snapshot is still a good fall-back option for some edge cases where the buffer can't be stored. Pull Request resolved: #79479 Approved by: https://github.com/ejguan Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/35d97e21c8e2e28fae4a4744ce42f1544972ba1f Original Phabricator Test Plan: Imported from OSS Reviewed By: osalpekar, ejguan Differential Revision: D37943406 Pulled By: NivekT fbshipit-source-id: 7c50fd277a3d855a1b25d4c0f15face941d47a9d

[DataPipe] Fast-forwarding feature

3f1853c

[ghstack-poisoned]

NivekT mentioned this pull request Jun 13, 2022

[DataPipe] Basic snapshotting with IterableWrapper #70373

Closed

facebook-github-bot added the cla signed label Jun 13, 2022

NivekT mentioned this pull request Jun 13, 2022

[DataPipe] Snapshotting prototype #70216

Closed

NivekT marked this pull request as draft June 13, 2022 23:58

NivekT added module: data torch.utils.data release notes: dataloader release notes category topic: new features topic category labels Jun 13, 2022

Update on "[DataPipe] Fast-forwarding feature"

d3574a8

[ghstack-poisoned]

This was referenced Jun 15, 2022

[DataPipe] Refactor hook_iterator for readability #79656

Closed

[DataPipe] Count number of successful yields for IterDataPipe #79657

Closed

[DataPipe] Full graph fast-forwarding #79658

Closed

NivekT added 2 commits June 15, 2022 19:42

Update on "[DataPipe] Fast-forwarding feature"

a71cffe

[ghstack-poisoned]

Update on "[DataPipe] Fast-forwarding feature"

c28286c

[ghstack-poisoned]

NivekT changed the title ~~[DataPipe] Fast-forwarding feature~~ [DataPipe] Simple graph fast-forwarding Jun 16, 2022

Update on "[DataPipe] Simple graph fast-forwarding"

b8cf52c

[ghstack-poisoned]

NivekT mentioned this pull request Jun 16, 2022

[DataPipe] Enable profiler record context in __next__ branch #79757

Closed

NivekT added 3 commits June 17, 2022 18:36

Update on "[DataPipe] Simple graph fast-forwarding"

774bd01

[ghstack-poisoned]

Update on "[DataPipe] Simple graph fast-forwarding"

b35c0f7

[ghstack-poisoned]

Update on "[DataPipe] Simple graph fast-forwarding"

c6d78ef

[ghstack-poisoned]

NivekT mentioned this pull request Jun 24, 2022

[DataPipe] Snapshotting with simple fast-forwarding #80250

Closed

Update on "[DataPipe] Simple graph fast-forwarding"

f104efe

[ghstack-poisoned]

NivekT requested review from VitalyFedyunin and ejguan June 24, 2022 21:57

NivekT marked this pull request as ready for review June 24, 2022 21:57

ejguan reviewed Jun 27, 2022

View reviewed changes

ejguan reviewed Jul 18, 2022

View reviewed changes

NivekT commented Jul 19, 2022

View reviewed changes

torch/utils/data/datapipes/_typing.py Outdated Show resolved Hide resolved

VitalyFedyunin reviewed Jul 20, 2022

View reviewed changes

test/test_datapipe.py Show resolved Hide resolved

test/test_datapipe.py Show resolved Hide resolved

test/test_datapipe.py Show resolved Hide resolved

NivekT requested review from VitalyFedyunin and ejguan July 22, 2022 19:04

ejguan approved these changes Jul 22, 2022

View reviewed changes

torch/utils/data/datapipes/iter/combining.py Show resolved Hide resolved

pytorchmergebot added the Merged label Jul 23, 2022

pytorchmergebot closed this in 35d97e2 Jul 23, 2022

facebook-github-bot deleted the gh/nivekt/49/head branch July 26, 2022 14:20

ejguan mentioned this pull request Aug 17, 2022

[Discussion] Snapshot state for DataPipe meta-pytorch/data#741

Closed

Conversation

NivekT commented Jun 13, 2022 • edited by pytorchmergebot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation

Usage

Next Steps

Uh oh!

facebook-github-bot commented Jun 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

✅ No Failures (0 Pending)

Uh oh!

ejguan left a comment

Choose a reason for hiding this comment

Uh oh!

NivekT commented Jun 27, 2022

Uh oh!

ejguan commented Jun 27, 2022

Uh oh!

NivekT commented Jun 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NivekT commented Jul 18, 2022

Uh oh!

NivekT commented Jul 19, 2022

Uh oh!

NivekT left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NivekT commented Jul 20, 2022

Uh oh!

ejguan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NivekT commented Jul 22, 2022

Uh oh!

pytorchmergebot commented Jul 22, 2022

Uh oh!

pytorchmergebot commented Jul 22, 2022

Uh oh!

NivekT commented Jul 22, 2022

Uh oh!

pytorchmergebot commented Jul 22, 2022

Uh oh!

pytorchmergebot commented Jul 22, 2022

Uh oh!

NivekT commented Jul 23, 2022

Uh oh!

NivekT commented Jul 23, 2022

Uh oh!

pytorchmergebot commented Jul 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

NivekT commented Jun 13, 2022 •

edited by pytorchmergebot

Loading

facebook-github-bot commented Jun 13, 2022 •

edited

Loading

NivekT commented Jun 27, 2022 •

edited

Loading