@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Just to be clear, does the first version assumes 1 process per rank?

Just to be clear, does the first version assumes 1 process per rank?

Yeah. I think our next step is to support mixed reading service (distributed reading service + multiprocessing reading service)

1. Are we expecting users to always use something like torch.multiprocessing.spawn to start distributed training? And that will properly start/clean up all the processes?

Nope. I can add a different test for elastic training.

This PR should be ready to review. cc: @NivekT @VitalyFedyunin
In terms of multiprocessing + distributed, I will figure out the design of mixed reading services first then add such feature.

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Actually, there is one thing left, which is doc and tutorial for DistributedReadingService. I will do a separate PR to document DataLoader2 and DistributedReadingService.

I will wait until pytorch/pytorch#83741 is landed and released into nightly because it will also use the updated API.

Failing DataPipe tests because the nightly binaries for mac haven't been updated yet.

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

IIUC, the difference between this and _test_distributed_rs is DL1 vs DL2?

• If so, we can potential remove the duplicate code (not urgent).
• Alternatively, we can just label _test_distributed_dl and _test_distributed_dataloader (and elastic_dl/elastic_training) so it will be obvious from the first glance that those two are mostly the same but testing different version of DL.

Fixed

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

I will land this PR until PyTorch nightly is updated.

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.