Fix tests for Azure and DataLoader#905
Conversation
|
@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
| timeout: float | ||
| worker_init_fn: Optional[Callable[[int], None]] | ||
| prefetch_factor: int | ||
| prefetch_factor: Optional[int] |
There was a problem hiding this comment.
noob question: why it's called "factor" ?
There was a problem hiding this comment.
Argument from DataLoader1
https://github.com/pytorch/pytorch/blob/837ca8f344380f2356b01662f215ff561b09401f/torch/utils/data/dataloader.py#L176
prefetch_factor * num_workers
There was a problem hiding this comment.
It doesn't have to be done in this PR; maybe add docsting for arguments in MultiProcessingReadingServicer? Thanks.
There was a problem hiding this comment.
2means there will be a total of 2 * num_workers batches prefetched across all workers.
noob question: it sounds like different data loader workers may have different numbers of prefetched batches? -- because otherwise, the doc should just say "2 means each worker will prefetech 2 batches"
There was a problem hiding this comment.
For DataLoader1, each worker always prefetches prefetch_factor batches.
Changes
prefetch_factoris specified whennum_workers>0.