Potential race conditions between multiple workers trying to download and cache the same file in torch.hub.load_state_dict_from_url and torch.hub.download_url_to_file <- duplicate dataset/model downloads across DDP workers #68320
Description
Line 585 in 577a4d3
There should be a recommendation how to cache / download files and checkpoints correctly betwen multiple workers or at least warn of these problems
Same might happen if all of workers are using torchvision pretrained models? (which lead to this caching presumably)
cc @ezyang @gchanan @zou3519 @nairbv @NicolasHug @vmoens @jdsgomes