torchvision.datasets.ImageNet() race condition when multi-process training launched with root containing only *.tar *.tar.gz downloaded files #8707
Description
🐛 Describe the bug
Hi,
When I launched multi-process training (8x A100) using torchvision.datasets.ImageNet() with fresh-prepared root (i.e. containing only ILSVRC2012_devkit_t12.tar.gz, ILSVRC2012_img_train.tar, and ILSVRC2012_img_val.tar) I got this kind of errors:
$ NUMEXPR_MAX_THREADS=116 $PYTHON $MUPVIT_MAIN /data/ImageNet/ --workers $N_WORKERS --multiprocessing-distributed --batch-size 1024 --log-steps 100
Use GPU: 4 for training
Use GPU: 2 for training
Use GPU: 6 for training
Use GPU: 1 for training
Use GPU: 5 for training
Use GPU: 0 for training
Use GPU: 7 for training
Use GPU: 3 for training
[rank0]:[W1101 02:49:23.260690939 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any p
ending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been add
ed since PyTorch 2.4 (function operator())
W1101 02:49:24.409000 336226 torch/multiprocessing/spawn.py:160] Terminating process 336321 via signal SIGTERM
W1101 02:49:24.409000 336226 torch/multiprocessing/spawn.py:160] Terminating process 336322 via signal SIGTERM
W1101 02:49:24.411000 336226 torch/multiprocessing/spawn.py:160] Terminating process 336323 via signal SIGTERM
W1101 02:49:24.413000 336226 torch/multiprocessing/spawn.py:160] Terminating process 336324 via signal SIGTERM
W1101 02:49:24.417000 336226 torch/multiprocessing/spawn.py:160] Terminating process 336326 via signal SIGTERM
W1101 02:49:24.418000 336226 torch/multiprocessing/spawn.py:160] Terminating process 336327 via signal SIGTERM
W1101 02:49:24.422000 336226 torch/multiprocessing/spawn.py:160] Terminating process 336328 via signal SIGTERM
Traceback (most recent call last):
File "/home/ubuntu/Downloads/mup-vit/main.py", line 713, in <module>
main()
File "/home/ubuntu/Downloads/mup-vit/main.py", line 189, in main
mp.spawn(main_worker, nprocs=args.ngpus_per_node, args=(args, ))
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 328, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 284, in start_processes
while not context.join():
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 203, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 4 terminated with the following error:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
fn(i, *args)
File "/home/ubuntu/Downloads/mup-vit/main.py", line 337, in main_worker
train_dataset = datasets.ImageNet(args.data, split='train', transform=v2.Compose(transform))
File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/datasets/imagenet.py", line 53, in __init__
self.parse_archives()
File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/datasets/imagenet.py", line 70, in parse_archives
parse_train_archive(self.root)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/datasets/imagenet.py", line 183, in parse_train_archive
extract_archive(archive, os.path.splitext(archive)[0], remove_finished=True)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/datasets/utils.py", line 362, in extract_archive
suffix, archive_type, compression = _detect_file_type(from_path)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/datasets/utils.py", line 268, in _detect_file_type
raise RuntimeError(
RuntimeError: File '/data/ImageNet/train/n02104365' has no suffixes that could be used to detect the archive type and compression.The other errors are all about certain files exist already or don't exist yet. I am quite sure this is due to untar race condition and I worked around it by deleting all the intermediate files (meta.bin, train/ and val/ folders) and forcing a single process training launch to extract & place the files first.
It might be difficult to detect distributed training launch like this, but can we at least provide a warning in the documentation? Some tools to extract & place the files beforehand like python3 -m big_vision.tools.download_tfds_datasets imagenet2012 would also be helpful. This is kind of like the flip side of #2023.
Versions
The output below shows PyTorch version: 2.3.1 and torchvision==0.18.1, but I am actually running
$ pip freeze
(...)
torch==2.5.1
torchaudio==2.5.1
torchvision==0.20.1
(...)
I don't know why python3 -mpip list --format=freeze finds the old packages.
Collecting environment information...
PyTorch version: 2.3.1
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35
Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.8.0-47-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB
Nvidia driver version: 550.90.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 124
On-line CPU(s) list: 0-123
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7542 32-Core Processor
CPU family: 23
Model: 49
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 124
Stepping: 0
BogoMIPS: 5800.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
Virtualization: AMD-V
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 7.8 MiB (124 instances)
L1i cache: 7.8 MiB (124 instances)
L2 cache: 62 MiB (124 instances)
L3 cache: 1.9 GiB (124 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-123
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] flake8==4.0.1
[pip3] numpy==1.21.5
[pip3] optree==0.12.1
[pip3] torch==2.3.1
[pip3] torchvision==0.18.1
[pip3] triton==2.3.1
[conda] Could not collect