🐛 Describe the bug
I am getting radically different results running the CURL model on the cpu versus the mps device (on pytorch 1.12.1 and nightly). I stepped through the debugger to find the underlying difference in calculation, In short, it appears that mps is using the "fast math" version of the standard library, leading to unacceptable results for some models. This is not a bug with CURL.
Here's a minimal example that doesn't work the way I expect:
import torch
mps_device = torch.device("mps")
x = 0.1
cpu_tensor = torch.exp(torch.tensor(x))
mps_tensor = torch.exp(torch.tensor(x, device=mps_device))
print(cpu_tensor - cpu_tensor) # prints 0
print(mps_tensor - mps_tensor) # prints 0
print(cpu_tensor - mps_tensor) # prints 1.1921e-07
print(cpu_tensor - mps_tensor.cpu()) # prints 1.1921e-07
print(cpu_tensor.to(mps_device) - mps_tensor) # prints 1.1921e-07Versions
Here's my pytorch 1.12.1 environment info:
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 13.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: version 3.23.2
Libc version: N/A
Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00) [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-13.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] adabelief-pytorch==0.2.1
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] torch==1.12.1
[pip3] torch-tb-profiler==0.4.0
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.13.1
[pip3] torchviz==0.0.2
[conda] adabelief-pytorch 0.2.1 pypi_0 pypi
[conda] numpy 1.23.3 pypi_0 pypi
[conda] torch 1.12.1 pypi_0 pypi
[conda] torch-tb-profiler 0.4.0 pypi_0 pypi
[conda] torchmetrics 0.9.3 pypi_0 pypi
[conda] torchvision 0.13.1 pypi_0 pypi
[conda] torchviz 0.0.2 pypi_0 pypi
And here's my pytorch nightly env:
PyTorch version: 1.13.0.dev20220913
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 13.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: version 3.23.2
Libc version: N/A
Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00) [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-13.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] adabelief-pytorch==0.2.1
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] torch==1.13.0.dev20220913
[pip3] torch-tb-profiler==0.4.0
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.13.1
[pip3] torchviz==0.0.2
[conda] adabelief-pytorch 0.2.1 pypi_0 pypi
[conda] numpy 1.23.3 pypi_0 pypi
[conda] torch 1.13.0.dev20220913 pypi_0 pypi
[conda] torch-tb-profiler 0.4.0 pypi_0 pypi
[conda] torchmetrics 0.9.3 pypi_0 pypi
[conda] torchvision 0.13.1 pypi_0 pypi
[conda] torchviz 0.0.2 pypi_0 pypi
🐛 Describe the bug
I am getting radically different results running the CURL model on the cpu versus the mps device (on pytorch 1.12.1 and nightly). I stepped through the debugger to find the underlying difference in calculation, In short, it appears that mps is using the "fast math" version of the standard library, leading to unacceptable results for some models. This is not a bug with CURL.
Here's a minimal example that doesn't work the way I expect:
Versions
Here's my pytorch 1.12.1 environment info:
And here's my pytorch nightly env: