CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 116
CUDA SETUP: Loading binary /opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg/bitsandbytes/libbitsandbytes_cuda116.so...
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 116
CUDA SETUP: Loading binary /opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg/bitsandbytes/libbitsandbytes_cuda116.so...
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=116 make cuda11x
python setup.py install
Traceback (most recent call last):
  File "train_dreambooth.py", line 811, in <module>
    main(args)
  File "train_dreambooth.py", line 528, in main
    import bitsandbytes as bnb
  File "/opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg/bitsandbytes/autograd/_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "/opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg/bitsandbytes/functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg/bitsandbytes/cextension.py", line 121, in <module>
    https://github.com/TimDettmers/bitsandbytes/issues''')
RuntimeError: 
        CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs aboveto fix your environment!
        If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
        https://github.com/TimDettmers/bitsandbytes/issues

git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=116 make cuda11x
python setup.py install

byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/__main__.py to __main__.cpython-37.pyc
  File "<fstring>", line 1
    (COMPILED_WITH_CUDA = )
                        ^
SyntaxError: invalid syntax

byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/optimizer.py to optimizer.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/lamb.py to lamb.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/adamw.py to adamw.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/adagrad.py to adagrad.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/lars.py to lars.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/sgd.py to sgd.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/adam.py to adam.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/rmsprop.py to rmsprop.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/utils.py to utils.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/nn/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/nn/modules.py to modules.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/autograd/_functions.py to _functions.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/autograd/__init__.py to __init__.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
bitsandbytes.__pycache__.cextension.cpython-37: module references __file__
creating 'dist/bitsandbytes-0.35.4-py3.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing bitsandbytes-0.35.4-py3.7.egg
removing '/opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg' (and everything under it)
creating /opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg
Extracting bitsandbytes-0.35.4-py3.7.egg to /opt/conda/envs/dreambooth/lib/python3.7/site-packages
  File "<fstring>", line 1
    (COMPILED_WITH_CUDA = )
                        ^
SyntaxError: invalid syntax

bitsandbytes 0.35.4 is already the active version in easy-install.pth
Installing debug_cuda script to /opt/conda/envs/dreambooth/bin

Installed /opt/conda/envs/dreambooth/lib/python3.7/site-packages/bitsandbytes-0.35.4-py3.7.egg
Processing dependencies for bitsandbytes==0.35.4
Finished processing dependencies for bitsandbytes==0.35.4