library not found: failed to load library - Windows 11, GPU, CUDA 12.3 #2
Description
Hi, thank you for the great project!
I managed to run various models using CPU, but when using GPU I encounter an issue with "library not found". Not sure if I am doing something wrong, but after installing nvcc and added path to cl.exe, the libraries seem to have compiled, but then the error happens. Here is a longer log:
C:\work\llamafile>llamafile.exe -m starling-lm-7b-alpha.Q4_K_M.gguf
extracting /zip/llama.cpp/ggml.h to ./.llamafile/ggml.h
extracting /zip/llamafile/compcap.cu to ./.llamafile/compcap.cu
extracting /zip/llamafile/llamafile.h to ./.llamafile/llamafile.h
extracting /zip/llama.cpp/ggml-cuda.h to ./.llamafile/ggml-cuda.h
extracting /zip/llama.cpp/ggml-cuda.cu to ./.llamafile/ggml-cuda.cu
building ggml-cuda with nvcc -arch=native...
ggml-cuda.cu
cl : Command line warning D9002 : ignoring unknown option '-fPIC'
cl : Command line warning D9002 : ignoring unknown option '-O3'
cl : Command line warning D9002 : ignoring unknown option '-march=native'
cl : Command line warning D9002 : ignoring unknown option '-mtune=native'
ggml-cuda.cu
cl : Command line warning D9002 : ignoring unknown option '-fPIC'
cl : Command line warning D9002 : ignoring unknown option '-O3'
cl : Command line warning D9002 : ignoring unknown option '-march=native'
cl : Command line warning D9002 : ignoring unknown option '-mtune=native'
ggml-cuda.cu
cl : Command line warning D9002 : ignoring unknown option '-fPIC'
cl : Command line warning D9002 : ignoring unknown option '-O3'
cl : Command line warning D9002 : ignoring unknown option '-march=native'
cl : Command line warning D9002 : ignoring unknown option '-mtune=native'
tmpxft_00000970_00000000-11_ggml-cuda.cudafe1.cpp
cl : Command line warning D9002 : ignoring unknown option '-fPIC'
cl : Command line warning D9002 : ignoring unknown option '-O3'
cl : Command line warning D9002 : ignoring unknown option '-march=native'
cl : Command line warning D9002 : ignoring unknown option '-mtune=native'
cl : Command line warning D9002 : ignoring unknown option '-fPIC'
cl : Command line warning D9002 : ignoring unknown option '-O3'
cl : Command line warning D9002 : ignoring unknown option '-march=native'
cl : Command line warning D9002 : ignoring unknown option '-mtune=native'
Creating library .\.llamafile\ggml-cuda.dll.lib and object .\.llamafile\ggml-cuda.dll.exp
**library not found: failed to load library**
{"timestamp":1701310901,"level":"INFO","function":"main","line":2258,"message":"build info","build":1500,"commit":"a30b324"}
{"timestamp":1701310901,"level":"INFO","function":"main","line":2261,"message":"system info","n_threads":8,"n_threads_batch":-1,"total_threads":16,"system_info":"AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | "}
llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from starling-lm-7b-alpha.Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: - tensor 0: token_embd.weight q4_K [ 4096, 32002, 1, 1 ]
llama_model_loader: - tensor 1: blk.0.attn_norm.weight f32 ...
Any thoughts on what could be wrong?
Thank you!!