Update: I just merged some code to fix the following:

• --fit error (the issue was related to how ggml buffers were freed and we fixed it the same way we did with Metal, see these old notes)
• fit ignoring the mmproj model weights (so if a multimodal model is loaded, the context size is not reduced enough to fit both the model's and the projector's weights, resulting in an OOM + coredump)
• calculation of free memory in the table at exit resulted in an underflow

We should do some tests to try this on different platforms, at least:

• Linux with AMD GPU (steam deck)
• Linux with CUDA
• Windows
• containerized linux (@dpoulopoulos I know the answer already ;-), just wanted to sync the merge of this PR with some pointers to your work 🙏 )

@wingenlit if you have some time to try it, let us know how it works with your setup! Happy to iterate on this until we are confident enough this works reliably.

hello @aittalam thank for fixing.

I made a build here. so far things work. will report issue if seen. Could you add windows dll build or the recompile flag (in the past)? I got no luck with AI migrated scripts at linking the core libs.

hello @aittalam thank for fixing.

I made a build here. so far things work. will report issue if seen. Could you add windows dll build or the recompile flag (in the past)? I got no luck with AI migrated scripts at linking the core libs.

Wooohoo I love the sparkfile idea 💙
We are (still, sorry!) working on the win dll build - but it's great to know you'll be willing to test it afterwards 🙏😇
I think we'll first try to make sure we have a stable build outside of the APE, then reintroduce the rebuild option if ppl are happy with it (just out of curiosity, were you an active user of it?).

hello @aittalam thank for fixing.
I made a build here. so far things work. will report issue if seen. Could you add windows dll build or the recompile flag (in the past)? I got no luck with AI migrated scripts at linking the core libs.

Wooohoo I love the sparkfile idea 💙 We are (still, sorry!) working on the win dll build - but it's great to know you'll be willing to test it afterwards 🙏😇 I think we'll first try to make sure we have a stable build outside of the APE, then reintroduce the rebuild option if ppl are happy with it (just out of curiosity, were you an active user of it?).

Hi, I’ve been on llamafile since tinyblas came out. What I like is how llamafile hides many os/arch complexity, so it’s easy to share with less technical people to run gguf. tinyblas out-of-the-box on gpu gives a nice baseline acceleration. However, to take advantage of platform acceleration (like cublas) without writing a lengthy instruction of how-to, the recompile option is the one important trick. It’s been a good way to get near-full performance while keeping things lightweight.

• Worked properly on a linux pod, after rebuilding the library from scratch. Performance is not quite the same as with CUDA+cuBLAS drivers but looks quite good anyway.

GPU: L40
Model: Qwen3.5-9B-Q5_K_S
CUDA (with cuBLAS, necessary as Qwen3.5 models require cuBLAS TRSM which is not available with TinyBLAS):

Vulkan:

• Tested with the library built on the steam deck in a distrobox: the library can be ported on
  ubuntu:24.04 as it shares the same libs as the current version of SteamOS

Steam Deck setup:

# create an ubuntu container to build the lib
distrobox create --name devbuntu --image ubuntu:24.04
distrobox enter devbuntu

# install all the required deps
sudo apt update && sudo apt install git glslc build-essential libvulkan-dev glslang-tools

Code review

Found 4 issues:

1. Cross-module delete on host buffer struct that may be allocated by the main executable. ggml_backend_vk_host_buffer_type_alloc_buffer calls ggml_backend_cpu_buffer_from_ptr, which allocates the ggml_backend_buffer struct via the main executable's new operator. The patch then sets free_struct = ggml_backend_vk_buffer_free_struct, which calls delete buffer from the dylib's allocator. This is a heap mismatch -- the regular Vulkan buffer path is correct (dylib allocates and frees), but the host buffer path crosses module boundaries.

2. Missing log suppression before ggml_backend_register() in ImportVulkanImpl(). In cuda.c, there is an explicit pattern (lines 165-169) that suppresses GGML logging before backend registration to prevent noisy device enumeration output when --verbose is not set. The Vulkan implementation registers the backend without this suppression.

Compare with the CUDA pattern:

3. llamafile_vulkan_log_set() is defined and declared but never called in main.cpp or chatbot_main.cpp. Both files call llamafile_metal_log_set and llamafile_cuda_log_set to suppress verbose GPU logging when --verbose is off, but neither calls llamafile_vulkan_log_set. This same gap was flagged and fixed in PR Add cuda support #859 for CUDA.

4. llamafile_vulkan_log_set() calls llamafile_has_vulkan() first (line 208), which triggers the full initialization path including ggml_backend_register() before the log callback is set. This violates the contract documented in metal.c (line 663: "This must be set BEFORE llamafile_has_metal() is called"). The Metal implementation uses a pending callback pattern -- Vulkan should do the same.

Compare with the Metal pattern:

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

• Issue 1 (Cross-module memory management): The DSO compiles its own copy of ggml-backend.cpp (via compile_ggml_core in build-functions.sh), so ggml_backend_cpu_buffer_from_ptr resolves to the DSO's own copy and both allocation and deallocation happen within the same module

• Issue 2 (Missing log suppression): good catch! Added log suppression when --verbose is not set, matching the CUDA pattern

• Issue 3 (Unimplemented log callback): added llamafile_vulkan_log_set() calls in all three entry points (main.cpp, chatbot_main.cpp, and server.cpp)

• Issue 4 (Premature initialization): updated llamafile_vulkan_log_set() to use a pending callback mechanism matching Metal's pattern. It now stores the callback without triggering initialization

(see commit f3d1428)

speed test: model used Qwen3.5-9B-Q8_0.gguf

llama-benchy (0.3.5)

uv run llama-benchy --base-url local_url --model model --tokenizer Qwen/Qwen3.5-9B --concurrency 1

llama.cpp
commit 7f5ee549683d600ad41db6a295a232cdd2d8eb9f (HEAD, tag: b8198)

Results

GB10: (updated)
current llamafile vulkan

native llama.cpp vulkan

AMD MI50:
current llamafile vulkan

native llama.cpp vulkan

Reference llamafile CUDA GB10

speed test: model used Qwen3.5-9B-Q8_0.gguf

Uuuuh thanks for running this! So the take-home messages are:

• text generation on the DGX's GB10 is not much lower than native llama.cpp (good)
• on AMD MI50 the difference is much larger (bad for llamafile, but also why? Would it be better with ROCm as CUDA is better on the GB10?)
• in general, pp2048 is where llamafile's vulkan suffers the most - I'll need to check out what happens and how I can improve this...

This information is golden, thank you so much @wingenlit ! 🙏