I'm not familiar with ggml-virtgpu, so I can't comment on how to cache cgraph.

One workaround for you is to disable CUDA graph by setting GGML_CUDA_DISABLE_GRAPHS=1 env variable.

I think it will be good to detect such cases in ggml-cuda and disable CUDA graphs, instead of OOM. If too many graph instances are created without ever getting used more than once, we can just disable CUDA graphs.
CC: @am17an @JohannesGaessler

Yes that's probably a good idea. It would also be nice that the memory would be reclaimed, currently we store the cgraphs in an unordered_map which doesn't delete anything, a LRU type fixed size vector would be better perhaps.

It would also be nice that the memory would be reclaimed

would be nice indeed, that was my first try, cleanup the cgraph, but that didn't help.

One workaround for you is to disable CUDA graph by setting GGML_CUDA_DISABLE_GRAPHS=1 env variable.

nice, I'll try that ASAP

Is there a structural reason why this backend returns a new cgraph everytime?

the ggml-virtgpu is actually split in two, one GGML backend running in the VM, one "API Remoting" backend running in the host.
both sides exchange the relevant data to make everything work, but in the current implementation, the cgraph object is reconstructed every time. Same logical content, different memory objects.

I tried to implement a caching mechanism, but nothing could work, and I lack knowledge to really understand what changes/what remains constant in the object between two cgraph_compute calls ...

I see, since CUDA graphs are being recreated after graph_compute, you should just disable them for now as it wouldn't provide a speedup. A possible solution would be to store the graphs via the first node's node->name, but I'm unsure how reliable that is, apart from the string comparison in the fast path

I have a PR #19754 open that should also address this issue.

just for information, the PR doesn't solve my issue. I'll try to investigate in the next weeks what's going on.

I hit this crash with b8163

/src/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:96: CUDA error
/app/llama.cpp/build/bin/libggml-base.so.0(+0x16608)[0x7f20c406f608]
/app/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1e6)[0x7f20c406f9d6]
/app/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x11d)[0x7f20c406fb5d]
/usr/bin/libggml-cuda.so(+0x188c03)[0x7f20ba40cc03]
/usr/bin/libggml-cuda.so(+0x19b662)[0x7f20ba41f662]
/usr/bin/libggml-cuda.so(+0x19c32d)[0x7f20ba42032d]
/app/llama.cpp/build/bin/libggml-virtgpu-backend.so(_Z29backend_backend_graph_computeP12apir_encoderP12apir_decoderP18virgl_apir_context+0x16e)[0x7f20c40fdcee]
/app/llama.cpp/build/bin/libggml-virtgpu-backend.so(apir_backend_dispatcher+0x68)[0x7f20c40fda08]
/usr/libexec/virgl_render_server[0x40549b]
/usr/libexec/virgl_render_server[0x406766]
/usr/libexec/virgl_render_server[0x403365]
/usr/libexec/virgl_render_server[0x4026b3]
/lib64/libc.so.6(+0x2a610)[0x7f20c5962610]
/lib64/libc.so.6(__libc_start_main+0x80)[0x7f20c59626c0]
/usr/libexec/virgl_render_server[0x4026e5]
srv          stop: cancel task, id_task = 918

but when I set GGML_CUDA_DISABLE_GRAPHS=1 it works

• prompt-processing benchmark

• token-generation benchmark

Can you paste the error log? It should print a more descriptive CUDA error and the place where the error occurred.

this (see above) is the error message that was captured, nothing more

/src/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:96: CUDA error
/app/llama.cpp/build/bin/libggml-base.so.0(+0x16608)[0x7f20c406f608]
/app/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1e6)[0x7f20c406f9d6]
/app/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x11d)[0x7f20c406fb5d]
/usr/bin/libggml-cuda.so(+0x188c03)[0x7f20ba40cc03]
/usr/bin/libggml-cuda.so(+0x19b662)[0x7f20ba41f662]
/usr/bin/libggml-cuda.so(+0x19c32d)[0x7f20ba42032d]
/app/llama.cpp/build/bin/libggml-virtgpu-backend.so(_Z29backend_backend_graph_computeP12apir_encoderP12apir_decoderP18virgl_apir_context+0x16e)[0x7f20c40fdcee]

that's the GGML_ABORT call, so it seems that the GGML_LOG_ERROR messages weren't logged :/

Maybe try using -v or -lv 2 command line option.