Conversation
Member
|
It's not very clear to me how to handle SWA with a unified cache where there may be multiple sequences, and it is not always obvious what tokens can be dropped from the cache. However I think it is definitely worth it for the single user case, which after all is the main use case of llama.cpp. |
Collaborator
Yes this is what I was thinking about for months now. There is no better solution than to disable context caching in this case. An alternative solution is to allow user to choose one of the 2: either a proper SWA cache (good for memory) or allocate full (good for reusing cache)
I'm feeling 50/50 here. One of the biggest use case would be to process large and diverse set of documents locally. In this case, user may never reuse the cache because each new request is a new document |
Member
Author
The way I am approaching it is to have the "KV cells" information maintained separately for the non-SWA and SWA layers. This way, upon each KV cache commit (see #12799), we can do a pass over the SWA cells and automatically remove those that have position The rest of the logic is the same - it just operates on both set of cells. For example, |
Collaborator
|
My experience with the Gemma models in the context of Elo HeLLM has been that they required a disproportionate amount of computational resources to run benchmarks. The reason is that I was able to fit comparatively fewer parallel slots on 1 or 2 GPUs and my throughput was lower as a consequence. At least for my use case I value low memory usage for the context more than I value prompt caching because I have O(10000) short prompts and I'm bottlenecked mostly by generation throughput. |
Member
Author
|
Continuing thinking about the logic for when to discard tokens from the cache, it's indeed tricky and not very clear how to do. For example, when doing speculative decoding, we can submit a draft batch with |
e37f112 to
7e4b545
Compare
Contributor
I second slaren's opinion. As far as I know, vllm also doesn't support iSWA while hf transformers and ollama does. vllm is geared toward multi-user server use case. I suppose that's why they don't support it. Ideally, it should be implemented as a switch to let user choose which one to use. By default, iSWA should be on for llama-cli but off for llama-server. |
Collaborator
Yes I was thinking about this too, I think it can be a bit complicated to manage this case, but totally possible. We can let user specify how many tokens are allocated in the sliding layers. For example, given We can further let And finally, we may need to add an API to return the furthest |
Contributor
|
I'd +1 the ability to allow the user to switch. Some use-cases benefit greatly from the prefix caching (example: on Metal systems with 48GB of RAM/VRAM, where pp is much slower than non-Metal pp and we have plenty of VRAM anyway) so allowing the user to choose would be optimal. |
Contributor
Is llama.cpp single user mode the most used case because that’s what the user base prefer or is it like that because the server performance goes down a lot with more than 3 users ? (#10860 ) We are really thankful of all the work you main contributors do on this project, but please do not fall in this « self-fulfilling prophecy » trap. |
8 tasks
Contributor
|
I personally use llama.cpp for server use (with multiple users). |
58115a2 to
7e79a42
Compare
|
According to the Gemma3 paper, interleaved Sliding Window Attention reduces KV Cache memory usage by 1/5, so it would be much easier to run as right now KV Cache size is much heavier than comparable models. If the drawback is the absence of prompt caching, then indeed it would make sense to give the option to the user and let them decide on a per use case basis. I think for cases where you use RAG/Vector DB it would prove to be very useful as prompt caching does not work when beginning of the context changes anyway. I would personally agree with Johannes here, faster token generation thanks to SWA would be more useful for me as well since I'm using vector DB. So for the use cases short prompts/RAG it would make a lot of sense. For simple chat use cases without any RAG, prompt caching would probably make it faster overall compared to SWA and no prompt cache. Overall, I think having the option would be a great addition to llama.cpp. If it helps, Ollama implemented iSWA support for Gemma 3, since the project is pretty similar to llama.cpp, perhaps it's useful to get a rough idea on how to implement it (although Ollama is a different coding language): https://github.com/ollama/ollama/blob/2fec73eef6e9482f606f185ebb2ae4f75ad1a37c/model/models/gemma3/model_text.go#L190 I've been thinking, does Ollama support prompt caching? Since Gemma 3 SWA is supported in Ollama, how did they handle it? |
1c69466 to
1e10743
Compare
Collaborator
|
Some people recently mentioned concerns with this PR - I think caching is quite important for a subset of users who don't have GPUs and run purely CPU only. They are fine spending initial minutes or more ingesting a large initial prompts which they then reuse for many future turns - generation speed itself is usable, but the inability to cache would be crippling for such users. |
Member
Author
|
Both the old cache (i.e. more memory usage, but with advanced caching supported) and the new cache (less memory with just last-prefix caching) will be supported. Still figuring the implementation details - will likely be supported via a flag or a parameter. |
3 tasks
Member
Author
|
Thanks for all the feedback in this discussion. This branch should be ready for testing - I've listed some important use cases that need to be exercised. If something does not work, please let me know - at the moment I've done very little testing, so there could be some issues remaining. I will soon write up a detailed summary of the changes and the approach taken. And after that will add some comments to the code and open the PR for review. Regarding the parameter for controlling the size of the SWA cache - for now I haven't introduced it because some initial tests show that Gemma 3 remains coherent even when it "forgets" the local SWA cache - likely thanks to the data in the non-SWA cache. So I am thinking about giving this approach a try because it keeps the UX simple (i.e. we won't have to add new parameter and handle the use cases where context editing is not possible). If we determine that this breaks some important use cases, we can add the parameter - the |
Contributor
|
To people who have the bandwidth to test models, FYI Cohere 2 arch includes R7B which is much smaller than Command-A |
Does this mean in the current implementation the model isn't executed correctly? |
|
FWIW, Gemma 3 worked better for me on main with Q8 cache quantization than on this branch + unquantized kv cache. |
Member
Author
|
@andportnoy It's evaluated correctly, as long as you don't use context shift, cache reuse or branching from old states. Do you do any of that in your tests? Can you provide a repro? Edit: Also don't change 2 things at the same time when testing. Use the same KV cache type, so we can rule out differences that are not relevant to the changes in this branch. |
|
can t get an efficient k type for medgemma-4b-it-Q4_K_M, gemma 3 q4_k_m works fine with type 1 |
Contributor
6 tasks
Using Qwen 3 Coder Next: Any quant I use, I am getting this error: forcing full prompt re-processing due to lack of cache data as shown below, reloading 50,60,70,150k context over and over is quite miserable.. all latest build and fresh quant dl just in case as of today 2/8/26. Any guidance or insight would be appreciated. It has a link to come to this thread, but I have not found specific guidance.
|
Contributor
im getting this error as well. Did you get a fix? |
|
Running Unsloth's Qwen3.5-397B-A17B Q6_K_XL GGUF on latest llama.cpp (with pwilkin/autoparser tool calling fixes) and it throws errors directing me to this issue. I'm running Claude cli via LiteLLM into llama.cpp. This is the error: Full log: |
|
I built llama.cpp about 2 days ago and I still see this issue with both Qwen3-Coder-Next-Q6_K-00001-of-00004.gguf, and Qwen3-Coder-Next-Q8_0-00001-of-00003.gguf. It works for a few checkpoints then invalidates and has to process everything, then works for a few more checkpoints and repeats. Could any of these switches be causing an issue? |
|
Edit: After reading through this PR --swa-full worked! The log message should be updated to mention this option and how it will use more memory and any other effects. This fixed 1 of my 2 issues with the cache misses and my second issue was open webui I had set RAG_SYSTEM_CONTEXT=True causing the system prompt to change with web_search (not fetch_url). I think this being set True for RAG is ideal for cache hits, but for web_search (which isn't RAG) this needs to be False for cache hits. When I have both RAG and web_search I am not sure how I will solve it, but that's not a llama.cpp issue. I'm also hitting this with llama-server and gpt-oss-20b: It's making web search calls and causing the whole prefill to start over from scratch all the time. The agentic behaviour is slowed to a crawl due to all the full prompt reprocessing occurring. Is there a workaround or fix in another PR for this being worked on? Is there an issue with my settings? I have been playing around with --swa-checkpoints being 0 and a high value and neither helped. |
apmantza
pushed a commit
to apmantza/helots
that referenced
this pull request
Mar 6, 2026
…vailabl: id ... whats happening now. getting this on server side: slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.887 slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> temp-ext -> dist slot launch_slot_: id 0 | task 74 | processing task, is_child = 0 slot update_slots: id 0 | task 74 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 557 slot update_slots: id 0 | task 74 | n_past = 557, slot.prompt.tokens.size() = 628, seq_id = 0, pos_min = 627, n_swa = 1 slot update_slots: id 0 | task 74 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see ggml-org/llama.cpp#13194 (comment)) slot update_slots: id 0 | task 74 | n_tokens = 0, memory_seq_rm [0, end) slot update_slots: id 0 | task 74 | prompt processing progress, n_tokens = 45, batch.n_tokens = 45, progress = 0.080790 slot update_slots: id 0 | task 74 | n_tokens = 45, memory_seq_rm [45, end) slot init_sampler: id 0 | task 74 | init sampler, took 0.05 ms, tokens: text = 557, total = 557 slot update_slots: id 0 | task 74 | prompt processing done, n_tokens = 557, batch.n_tokens = 512 srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200 slot print_timing: id 0 | task 74 | prompt eval time = 314.07 ms / 557 tokens ( 0.56 ms per token, 1773.47 tokens per second) eval time = 857.92 ms / 93 tokens ( 9.22 ms per token, 108.40 tokens per second) total time = 1171.99 ms / 650 tokens slot release: id 0 | task 74 | stop processing: n_tokens = 649, truncated = 0 srv update_slots: all slots are idle
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Add
class llama_kv_cache_unified_iswafor interleaved SWA attention support.The implementation internally utilizes 2 instances of the existing
llama_kv_cache_unified- one for the non-SWA and one for the SWA layers of the model. To achieve that, thellama_kv_cache_unifiedimplementation is updated to be able to cache a subset of the model's layers (instead of always caching all layers as it is onmaster). The 2 internal caches behave almost in exactly the same way with 2 main differences:The size of the SWA cache is computed as:
This way we can store the cache data for the last
n_swatokens for all sequences and we also have room to evaluate a new batch of tokens with size up ton_batch.Note that advanced cache operations such as removing tokens or shifting their positions are not possible when using SWA cache, because token information becomes lost when the window slides. For such cases, we can "fallback" to the old implementation by expanding the SWA cache size to the full context and disabling the SWA token pruning. This of course would lead to more memory usage. See the
swa_fullflag for more info.The new
llama_kv_cache_unified_iswacan be used for non-SWA models withn_swa = n_ctx_train.Main changes
llama-graphtollama-kv-cachellama-graphtollama-kv-cachebuild_attn_mha()are now not permutedenum hparams.swa_typeto support chunked and non-chunked SWA (removehparams.n_attn_chunk)class llama_kv_cache_unified_iswa- new iSWA cache that internally utilizes 2 standardllama_kv_cache_unifiedinstancesllama_kv_cache_unifiedimplementation more private and polish the interfacellm_build_llama_iswa()llama-servernow respectsllama_kv_self_can_shift(ctx)llama_decodenow attempts to do a defrag if it fails to fit the input batch in the cachellama_decodenow correctly restores the cache state in all cases--swa-fullAPI changes
llama_context_params- addbool swa_fullTODO
llama_kv_cache_unified_iswa::commit()n_seq_maxandn_batchto the KV cache and utilize it to determine SWA cache sizellama-servercheck forllama_kv_self_can_shiftTesting
Any help with testing the following scenarios and reporting the results are highly appreciated:
Next PRs
llama_kv_cache_viewAPI (not useful, can be replaced with internal debugging functions)struct kv_cellsand simplify logic with modifying the cellsllama_kv_cachelogic to allow SWA cache with sizen_swa + n_ubatchllama_decodedistinguish return code when we are sure that even after defrag there is no space availablellama_context_paramsllama_kv_cache::set_full()llama_kv_cacheto not maintain the batching state (kv-cache : add SWA support #13194 (review))template <bool SWA> llm_build_llama()outdated
This is still very WIP - the goal is to redesign the unified KV cache to properly support layers with sliding-window attention (SWA) in order to reduce the memory usage for models such as Gemma3.
However, while working on this, I realized that enabling this option would prevent context caching, which IMO is a pretty big deal. So I am wondering if I am missing something.
The reason we cannot do context caching with SWA enabled is because when the window slides, we "forget" the old KV stuff and there is no way to recover it without recomputing it. This means, no prefix cache in
llama-server(ok, just last-prefix caching works), no context shift, no context reuse, etc. So I am having some doubts if this is really worth supporting.Any thoughts?