Testing DeepSeek V3.2 Lightning Indexer and Sparse Attention implementation #21183
Replies: 3 comments 52 replies
-
|
Thank you for the details and the reproduction instructions. I'll follow up in the next few days with my findings and results. |
Beta Was this translation helpful? Give feedback.
-
|
@AesSedai I rented 8x RTX PRO 6000S instance and did some experiments to see what llama-server parameters would be best. Since we don't care much about prompt processing (prompts are short in lineage-bench, from few hundred tokens for lineage-8 up to a few thousands tokens for lineage-1024) I think the best approach is to lower ubatch size so that the model compute buffer is small (but prompt processing rate will suck) and this way we can fit larger context for more concurrent sequences. I included rough estimates of how much time lineage-512 run would take with -n 25 (100 quizzes). These numbers look very bad. I guess Q8_0 is out of the question and we should go either with Q4_K_M with 32 parallel requests in case we want to do 100 quizzes or with 8 parallel requests in case we want to do only 40 quizzes (that's the absolute minimum I'd like to do, I think it would take ~13 hours). Let me know how much compute time you are willing to spend on this. Don't worry about power consumption, from what I saw in Q8_0NP 1Command: Used memory: Example performance: Expected lineage-512 run time: ~107 hours NP 4:Command: Used memory: Example performance: Expected lineage-512 run time: ~56 hours Q4_K_MNP 4Command: Used memory: Example performance: Expected lineage-512 run time: 43 hours NP 8Command: Used memory: Example performance: Expected lineage-512 run time: 32 hours NP 16Command: Used memory: Example performance: Expected lineage-512 run time: 29 hours NP 32Command: Used memory: Example performance: Expected lineage-512 run time: 21 hours |
Beta Was this translation helpful? Give feedback.
-
|
@AesSedai I tested 40 prompts of lineage-512 with DeepSeek V3.2 Speciale Q8_0 (4 parallel streams, took 20 hours) and got accuracy 0.85. So I suppose it's Q4_K_M quantization that lowered your score. Implementation is optimized now so generation doesn't slow down much with long context size. |
Beta Was this translation helpful? Give feedback.
-
|
@fairydreaming 👀 sorry I don't check notifs often, interesting that the Q4_K_M was the culprit. I saw that you posted a couple of things about speeding up the inference so that is good to see, I could try a Q8_0 run sometime in the near future if that's what you're thinking again. The 100 prompt run was a little much but I'd do a 50 prompt run (or similar) if you'd like. |
Beta Was this translation helpful? Give feedback.
-
@AesSedai It would be great to have an independent evaluation of the implementation but there's no hurry, I'd like to finish it first so that the implementation is tested in its final form. |
Beta Was this translation helpful? Give feedback.
-
|
@fairydreaming okay, send me a ping when you want me to re-test it. Will you have a Q8_0 uploaded to HF to test with? |
Beta Was this translation helpful? Give feedback.
-
@AesSedai Q8_0 was there right from the beginning. Will let you know when the implementation is ready for testing, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@fairydreaming I have some spare time this weekend, so I dusted off my LLM rig and ran your branch at commit This is with the following startup command: Details./build/bin/llama-server \
--model /data2/DeepSeek-V3.2-light-GGUF/Q4_K_M/DeepSeek-V3.2-light-Q4_K_M-00001-of-00031.gguf \
--alias DeepSeek-V3.2-light-GGUF:671b-q4_k_m \
--numa numactl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot "blk\.(3|4|5|6|7|8|9|10)\.ffn_.*=CUDA0" \
-ot exps=CPU \
-ub 4096 -b 4096 \
--seed 3407 \
--temp 0.6 \
--top-p 0.95 \
--min-p 0.1 \
--log-colors on \
--flash-attn on \
--host 0.0.0.0 \
--prio 2 \
--jinja \
--verbose \
--port 11434 \
--chat-template-file /data2/DeepSeek-V3.2/chat_template.jinjaI have 768gb of system ram and a 96gb blackwell 6000 pro. |
Beta Was this translation helpful? Give feedback.
-
|
@fairydreaming yes, I always run with 
|
Beta Was this translation helpful? Give feedback.
-
|
@fairydreaming I had to restart it 20 or 30 times to slog through the tests. It failed on the last test five or six times, so I eventually gave up. I even tried lowering ubatch to 512 on the last one, but it didn't help. The results are good, other than the reliability issues: I'm going to reboot the machine so it'll run a self check, just to make sure I didn't fry an RDIMM. |
Beta Was this translation helpful? Give feedback.
-
|
@createthis was it always with the same stacktrace (alloc in top_k)? |
Beta Was this translation helpful? Give feedback.
-
|
I take it back... it didn't crash when I switched to ubatch 512. I got this message in Aider, which is the same message I got when it would crash with ubatch 4096: But looking at the server logs, it never stack traced, it just kept going. This is the server log when it times out: So maybe it's just hitting the Aider timeout. I'll try increasing it. |
Beta Was this translation helpful? Give feedback.
-
|
@fairydreaming Increasing the timeout to 3200 and lowering the ublock to 512 worked. Final score: This is very similar to the run Run command: ./build/bin/llama-server \
--model /data2/DeepSeek-V3.2-light-GGUF/Q4_K_M/DeepSeek-V3.2-light-Q4_K_M-00001-of-00031.gguf \
--alias DeepSeek-V3.2-light-GGUF:671b-q4_k_m \
--numa numactl \
--threads 32 \
--ctx-size 163840 \
--n-gpu-layers 62 \
-ot "blk\.(3|4|5|6|7|8|9|10)\.ffn_.*=CUDA0" \
-ot exps=CPU \
-ub 512 -b 4096 \
--seed 3407 \
--temp 0.6 \
--top-p 0.95 \
--min-p 0.1 \
--log-colors on \
--flash-attn on \
--host 0.0.0.0 \
--prio 2 \
--jinja \
--port 11434 \
--chat-template-file ./models/templates/deepseek-ai-DeepSeek-V3.2.jinja \
--chat-template-kwargs '{"thinking": true}'.aider.model.settings.yml - name: openai/DeepSeek-V3.2-light-GGUF-q4_k_m
use_temperature: 0.6
extra_params:
top_p: 0.95
temperature: 0.6
min_p: 0.1
top_k: 0.0
timeout: 3200 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Recently I submitted PR #21149 that adds support for DeepseekV32ForCausalLM with lightning indexer and DeepSeek Sparse Attention (DSA) implementation. I'd like to have it properly tested to confirm that the implementation works correctly and can serve as a reference for further optimizations.
Some background: in DeepSeek V3.2 (and related) DSA attention works exactly like MLA up to 2048 tokens, then they may start slightly diverge. So benchmarks with short/moderate length questions and answers are not good for this. What we need are very long reasoning chains where a slight mistake is fatal for the final answer or maybe some other long-context tests.
My initial idea is to use lineage-bench with very large number of lineage graph nodes (at least 512). I already used lineage-bench to test DeepSeek V3.2 Speciale in sglang with both dense (MLA) and sparse (DSA) attention up to lineage-1024, results are here:
If we can reproduce this result (whole or limited subset) with llama.cpp this is convincing enough for me. But running this requires a lot of VRAM (at least 8x RTX PRO 6000 for Q8_0) and may take a long time. So if anyone has a better idea let me know.
During my work on the implementation I did some lineage-128 tests with DeepSeek V3.2 Exp model (20 quizzes due to limited compute). I found that the model underperformed (accuracy around 70%) compared to OpenRouter benchmark run (accuracy 90%). This allowed to me to find a bug in Hadamard transform implementation, after fixing it accuracy in the limited local benchmark run went up to 95%. This is may be anecdotal evidence, but it's a data point confirming that lineage-bench is well suited for detecting subtle flaws in the implementation.
Detailed steps for reproducing the benchmark results:
Environment
I confirmed that the implementation works without problems with CUDA 13.2 and CCCL 13.2.27. Earlier versions of CUDA currently have problems with ggml_top_k() implementation.I rebased PR, so fix for
ggml_top_k()is already present. CCCL is no longer needed, CUDA should work even for 12.X versions.Downloading the model
Building llama.cpp
Starting llama-server
Exact context size and the number of parallel sequences to be set later.
Installing lineage-bench
Running lineage-bench
Full benchmark run:
Limited lineage-512 benchmark run:
cc @AesSedai who volunteered his hardware for the benchmark runs.
Beta Was this translation helpful? Give feedback.
All reactions