slightly updated the test to have larger prefix so that we can trigger the code path to split at SPLIT_SIZE=1024

Testing Trial 12 with batch size 12, # prefix length 1: 3, # prefix length 4097: 2, # prefix length 5120: 2, # prefix length 8191: 5,
Prompt 0 with prefix length 1: total samples: 22, Unique samples: 1
Prompt 1 with prefix length 4097: total samples: 11, Unique samples: 1
Prompt 2 with prefix length 5120: total samples: 19, Unique samples: 3
Prompt 3 with prefix length 8191: total samples: 26, Unique samples: 3

Now we're getting a few non-determinism but still better than without the SPLIT_SIZE code path at all.

well, this non-determinism doesn't come from the radix cache: with --disable-radix-cache I can still get:

Testing Trial 12 with batch size 12, # prefix length 1: 4, # prefix length 4097: 2, # prefix length 5120: 2, # prefix length 8191: 4,
Prompt 0 with prefix length 1: total samples: 21, Unique samples: 1
Prompt 1 with prefix length 4097: total samples: 20, Unique samples: 1
Prompt 2 with prefix length 5120: total samples: 15, Unique samples: 3
Prompt 3 with prefix length 8191: total samples: 22, Unique samples: 2

with my slightly modified prefix lengths. guess we have some issues with other parts of the code.

(orSGLANG_FLASHINFER_PREFILL_SPLIT_TILE_SIZE=1024 is not practical to test this)

This patch also includes #10637; I can close that one and keep this

Also please post the result of

python3 -m sglang.test.test_deterministic --test-mode single
python3 -m sglang.test.test_deterministic --test-mode mixed
python3 -m sglang.test.test_deterministic --test-mode prefix

on both flashinfer and triton backends

When do the testing, please make sure the align size is the same as prefill_split_size/prefill_truncation_size

@hnyls2002 @xiezhq-hermann
This code changes some of the logics in radix cache, can you please help checking together.
Especially regarding potential memory leaks

python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B --attention-backend flashinfer --enable-deterministic-inference

Flashinfer:

single - Total samples: 50, Unique samples: 1 ; the logic added in this patch is not triggered b/c all sequences are < split size, and therefore the cache always return un-match on prefill as part of the fast path (I assume this is expected behavior?)

mixed

Prompt 1: total samples: 583, Unique samples: 1
Prompt 2: total samples: 468, Unique samples: 1
Long prompt: total samples: 224, Unique samples: 1

same, I don't see the logic of reconstruction is triggered, probably due to the prompt length is too small

prefix

Prompt 0 with prefix length 1: total samples: 321, Unique samples: 1
Prompt 1 with prefix length 511: total samples: 332, Unique samples: 1
Prompt 2 with prefix length 2048: total samples: 311, Unique samples: 1
Prompt 3 with prefix length 4097: total samples: 311, Unique samples: 1

"prefix length 4097" does not yield a call to radix cache with >= 4096 (split size) requests, probably because of tokenization?

triton:

single - Total samples: 50, Unique samples: 1

Maybe it makes sense for me to increase the prompt length so that we can actually test the behavior of radix cache?

I'm resolving the comments meanwhile, thanks for reviews :)

okay I need to do a benchmark again 🤣 I lost the changes to remove disable radix cache logic after rebasing

^ still not hitting the radix cache due to prompt length too small, should we update the tests?

^ still not hitting the radix cache due to prompt length too small, should we update the tests?

Yes, we should try some longer prefices to test radix cache logic

with e207d8c I can confirm the cache gets a lot of hits,

trition:

Testing Trial 50 with batch size 50, # prefix length 1: 13, # prefix length 2048: 11, # prefix length 10000: 10, # prefix length 20000: 16,
Prompt 0 with prefix length 1: total samples: 305, Unique samples: 1
Prompt 1 with prefix length 2048: total samples: 334, Unique samples: 1
Prompt 2 with prefix length 10000: total samples: 302, Unique samples: 1
Prompt 3 with prefix length 20000: total samples: 334, Unique samples: 1

flashinfer:

Testing Trial 50 with batch size 50, # prefix length 1: 13, # prefix length 2048: 10, # prefix length 10000: 13, # prefix length 20000: 14,
Prompt 0 with prefix length 1: total samples: 296, Unique samples: 1
Prompt 1 with prefix length 2048: total samples: 312, Unique samples: 1
Prompt 2 with prefix length 10000: total samples: 350, Unique samples: 1
Prompt 3 with prefix length 20000: total samples: 317, Unique samples: 1

@skyzh Can you also provide some data for performance? Like how much it degrades from normal mode?

Thank you @skyzh for the PR, it is indeed an interesting and important feature. But I am not sure whether enforcing a split is the best way of achieving this. Does it really guarantee determinism or more like reducing variances, especially if the split length is small. Would a large page size somehow achieving similar results?

Hi @xiezhq-hermann,

• when split size is small: yes, I tried split size like 128 and it can reduce non-determinism compared with not considering the split size at all. However, there're other parts of the system that is not determinism with the split size is very small.
• large page size: yes. The initial implementation of this patch simply sets page_size to split_size. But it leads to some memory mismatch (leak) errors. Before we hit those errors it gives good results. So I think it's better to consider both page_size and split_size in this patch.

(still gathering perf data, but my impression was that there're no significant speed different before/after enable if split_size is small; in other words, as we always align the result with the split size, when it's large, we have to decode more)

perf data: no significant difference with Qwen3-8B:

python3 -m sglang.test.test_deterministic --test-mode prefix --temperature 0.7

• with radix cache with deterministic, batch = 10: 3.3927013874053955 seconds
• with radix cache w/o deterministic, batch = 10: 3.073150634765625 seconds

@Fridge003 @skyzh is it ready to merge?

@hebiao064 not yet - I have mid-confidence on whether this patch makes sense

@hebiao064 Let's wait for opinions from @hnyls2002

addressed the comments and ready for review again :) thanks!