Backbone: Attention Backend, batch_invariant library integration

• Support cuda graph [1/2] Support deterministic inference with flashinfer attention backend #10645 @Fridge003
• Support temperature > 0 [2/2] Support deterministic inference for temperature > 0 #10678 @Qiaolin-Yu @Fridge003
• Attention Backend: Flashinfer: feat: Batch-size invariant FA2 Prefill & Decode flashinfer-ai/flashinfer#1675 [1/2] Support deterministic inference with flashinfer attention backend #10645 @Edenzzzz @Fridge003
• Attention Backend: Triton Add split tile size for Triton attention #10425 Support deterministic inference with triton backend (Hardware test: NV and AMD GPUs) #10694 @yushengsu-thu @ispobock
• Attention Backend: FA3 [Feature] Support deterministic inference with FA3 backend #10651 @hebiao064
• Move batch invariant triton kernels to sglang repo [deterministic inference] Move batch invariant pkg to sglang #10695 @hebiao064
• Add unit tests convert test_deterministic into unit tests #11095 [logprobs] Enable local deterministic logrprobs testing with strict threshold #10994 Fix batch invariant ops #11368 @skyzh

Communication (NCCL)

• Support tensor parallelism with deterministic all-reduce kernels [Feature] Support deterministic inference for MoE in large TP #10785 fix: make inference deterministic for large TP #10930 @JustinTong0323 @yuan-luo
• Support tensor parallelism with deterministic all-reduce kernels (AMD) [amd] Add deterministic all-reduce kernel for AMD (ROCm) #15340 @sunxxuns
• Not deterministic on Blackwell for TP4 [Bug] Deterministic Inference doesn't work on Blackwell TP4 #11513

Radix Cache Support

• FA3 supported given its already 1-stage prefill at the beginning
• Triton Supported by PR: Deterministic Mode: Add 1-stage triton kernel for prefill #11147 @hebiao064 @zminglei @byjiang1996
• FlashInfer Support
• Making Prefill with Radix Cache has the same output as Prefill without Radix cache @hebiao064 @hanming-lu

Model Support

• Qwen3 Dense
• Qwen3 MOE
• Deepseek v3 Deterministic Support @zminglei @b8zhong [2/2] Deepseek deterministic: support deepseek v3 deterministic inference on 8 x H200 #12095
• Qwen3 Next (Linear Attention) @zminglei @vedantjh2 [Feature] Support Deterministic Inference for QWen3-Next #12845

Quantization

• Blockwise FP8 Kernel [Deterministic] Blockscale FP8 kernel #11491 @b8zhong
• per-channel fp8 Gemm
• fp8 fused MoE
• nvfp4 Gemm/MoE

Parallelism

• Support DP Attention [Fix] fix enable_deterministic_inference failed with enable_dp_attention #11023
• Support EP

Spec Decoding

• Speculative decoding drafters [question] appropriate benchmarks #8391 [Question] Speculative Decoding: Sampling from the draft probs vector #9877

Perf

• Accelerate batch invariant triton kernels Support DeepGEMM for deterministic inference #12142 Add Batch‑Invariant RMSNorm #12144

Issues

Usability & Documentation

• Set Default Argument (e.g Attention Backend): set default attention backend for deterministic inference #11801 @zminglei
• Write Documentation to https://docs.sglang.ai/ @zminglei [Documentation] add doc for deterministic inference #11956

Related resources

https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
https://github.com/thinking-machines-lab/batch_invariant_ops