Motivation

According to the kernel design, group_k is expected to be divisible by BLOCK_SIZE_K. However, when BLOCK_SIZE_K is smaller than group_k, scale_step_k is always computed as 0, which prevents the scaling pointer from advancing.

For example, with BLOCK_SIZE_K = 64 and group_k = 128, the current implementation results in scale_step_k = 0.

This fix ensures the kernel correctly handles such cases by properly updating the scaling pointer.

Modifications

1. Compute the number of blocks within each group_k (i.e., how many blocks share the same scaling parameters).
2. Update scale_step_k to 1 only after the last block in the group has consumed the shared scaling parameters, ensuring the pointer advances correctly.

Accuracy Tests

Server Launching Scripts:

# before using default fp8 config, which mean BLOCK_SIZE_K is always equal to group_k
python -m sglang.launch_server --model-path /models/Qwen3.5-35B-A3B-FP8/0b2752837483aa34b3db6e83e151b150c0e00e49 --tp-size 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --port 8888 --mem-fraction-static 0.9


# after using tuned fp8 config
python -m sglang.launch_server --model-path /models/Qwen3.5-35B-A3B-FP8/0b2752837483aa34b3db6e83e151b150c0e00e49 --tp-size 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --port 8889 --mem-fraction-static 0.9

E2E Accuracy Tests

MMLU

# before
python3 -m sglang.test.run_eval --port 8888 --eval-name mmlu --num-examples 200 --repeat 3  --temperature 1.0 --top-p 0.95 --top-k 20 --min-p 0 --max-tokens 4096

# after
python3 -m sglang.test.run_eval --port 8889 --eval-name mmlu --num-examples 200 --repeat 3 --temperature 1.0 --top-p 0.95 --top-k 20 --min-p 0 --max-tokens 4096

GSM8k

Kernel Accuracy Tests

The maximum element-wise difference of 1 can be attributed to accumulated numerical errors when using BLOCK_SIZE_K = 64. However in this configuration, the current kernel is more prone to error accumulation, which can lead to noticeable inaccuracies.

Benchmarking and Profiling

Kernel Performance Comparison

The tiny us overhead towards this PR, but this can be fine to make the computation logic correct

E2E Benchmark
Before:

After:

The correctness check and performance benchmark about kernel can be seen in https://github.com/Muqi1029/Awesome-LLM-Training-Serving/blob/main/tutorial/triton/benchmark/fp8_kernel.py

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.