Description

Refactor the DeepEP dispatcher to introduce structured output dtype control, replacing the
SGLANG_DEEPEP_BF16_DISPATCH environment variable with a DeepOutputDtype enum and
automatic detection. This is part of a broader effort to make the dispatch pipeline robust for
quantized MoE models — especially on Ascend NPU — while reducing memory overhead and
simplifying user configuration.

Motivation

The previous dispatch pipeline had several shortcomings:

• Hardcoded FP8 assumption. The dispatcher unconditionally enabled FP8 dispatch for
  non-NPU, non-flashinfer-cuteDSL paths, causing incorrect results for W8A8 quant and
  BF16-pretrained models.
• Fragile env-var workaround. Users were forced to set
  SGLANG_DEEPEP_BF16_DISPATCH=1 to disable FP8, but this variable was not properly
  plumbed through all code paths (e.g., _dispatch_core in low‑latency mode) and was
  globally scoped, leading to subtle bugs.
• Static buffer sizing. params_bytes was hardcoded to 2, which wasted up to 50%
  of the staging buffer memory when dispatching in FP8 or NVFP4 (where 1 byte
  per parameter is sufficient).
• No scheme‑aware dispatch. Quantization schemes like ModelSlimW4A4Int4MoE and
  NPU‑compressed‑tensors MoE methods needed manual tuning but lacked any automatic
  dtype selection.

Modifications

A concise summary of the changes follows; see the commit history for full details.

1. Introduce DeepOutputDtype enum (moe/utils.py)

• New enum with members BF16, FP8, NVFP4.
• Server argument --deepep‑dispatch‑output‑dtype accepts bf16, fp8, nvfp4, or
  the default auto.
• Deprecated SGLANG_DEEPEP_BF16_DISPATCH — removed from docs, scripts, and enforce‑
  path assert.

2. Automatic detection logic (moe/utils.py)

• New function get_deepep_output_dtype() that infers the correct output dtype from:
  0. Explicit deprecated env variables.
  1. Explicit server argument (if not auto).
  2. If quant_config contains input_global_scale → NVFP4
  3. If quant_config contains dispather_output_dtype → parse it
  4. If flashinfer_cutedsl runner backend → NVFP4 BF16 .
  5. Fallback to BF16 on NPU.
  6. Fallback to FP8 for standard FP8‑pretrained models.

3. Dispatcher refactor (deepep.py)

• _DeepEPDispatcherImplBase.__init__ now receives deepep_output_dtype (as
  DeepOutputDtype), sets self.use_fp8, self.use_nvfp4, and
  computes self.params_bytes dynamically based on the resolved dtype:
  • BF16 → params_bytes = 2
  • FP8 / NVFP4 → params_bytes = 1
• dispatch_a / _dispatch_core in both normal and low‑latency implementations
  no longer compute use_fp8 locally; they use the pre‑computed flags stored in self.
• Staging buffer sizes (nvl_buffer_size, rdma_buffer_size) shrink proportionally
  as params_bytes is now computed per‑layer.

4. NPU‑specific path enhancements (ep_moe/layer.py)

• Explicit isinstance(self.scheme, (ModelSlimW4A4Int4MoE,)) check that calls
  torch_npu.npu_dynamic_quant(hidden_states, dst_type=torch.quint4x2) before dispatch,
  enabling the correct BF16 output path for ModelSlimW4A4Int4MoE on Ascend.
• NPU compressed‑tensors fused W4A16 MoE method patched to pass through the proper
  deepep_output_dtype inferred from the scheme.
• AWQ DeepEP now supported on Ascend.

5. Additional improvements

• Kimi K2.5 (and some other compressed‑tensors models): loading speed improved up to 3× on NPU.
• CI: updated CI to mention the new server argument
  and remove references to the deprecated env var.
• Docs: updated expert‑parallelism documentation to mention the new server argument
  and remove references to the deprecated env var.

Accuracy Tests and Speed Tests

Kimi K2.5

Qwen3.5-35B-A3B-w8a8-mtp

server:
SGLANG_SET_CPU_AFFINITY=1 PYTORCH_NPU_ALLOC_CONF=expandable_segments:True STREAMS_PER_DEVICE=32 HCCL_BUFFSIZE=1536 ASCEND_RT_VISIBLE_DEVICES=12,13,14,15 python3 -m sglang.launch_server --model-path "./weights/Qwen/Qwen3.5-35B-A3B-w8a8-mtp/" --tp 4 --mem-fraction-static 0.8 --max-total-tokens 66000 --trust-remote-code --attention-backend ascend --device npu --host 127.0.0.1 --port 30088 --moe-a2a-backend deepep --deepep-mode auto --cuda-graph-max-bs 128 --disable-radix-cache

client:
python ./benchmark/gsm8k/bench_sglang.py --num-questions 1319 --port 30088 --data-path ./datasets/gsm8k/test.jsonl --parallel 128

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

2. Ping Merge Oncalls to start the process. See the PR Merge Process.
3. Get approvals from CODEOWNERS and other reviewers.
4. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
5. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.