Motivation

Fix #23142.

Quark-quantized GLM-5-MXFP4 checkpoints store MTP (NextN) weights — including eh_proj — in FP4-packed format. The existing code always creates eh_proj as nn.Linear, causing a shape mismatch ([6144, 6144] vs [6144, 12288]) during weight loading.

Modifications

• In DeepseekV3ForCausalLMNextN.__init__, check whether the MTP layer is listed in quark's exclude_layers. If so, set quant_config to None so the NextN model falls back to bf16 parameters.
• In DeepseekModelNextN.__init__, use ReplicatedLinear instead of nn.Linear for eh_proj when the quant config is quark, so that QuarkLinearMethod creates parameters matching the FP4-packed checkpoint format.
• Update forward() to handle the (output, output_bias) tuple returned by ReplicatedLinear.
• The existing modelopt_fp4 (NVFP4) code path is unchanged.

Accuracy Tests

python3 -m sglang.launch_server \
    --model-path /data/models/GLM-5-mxfp4/ \
    --port 9000 --trust-remote-code --tp 8 --chunked-prefill-size 131072 \
    --disable-radix-cache \
    --mem-fraction-static 0.85 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --watchdog-timeout 1200 \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.