Motivation

SGLang CI uses static thresholds for accuracy and performance checks. This causes:

• Flaky tests — natural variance triggers false failures, leading to repeated manual threshold adjustments (Relax flaky B200 GSM8K accuracy thresholds #20304, Fix nightly Mistral-Large-3 NVFP4 accuracy threshold #19402, Relax flaky test thresholds for MLA DeepSeek V3 and AutoRound #20068, fix: adjusting vlm accuracy thresholds #16593)
• Late-discovered regressions — no automated baseline comparison, regressions found only via user reports ([Feature] Performance regression and cache behavior change between sglang 0.5.5 and 0.5.6+ with Mooncake #16797, [Bug] Performance Regression: DeepSeek V3.2 Prefill Throughput is 1/3 of V3.1 in Disaggregation Mode #14498, [Bug] DeepSeek v3.1 Performance Regression: 0.5.1 vs 0.5.9 #21012)

The goal is regression-based checking: compare each CI run against historical baselines instead of hardcoded numbers.

Roadmap

Phase 0: dump_metric() foundation         ✅ DONE
Phase 1: Unify eval & bench entry points   🔄 IN PROGRESS
Phase 2: Wire dump_metric into all paths   ⏳ BLOCKED on Phase 1
Phase 3: Durable metrics storage           ⏳ BLOCKED on Phase 2
Phase 4: Regression detection logic        ⏳ BLOCKED on Phase 3

Phase 0: Metrics Collection Foundation — ✅ DONE

dump_metric() merged (#16064), nightly upload to GH Artifacts (#17696), performance dashboard (#17725).

Phase 1: Unify Entry Points — 🔄 IN PROGRESS

dump_metric only works if callers use it. Currently only run_eval.py calls it. Must consolidate fragmented eval/bench systems first.

• Accuracy eval: 5 GSM8K implementations, 4 MMLU — consolidate to ≤2 each. Tracked by Consolidate fragmented evaluation systems #21046.
• Performance bench: bench_serving.py + NightlyBenchmarkRunner + benchmark/ scripts — converge to unified runner. Tracked by [Refactor] Benchmark Scripts Refactor #10177.

Phase 2: Wire dump_metric Into All Paths

Once entry points are unified, add dump_metric() at each one — accuracy (score, latency) and performance (throughput, TTFT, ITL, E2E latency).

Phase 3: Durable Metrics Storage

Metrics must persist across CI runs for baseline comparison. S3 upload attempted (#21057, blocked on AWS creds). Need to decide: S3 vs GH Artifacts vs git-based vs dashboard DB.

Phase 4: Regression Detection

Automated current_metric >= baseline * (1 - tolerance) check in CI.

Key decisions TBD:

• Baseline selection (rolling median of last N nightly runs on main?)
• Per-metric tolerance (accuracy ±2%, throughput ±10%, latency ±15%)
• CI action (block merge vs warning comment)
• Roll out on nightly first, then expand to PR CI

Related