Add CI metrics collection infrastructure for stage-b jobs by harvenstar · Pull Request #16579 · sgl-project/sglang

harvenstar · 2026-01-06T16:33:01Z

Background

This PR implements step 2 of the roadmap outlined in #15996: Set up CI infrastructure to collect and upload test metrics artifacts.

Builds on top of #16064 which added the dump_metric() function and integrated it into run_eval.py.

Changes

New script: scripts/ci/merge_metrics.py
- Merges per-process JSONL files (test_metrics.*.jsonl) into single output
- Pure stdlib implementation (no sglang dependency)
- JSON validation with skip-on-error for robustness
- UTF-8 encoding to avoid runner differences
- Creates parent directories automatically
CI workflows: pr-test.yml and pr-test-amd.yml
- Set SGLANG_TEST_METRICS_OUTPUT in all stage-b jobs:
  - stage-b-test-small-1-gpu (8 partitions)
  - stage-b-test-large-1-gpu
  - stage-b-test-large-2-gpu
  - stage-b-test-4-gpu-b200
  - AMD equivalents (stage-b-test-small-1-gpu-amd, stage-b-test-large-2-gpu-amd)
- Merge metrics with if: always() (runs even on test failure)
- Upload entire $METRICS_DIR/ as artifact with if: always()
Artifact contents
- test_metrics.jsonl - merged metrics for trend analysis
- test_metrics.*.jsonl - per-worker raw files for debugging

Reason

Flaky test diagnosis: Collect metrics even when tests fail
Root cause analysis: Raw per-worker files help identify which process had issues
Regression detection: Merged file enables historical trend analysis
Future work: Foundation for automated regression detection system

Verification

CI jobs will produce regression-metrics-{run_id}-{job}-[partition-{N}] artifacts
Each artifact contains both merged and raw JSONL files
Works on both NVIDIA and AMD runners (handles container mount correctly)
Artifact names are unique for parallel workers

Next Steps

After this PR:

Step 3: Migrate test_standalone_speculative_decoding.py to use dump_metric()
Step 4: Migrate test_hicache_variants.py to use dump_metric()

Related: #15996
Depends on: #16064

gemini-code-assist · 2026-01-06T16:33:15Z

Summary of Changes

Hello @harvenstar, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes the foundational infrastructure for collecting and uploading test metrics within the CI system. It introduces a new Python utility to consolidate individual process-level metric logs into a unified file, and integrates this process into existing CI workflows. The primary goal is to enhance diagnostic capabilities for flaky tests, facilitate root cause analysis, and enable future regression detection by consistently capturing and archiving detailed performance metrics.

Highlights

New merge_metrics.py script: Introduces a Python script (scripts/ci/merge_metrics.py) to merge per-process JSONL metric files into a single output, with robust JSON validation and error skipping for invalid lines.
CI Workflow Updates: Modifies pr-test.yml and pr-test-amd.yml to set SGLANG_TEST_METRICS_OUTPUT in all stage-b jobs, merge metrics using the new script, and upload the resulting metric artifacts, even if tests fail.
Metric Artifacts: Configures CI to upload both test_metrics.jsonl (merged for trend analysis) and test_metrics.*.jsonl (raw per-worker files for debugging) as artifacts, providing comprehensive data for analysis.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

Ignored by pattern: .github/workflows/** (2)
- .github/workflows/pr-test-amd.yml
- .github/workflows/pr-test.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a new script, scripts/ci/merge_metrics.py, to collect and merge CI metrics, which is a solid step towards better test analysis and regression detection. The script is well-structured and robust in handling invalid data.

My review focuses on improving the script's robustness and maintainability. I've identified a potential bug where the script could fail if no input metric files are found and the output directory doesn't exist. I've also suggested using Python's standard argparse module for command-line argument parsing to make the script more user-friendly and maintainable.

Overall, these are great changes that build a strong foundation for CI metrics collection.

- Add scripts/ci/merge_metrics.py to merge per-process JSONL files - Configure SGLANG_TEST_METRICS_OUTPUT in stage-b jobs - Upload metrics artifacts with if: always() for flaky test diagnosis - Include both merged and raw per-worker files for debugging Related to sgl-project#15996, follows up on sgl-project#16064

- Fix bug: ensure parent directory exists before creating empty output file - Add IOError exception handling when creating empty files - Replace manual argument parsing with argparse module for better UX

Resolved conflicts by preserving metrics collection functionality while adopting upstream changes: - pr-test.yml: Keep metrics collection for stage-b-test-4-gpu-b200 - pr-test-amd.yml: Keep metrics collection and update suite names to match upstream naming conventions (stage-b-test-small-1-gpu-amd), update partition size to 12, and include new mi35x job

Add cleanup step to remove stale metrics from previous runs on self-hosted runners. This ensures each CI run starts with a clean metrics directory and only uploads metrics from the current run.

merrymercy

Can you show us an example run with example output file?

merrymercy · 2026-01-11T13:25:25Z

      matrix:
        runner: [linux-mi325-gpu-2]
    runs-on: ${{matrix.runner}}
+    env:


Setting this for every job is too redundant. Just use a global default value for all cases.

merrymercy · 2026-01-11T13:31:54Z

+        uses: actions/upload-artifact@v4
+        with:
+          name: regression-metrics-${{ github.run_id }}-${{ github.job }}
+          path: ${{ env.METRICS_DIR }}/


Adding three steps for each job is too redundant! Please do the following

Unify your code into a github action like actions/checkout@v4, so we can update the code in a single place without changing everywhere.

You can register post processing in the action, so the last two steps "Merge metrics" and "Upload metrics artifact" do not need to be there. Please refer to the "Post checkout code". It will run but users do not need to add one more step in the workflow file. It is coming form the actions/checkout@v4. Also, please merge the last two steps.

- Create .github/actions/collect-test-metrics composite action - Action handles setup, merging, and uploading of test metrics - Update 4 jobs in pr-test.yml to use the new action: * stage-b-test-small-1-gpu * stage-b-test-large-1-gpu * stage-b-test-large-2-gpu * stage-b-test-4-gpu-b200 - Add global env variables for METRICS_DIR and SGLANG_TEST_METRICS_OUTPUT - Remove redundant per-job env definitions in pr-test-amd.yml

Resolved conflicts in .github/workflows/pr-test-amd.yml by: - Keeping global env variables (METRICS_DIR, SGLANG_TEST_METRICS_OUTPUT) - Adopting upstream changes: removed metrics collection steps, added --timeout-per-file flag - Adopting upstream run-name for /rerun-stage commands

- Add .github/actions/.*/dist/.* to codespell exclude pattern - Fix trailing whitespace in dist/index.js and dist/licenses.txt - Fix end-of-file in dist/index.js The dist/ directory contains bundled third-party dependencies with pre-existing spelling variations that should not be modified.

harvenstar requested review from Fridge003, Kangyan-Zhou, ispobock and merrymercy as code owners January 6, 2026 16:33

github-actions Bot added the amd label Jan 6, 2026

gemini-code-assist Bot reviewed Jan 6, 2026

View reviewed changes

Comment thread scripts/ci/merge_metrics.py

Comment thread scripts/ci/merge_metrics.py

github-actions Bot added the run-ci label Jan 6, 2026

harvenstar added 3 commits January 8, 2026 05:16

Address Gemini code review feedback

c3265da

- Fix bug: ensure parent directory exists before creating empty output file - Add IOError exception handling when creating empty files - Replace manual argument parsing with argparse module for better UX

Make merge_metrics.py executable

a608c1c

harvenstar force-pushed the feat/ci-metrics-collection branch from 9b2b5ae to a608c1c Compare January 8, 2026 05:16

harvenstar added 3 commits January 11, 2026 05:36

Clean up metrics directory before each run

1b34a2c

Add cleanup step to remove stale metrics from previous runs on self-hosted runners. This ensures each CI run starts with a clean metrics directory and only uploads metrics from the current run.

Merge branch 'main' into feat/ci-metrics-collection

ac33698

merrymercy reviewed Jan 11, 2026

View reviewed changes

harvenstar added 2 commits January 12, 2026 14:20

Add built dist files for collect-test-metrics action

c4dca17

github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file labels Jan 12, 2026

harvenstar added 2 commits January 12, 2026 15:23

hnyls2002 mentioned this pull request Mar 23, 2026

[CI Infrastructure] Roadmap: Regression-Based CI Checks #21157

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CI metrics collection infrastructure for stage-b jobs#16579

Add CI metrics collection infrastructure for stage-b jobs#16579
harvenstar wants to merge 10 commits intosgl-project:mainfrom
harvenstar:feat/ci-metrics-collection

harvenstar commented Jan 6, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jan 6, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

merrymercy left a comment

Uh oh!

merrymercy Jan 11, 2026

Uh oh!

merrymercy Jan 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

harvenstar commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Changes

Reason

Verification

Next Steps

Uh oh!

gemini-code-assist Bot commented Jan 6, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

merrymercy left a comment

Choose a reason for hiding this comment

Uh oh!

merrymercy Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

merrymercy Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

harvenstar commented Jan 6, 2026 •

edited

Loading