Add unified metrics collection framework (v1) by harvenstar · Pull Request #16064 · sgl-project/sglang

harvenstar · 2025-12-29T09:11:27Z

Implement dump_metric() function for test metrics collection:

Outputs to JSONL (${SGLANG_TEST_METRICS_OUTPUT}.${pid}.jsonl)
Outputs to stdout ([METRIC] format)
Schema: required {filename, test_case, metric_name, value}
Optional: {ts, labels}
Supports int/float/str, coerces bool to int
Auto-captures test context via PYTEST_CURRENT_TEST or inspect
Never fails tests, all errors silently caught

Includes comprehensive unit tests covering:

JSONL format validation
Value type conversions
Labels serialization handling
Environment variable behavior

Part of #15996 - unified CI metrics collection framework.

Implement dump_metric() function for test metrics collection: - Outputs to JSONL (${SGLANG_TEST_METRICS_OUTPUT}.${pid}.jsonl) - Outputs to stdout ([METRIC] format) - Schema: required {filename, test_case, metric_name, value} - Optional: {ts, labels} - Supports int/float/str, coerces bool to int - Auto-captures test context via PYTEST_CURRENT_TEST or inspect - Never fails tests, all errors silently caught Includes comprehensive unit tests covering: - JSONL format validation - Value type conversions - Labels serialization handling - Environment variable behavior Part of sgl-project#15996 - unified CI metrics collection framework.

gemini-code-assist · 2025-12-29T09:11:45Z

Summary of Changes

Hello @harvenstar, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes a foundational, unified framework for collecting test metrics within the sglang project. It introduces a dump_metric function designed to standardize how performance and other test-related data are recorded, outputting them to both structured JSONL files for programmatic consumption and human-readable stdout logs. The system is built for robustness, automatically capturing test context and handling various data types without interrupting test execution, thereby facilitating consistent and reliable CI metrics collection.

Highlights

Unified Metric Collection: Introduced a new dump_metric function to standardize the collection of test metrics across the project.
Flexible Output Formats: Metrics are outputted to both a structured JSONL file (specified by SGLANG_TEST_METRICS_OUTPUT environment variable) and to stdout in a human-readable [METRIC] format.
Standardized Metric Schema: Each metric record adheres to a defined schema, including required fields like filename, test_case, metric_name, value, and optional fields such as ts (timestamp) and labels.
Robust Value and Context Handling: The function supports int, float, and str values, coercing booleans to integers (True=1, False=0), and attempts best-effort conversion for other types. It automatically captures test context (filename and test case) using PYTEST_CURRENT_TEST or Python's inspect module.
Error Resilience: All errors during metric collection are silently caught, ensuring that the metric dumping process never causes tests to fail.
Comprehensive Unit Tests: A new test file python/sglang/test/test_dump_metric.py has been added, providing extensive coverage for JSONL format validation, value type conversions, label serialization, and environment variable behavior.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a valuable framework for unified metrics collection in tests, which will be very useful for CI performance tracking. The implementation is robust, featuring silent error handling to prevent test failures and a comprehensive set of unit tests. I have a few suggestions to enhance it further:

In test_dump_metric.py, I recommend refactoring to avoid code duplication in setUp and tearDown for better maintainability.
In dump_metric, I suggest adding logging for caught exceptions. This will significantly aid debugging when metrics are not being reported, without violating the "never fail" contract.
To improve the consistency and parsability of stdout logs, I propose using JSON serialization for the labels dictionary.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Per reviewer feedback, integrate the unified metrics collection framework into run_eval() function to automatically report eval metrics in both single-run and repeat-run scenarios. This enables all evaluations using run_eval() to automatically output metrics to the standardized JSONL format for CI performance tracking. Metrics reported: - {eval_name}_score: evaluation score (single-run mode) - {eval_name}_latency: evaluation latency (single-run mode) - {eval_name}_mean_score: mean score across runs (repeat mode) All metrics include labels with model name, eval name, and repeat count (when applicable) for better tracking and analysis.

Kangyan-Zhou · 2026-01-02T07:18:42Z

/tag-and-rerun-ci

Kangyan-Zhou · 2026-01-02T18:42:46Z

/tag-and-rerun-ci

- Add scripts/ci/merge_metrics.py to merge per-process JSONL files - Configure SGLANG_TEST_METRICS_OUTPUT in stage-b jobs - Upload metrics artifacts with if: always() for flaky test diagnosis - Include both merged and raw per-worker files for debugging Related to sgl-project#15996, follows up on sgl-project#16064

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

gemini-code-assist Bot reviewed Dec 29, 2025

View reviewed changes

Comment thread python/sglang/test/test_dump_metric.py

Comment thread python/sglang/test/test_utils.py Outdated

Comment thread python/sglang/test/test_utils.py Outdated

Comment thread python/sglang/test/test_utils.py Outdated

harvenstar and others added 6 commits December 29, 2025 18:25

Update python/sglang/test/test_dump_metric.py

1a79277

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update python/sglang/test/test_utils.py

a7ce167

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update python/sglang/test/test_utils.py

94e5c32

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update python/sglang/test/test_utils.py

57a9365

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Fix lint: remove unnecessary pass statements after logging

c384c7d

github-actions Bot added the run-ci label Jan 2, 2026

harvenstar added 4 commits January 2, 2026 20:15

Merge branch 'main' into feat/unified-metrics-framework

ea4ef02

trigger CI

e0ab333

Fix circular import by using lazy import for dump_metric

8ca7aa7

Fix isort formatting in run_eval.py

3334aa1

Merge branch 'main' into feat/unified-metrics-framework

c82e9ce

Kangyan-Zhou merged commit f4ab2ec into sgl-project:main Jan 4, 2026
103 of 108 checks passed

harvenstar mentioned this pull request Jan 6, 2026

Add CI metrics collection infrastructure for stage-b jobs #16579

Open

YChange01 pushed a commit to YChange01/sglang that referenced this pull request Jan 13, 2026

Add unified metrics collection framework (v1) (sgl-project#16064)

2bf7426

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

alisonshao mentioned this pull request Mar 20, 2026

feat: enable unified metrics collection in nightly CI with S3 upload #21057

Open

2 tasks

hnyls2002 mentioned this pull request Mar 23, 2026

[CI Infrastructure] Roadmap: Regression-Based CI Checks #21157

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add unified metrics collection framework (v1)#16064

Add unified metrics collection framework (v1)#16064
Kangyan-Zhou merged 12 commits intosgl-project:mainfrom
harvenstar:feat/unified-metrics-framework

harvenstar commented Dec 29, 2025

Uh oh!

gemini-code-assist Bot commented Dec 29, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kangyan-Zhou commented Jan 2, 2026

Uh oh!

Kangyan-Zhou commented Jan 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

harvenstar commented Dec 29, 2025

Uh oh!

gemini-code-assist Bot commented Dec 29, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kangyan-Zhou commented Jan 2, 2026

Uh oh!

Kangyan-Zhou commented Jan 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants