Skip to content

feat(core): use md reporter for AI agent by default#911

Merged
fi3ework merged 4 commits intomainfrom
md
Feb 2, 2026
Merged

feat(core): use md reporter for AI agent by default#911
fi3ework merged 4 commits intomainfrom
md

Conversation

@fi3ework
Copy link
Copy Markdown
Member

@fi3ework fi3ework commented Jan 27, 2026 •
edited
Loading

Summary

Group 1 (1)

Related Links

Checklist

  • Tests updated (or not required).
  • Documentation updated (or not required).

@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist bot commented Jan 27, 2026

Summary of Changes

Hello @fi3ework, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the test runner's capabilities for AI-driven development by introducing a dedicated Markdown (md) reporter. This new reporter outputs test results in a highly structured and machine-readable Markdown format, making it ideal for AI agents and LLMs to process and understand test outcomes. Crucially, it is now the default reporter in AI agent environments when no other reporting configuration is provided, ensuring seamless integration with automated systems. The changes also include robust testing and updated documentation to support this new feature.

Highlights

  • New Markdown Reporter: Introduced a new md reporter designed to output test results in a structured Markdown format, optimized for consumption by AI agents and Large Language Models (LLMs).
  • Default Reporter for AI Agents: The md reporter is now the default reporter in AI agent environments if no other reporters are explicitly configured, streamlining AI integration.
  • Detailed Failure Reporting: The md reporter provides comprehensive details for test failures, including error types (e.g., AssertionError, SnapshotMismatchError), console output, code frames, stack traces, and reproduction commands.
  • Extensive E2E Tests: Added a suite of end-to-end tests to validate the functionality and output of the new md reporter across various failure scenarios like thrown errors, console logs, snapshot mismatches, and timeouts.
  • Documentation Updates: Updated both English and Chinese documentation to include details about the new md reporter, its purpose, usage, and the default behavior in AI agent environments.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jan 27, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Markdown reporter (MdReporter) designed for AI agent and LLM integrations. It intelligently defaults to this reporter in agent environments when no other reporter is explicitly configured, enhancing the tool's usability in automated contexts. The changes include the full implementation of the MdReporter, updates to the CLI to conditionally display the rstest banner, and comprehensive documentation in both English and Chinese. The new reporter provides detailed, structured output for test failures, including code frames, stack traces, and reproduction commands, which is highly beneficial for debugging and automated analysis.

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages bot commented Jan 27, 2026 •
edited
Loading

Deploying rstest with  Cloudflare Pages  Cloudflare Pages

Latest commit: a465df1
Status: ✅  Deploy successful!
Preview URL: https://0b767976.rstest.pages.dev
Branch Preview URL: https://md.rstest.pages.dev

View logs

@fi3ework fi3ework force-pushed the md branch 6 times, most recently from 706283b to c48e232 Compare January 28, 2026 05:50
@fi3ework fi3ework marked this pull request as ready for review January 28, 2026 06:24
Copilot AI review requested due to automatic review settings January 28, 2026 06:24
Copilot AI reviewed Jan 28, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an AI/LLM-friendly markdown (md) reporter, wires it into the core reporter system, makes it the default in AI agent environments, and documents the new behavior along with comprehensive e2e coverage.

Changes:

  • Adds a new MdReporter implementation with configurable presets and options tailored for agents (repro commands, trimmed stacks, candidate files, console limits, etc.).
  • Updates the CLI to detect AI agent environments, default to the md reporter when no reporter is explicitly configured, and avoid printing the usual banner for agents.
  • Extends reporter type definitions, documentation (EN + ZH), and adds extensive e2e tests and fixtures for the new markdown reporter behavior (failures, timeouts, snapshot mismatches, console output, truncation, and passing runs).

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
website/docs/zh/guide/basic/reporters.mdx Documents the md reporter in the Chinese basic guide and links built-in reporter options to packages/core/src/types/reporter.ts.
website/docs/zh/config/test/reporters.mdx Notes the centralized reporter options type definitions and describes the AI agent defaulting to ['md'] when no reporters are configured (ZH config docs).
website/docs/en/guide/basic/reporters.mdx Adds the md reporter to the built-in reporters table and explains its markdown output for agent/LLM use in the English guide.
website/docs/en/config/test/reporters.mdx Adds a reporter options reference and an “AI agent environments” info block describing the ['md'] default behavior (EN config docs).
packages/core/src/types/reporter.ts Defines BuiltInReporterNames including 'md', introduces MdReporterOptions, extends BuiltinReporterOptions, and adds filterRerunTestPaths to the reporter onTestRunEnd contract.
packages/core/src/reporter/md.ts Implements MdReporter, including option resolution, stack/frames/sourcemap handling, candidate file scoring, code-frame generation, repro command construction based on package manager, and markdown report generation.
packages/core/src/core/rstest.ts Registers MdReporter in reportersMap, imports BuiltInReporterNames from the types module, passes fileFilters and command into reporter init, and re-exports the reporter name type.
packages/core/src/cli/init.ts After resolving config, detects AI agent environments and, when no reporters are configured and no --reporter is provided, overrides config.reporters to ['md'].
packages/core/src/cli/commands.ts Uses determineAgent() to skip showRstest() when running under an AI agent, avoiding noisy banners for automated consumers.
e2e/reporter/rstest.agentMd.truncated.config.ts E2E config targeting fixtures/agent-md/truncated.test.ts with md reporter in a compact, truncated-failures setup for testing failure list truncation behavior.
e2e/reporter/rstest.agentMd.timeout.config.ts E2E config for timeout behavior with compact md reporter, reduced testTimeout, and minimal extra output.
e2e/reporter/rstest.agentMd.throw.config.ts E2E config for thrown error formatting with compact md reporter and simplified stack/candidate output.
e2e/reporter/rstest.agentMd.snapshotMismatch.config.ts E2E config to exercise snapshot mismatch labeling under the compact md reporter.
e2e/reporter/rstest.agentMd.pass.config.ts E2E config for passing/skip scenarios using the compact md reporter, used to validate focused vs non-focused passing run behavior.
e2e/reporter/rstest.agentMd.console.config.ts E2E config enabling console capture with specific console limits to verify console output handling in the md reporter.
e2e/reporter/rstest.agentMd.config.ts Shared E2E config for general agent-md failure cases with the compact md reporter preset.
e2e/reporter/md.test.ts Comprehensive E2E test suite asserting markdown output snapshots for diffs, snapshot mismatches, console logs, thrown errors, timeouts, truncation behavior, and passing test listing; contains a minor mismatch between one test’s description and its snapshot (commented).
e2e/reporter/index.test.ts Adjusts the “default - single file” reporter test to use a path relative to the reporter E2E directory (fixtures/index.test.ts) with cwd set to __dirname.
e2e/reporter/fixtures/agent-md/truncated.test.ts Fixture with multiple failing tests to drive and verify failure list truncation and repro command generation in the md reporter.
e2e/reporter/fixtures/agent-md/timeout.test.ts Fixture containing a deliberately non-resolving promise to trigger and test timeout error reporting.
e2e/reporter/fixtures/agent-md/throw.test.ts Fixture that throws a plain error to validate non-assertion error formatting in markdown output.
e2e/reporter/fixtures/agent-md/snapshotMismatch.test.ts Fixture with an inline snapshot mismatch to exercise snapshot-specific error typing in the markdown reporter.
e2e/reporter/fixtures/agent-md/index.test.ts Basic failing test fixture used to verify core markdown failure formatting and summary.
e2e/reporter/fixtures/agent-md/console.test.ts Fixture that logs to console.log/warn/error before a failure, used to validate captured console output and code-frame rendering.
e2e/reporter/fixtures/agent-md-pass/many.test.ts Many-passing-tests fixture supporting focused vs non-focused pass behavior checks for the md reporter.
e2e/reporter/fixtures/agent-md-pass/focusedSkip.test.ts Fixture with one passing and one skipped test to validate how passed and skipped lists are rendered in markdown reports.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@9aoy
Copy link
Copy Markdown
Collaborator

9aoy commented Jan 28, 2026 •
edited
Loading

image

fullName containing the > symbol can't be used for filtering. Maybe we could support this in the --testNamePattern filter?

@9aoy 9aoy mentioned this pull request Jan 28, 2026
Merged
2 tasks
@fi3ework fi3ework merged commit da6b94d into main Feb 2, 2026
20 of 25 checks passed
@fi3ework fi3ework deleted the md branch February 2, 2026 14:29
@fi3ework fi3ework mentioned this pull request Feb 3, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

change: feat

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fi3ework @9aoy