Skip to content

feat: support merge reports for non-sharded multi-environment runs (take 2)#10031

Merged
sheremet-va merged 87 commits intovitest-dev:mainfrom
hi-ogawa:feat-merge-report-label-v2
May 4, 2026
Merged

feat: support merge reports for non-sharded multi-environment runs (take 2)#10031
sheremet-va merged 87 commits intovitest-dev:mainfrom
hi-ogawa:feat-merge-report-label-v2

Conversation

@hi-ogawa
Copy link
Copy Markdown
Collaborator

@hi-ogawa hi-ogawa commented Mar 31, 2026

Description

Alternative take for #9967 (comment)

This PR adds new blob reporter option label so --merge-reports can distinguish results from the same test files across different environments, such as linux/macos/windows CI jobs.

Usage example

Set VITEST_BLOB_LABEL when creating blobs:

VITEST_BLOB_LABEL=linux vitest run --reporter=blob
VITEST_BLOB_LABEL=macos vitest run --reporter=blob

Or pass the label via the blob reporter option:

// vitest.config.ts
export default defineConfig({
  test: {
    reporters: [
      ['blob', { label: process.platform }],
    ],
  },
})

The label is included in the default blob filename (blob-linux.json) to avoid collisions. When merging, each labeled run is shown separately:

$ vitest --merge-reports
 ✓  linux  src/basic.test.ts (2 tests)
 ✓  macos  src/basic.test.ts (2 tests)

How it works

The label is a blob reporter option, with VITEST_BLOB_LABEL as an environment-variable shortcut for CI usage. During config resolution, Vitest extracts the label from ['blob', { label }] or process.env.VITEST_BLOB_LABEL and stores it as internal core config (mergeReportsLabel).

During the original test run, this label is not exposed as a normal built-in reporter label. This keeps terminal output unchanged for runs that create blobs with another reporter enabled, for example reporters: ['default', ['blob', { label: 'linux' }]].

The label is stored in blob task metadata and salts the file identity used for merged reports, so state deduplication can keep same-filepath files from different environments separate. The default blob output filename also uses the label as blob-(label)-(shard).json when no explicit output file is configured.

Reporters read the label from merged blob metadata and display it when replaying --merge-reports.

The difference from #9967

Concern #9967 (label as project.name) This PR (label as blob metadata)
Mechanism Clone TestProject per label at read time Store label in blob task metadata and salt file identity
readBlobs changes Significant — label discovery, ID rewrite, env module rewrite No project cloning or read-time project rewrite
file.projectName Mutated to include label projectName unchanged
API [["blob", { label }]] reporter option or VITEST_BLOB_LABEL [["blob", { label }]] reporter option or VITEST_BLOB_LABEL (same)
Original run output Label could appear as a project-style label Built-in reporters do not show the blob label during the original run

Demo

I setup CI to upload html reports for test/cli and test/core. It's viewable on newly introduced viewer #10023.

https://viewer.vitest.dev/?url=https://github.com/vitest-dev/vitest/actions/runs/24275938517/artifacts/6383832383

image

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
  • Ideally, include a test that fails without this PR but passes with it.
  • Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.
  • Please check Allow edits by maintainers to make review process faster. Note that this option is not available for repositories that are owned by Github organizations.

Tests

  • Run the tests with pnpm test:ci.

Documentation

  • If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

  • Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

hi-ogawa and others added 5 commits April 29, 2026 15:30
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
@sheremet-va
Copy link
Copy Markdown
Member

LGTM, but I inlined generateFileId in the orchestrator which is not completely correct now:

function generateFileId(file: string) {

Importing from utils also imports the runner for some reason and the size balloons. I am removing the runner package in the next PR anyway

sheremet-va
sheremet-va previously approved these changes Apr 29, 2026
Comment thread docs/guide/reporters.md Outdated
Comment thread packages/vitest/src/node/project.ts
Comment thread test/unit/vite.config.ts
: [['default', { summary: true }], 'hanging-process'],
reporters: [
...(process.env.GITHUB_ACTIONS
? [['default'], ['github-actions', { displayAnnotations: false }] as any]
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw we should probably add displayAnnotations to test utils where we spin up Vitest inside Vitest. Right now all those nested runs are spamming their results to Github's env, so Actions tab looks really verbose.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it should just be disabled by default overall

hi-ogawa and others added 2 commits April 29, 2026 17:16
Co-authored-by: Ari Perkkiö <ari.perkkio@gmail.com>
@hi-ogawa hi-ogawa requested a review from sheremet-va April 29, 2026 09:10
const kWellKnownMeta = new Set([
'benchmark',
'typecheck',
'label',
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't it renamed?

Copy link
Copy Markdown
Collaborator Author

@hi-ogawa hi-ogawa Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Just replaced.

Btw, this does't actually reach though because both typecheck and __vitest_label__ are file level only meta. (Ah typecheck does appear each suite and test).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think tests merge parent's meta, now? At least that was the behaviour I intended in 4.1

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably an inconsistency because our runtime collect does odd thing with "default root suite" (like fixing up hash after collect), so file is missing as parent suite.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does @module-tag work then 🤔

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't actually verify the raw data. I didn't see any meta in the html report from the commit before the fix, so I was inferring.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so conveniently there's a divergence between tags and meta inheritance

const task = function (name = '', options: TaskCustomOptions = {}) {
const currentSuite = collectorContext.currentSuite?.suite
const parentTask = currentSuite ?? collectorContext.currentSuite?.file
const parentTags = parentTask?.tags || []

const parentMeta = currentSuite?.meta

At the top level, we have currentSuite = undefined but parentTask = file, so file.tags can get inherited but file.meta doesn't.

Unless there's a concrete use cases or bugs, we can probably leave this divergence as is.

@hi-ogawa hi-ogawa requested a review from sheremet-va April 30, 2026 02:11
@sheremet-va
Copy link
Copy Markdown
Member

Lint is failing 😞

@sheremet-va sheremet-va merged commit e60b2f4 into vitest-dev:main May 4, 2026
16 of 17 checks passed
@hi-ogawa hi-ogawa deleted the feat-merge-report-label-v2 branch May 4, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support merging reports for non-sharded multi-environment runs

4 participants