feat: support merge reports for non-sharded multi-environment runs (take 2)#10031
feat: support merge reports for non-sharded multi-environment runs (take 2)#10031sheremet-va merged 87 commits intovitest-dev:mainfrom
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
|
LGTM, but I inlined Importing from utils also imports the runner for some reason and the size balloons. I am removing the runner package in the next PR anyway |
| : [['default', { summary: true }], 'hanging-process'], | ||
| reporters: [ | ||
| ...(process.env.GITHUB_ACTIONS | ||
| ? [['default'], ['github-actions', { displayAnnotations: false }] as any] |
There was a problem hiding this comment.
Btw we should probably add displayAnnotations to test utils where we spin up Vitest inside Vitest. Right now all those nested runs are spamming their results to Github's env, so Actions tab looks really verbose.
There was a problem hiding this comment.
Maybe it should just be disabled by default overall
Co-authored-by: Codex <noreply@openai.com>
… feat-merge-report-label-v2
Co-authored-by: Ari Perkkiö <ari.perkkio@gmail.com>
| const kWellKnownMeta = new Set([ | ||
| 'benchmark', | ||
| 'typecheck', | ||
| 'label', |
There was a problem hiding this comment.
Good catch. Just replaced.
Btw, this does't actually reach though because both typecheck and __vitest_label__ are file level only meta. (Ah typecheck does appear each suite and test).
There was a problem hiding this comment.
I think tests merge parent's meta, now? At least that was the behaviour I intended in 4.1
There was a problem hiding this comment.
This is probably an inconsistency because our runtime collect does odd thing with "default root suite" (like fixing up hash after collect), so file is missing as parent suite.
There was a problem hiding this comment.
Why does @module-tag work then 🤔
There was a problem hiding this comment.
I didn't actually verify the raw data. I didn't see any meta in the html report from the commit before the fix, so I was inferring.
There was a problem hiding this comment.
Okay, so conveniently there's a divergence between tags and meta inheritance
vitest/packages/runner/src/suite.ts
Lines 317 to 320 in 2739334
vitest/packages/runner/src/suite.ts
Line 350 in 2739334
At the top level, we have currentSuite = undefined but parentTask = file, so file.tags can get inherited but file.meta doesn't.
Unless there's a concrete use cases or bugs, we can probably leave this divergence as is.
|
Lint is failing 😞 |
… feat-merge-report-label-v2
Description
Alternative take for #9967 (comment)
This PR adds new blob reporter option
labelso--merge-reportscan distinguish results from the same test files across different environments, such as linux/macos/windows CI jobs.Usage example
Set
VITEST_BLOB_LABELwhen creating blobs:Or pass the label via the blob reporter option:
The label is included in the default blob filename (
blob-linux.json) to avoid collisions. When merging, each labeled run is shown separately:How it works
The label is a blob reporter option, with
VITEST_BLOB_LABELas an environment-variable shortcut for CI usage. During config resolution, Vitest extracts the label from['blob', { label }]orprocess.env.VITEST_BLOB_LABELand stores it as internal core config (mergeReportsLabel).During the original test run, this label is not exposed as a normal built-in reporter label. This keeps terminal output unchanged for runs that create blobs with another reporter enabled, for example
reporters: ['default', ['blob', { label: 'linux' }]].The label is stored in blob task metadata and salts the file identity used for merged reports, so state deduplication can keep same-filepath files from different environments separate. The default blob output filename also uses the label as
blob-(label)-(shard).jsonwhen no explicit output file is configured.Reporters read the label from merged blob metadata and display it when replaying
--merge-reports.The difference from #9967
TestProjectper label at read timereadBlobschangesfile.projectName[["blob", { label }]]reporter option orVITEST_BLOB_LABEL[["blob", { label }]]reporter option orVITEST_BLOB_LABEL(same)Demo
I setup CI to upload html reports for
test/cliandtest/core. It's viewable on newly introduced viewer #10023.https://viewer.vitest.dev/?url=https://github.com/vitest-dev/vitest/actions/runs/24275938517/artifacts/6383832383
Please don't delete this checklist! Before submitting the PR, please make sure you do the following:
pnpm-lock.yamlunless you introduce a new test example.Tests
pnpm test:ci.Documentation
pnpm run docscommand.Changesets
feat:,fix:,perf:,docs:, orchore:.