Skip to content

feat!: --merge-reports to support coverage#5736

Merged
sheremet-va merged 1 commit intovitest-dev:mainfrom
AriPerkkio:feat/merge-reports-coverage
May 25, 2024
Merged

feat!: --merge-reports to support coverage#5736
sheremet-va merged 1 commit intovitest-dev:mainfrom
AriPerkkio:feat/merge-reports-coverage

Conversation

@AriPerkkio
Copy link
Member

@AriPerkkio AriPerkkio commented May 16, 2024 •
edited
Loading

Description

Coverage providers can now have an optional generateCoverage function. The blob reporter will save output of this function in its reports. When --merge-reports is called, the saved blobs will be passed to new mergeReports method.

class MyCoverageProvider {
  generateCoverage(): MyCustomCoverage {
    return { some: "coverage result" } // This will be saved by blob reporter
  }
  // Function called during '--merge-reports'
  mergeReports(coverages: MyCustomCoverage[]) {
            // ^^ List of coverage results
  }
  reportCoverage(coverage: MyCustomCoverage, reportContext: ReportContext) {
              // ^^ Also available in reportContext now
  }
}

BREAKING CHANGES:

  • Affects only custom coverage providers
  • In reportCoverage the previous first argument reportContext is now second argument:
- reportCoverage: (reportContext?: ReportContext) => void | Promise<void>
+ reportCoverage: (coverage: CoverageResults, reportContext: ReportContext) => void | Promise<void>

Example

$ vitest --run --reporter=blob --shard=1/3

 ✓ test/first.test.ts  (1 test)
blob report written to /<root>/vitest-example/.vitest-reports/blob-1-3.json

$ vitest --run --reporter=blob --shard=2/3

 ✓ test/second.test.ts  (1 test) 1ms
blob report written to /<root>/vitest-example/.vitest-reports/blob-2-3.json

$ vitest --run --reporter=blob --shard=3/3

 ✓ test/third.test.ts  (1 test) 1ms
blob report written to /<root>/vitest-example/.vitest-reports/blob-3-3.json

$ vitest --run --merge-reports

 RUN  v2.0.0-beta.3 /<root>/vitest-example

 ✓ test/first.test.ts (1)
 ✓ test/third.test.ts (1)
 ✓ test/second.test.ts (1)

 Test Files  3 passed (3)
      Tests  3 passed (3)
   Start at  15:33:58
   Duration  1ms (transform 0ms, setup 0ms, collect 29ms, tests 3ms, environment 0ms, prepare 126ms)

 % Coverage report from v8
-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------|---------|----------|---------|---------|-------------------
All files  |     100 |      100 |     100 |     100 |                   
 first.ts  |     100 |      100 |     100 |     100 |                   
 second.ts |     100 |      100 |     100 |     100 |                   
 third.ts  |     100 |      100 |     100 |     100 |                   
-----------|---------|----------|---------|---------|-------------------

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
  • Ideally, include a test that fails without this PR but passes with it.
  • Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

  • Run the tests with pnpm test:ci.

Documentation

  • If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

  • Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

@AriPerkkio AriPerkkio force-pushed the feat/merge-reports-coverage branch 2 times, most recently from bcb3ba8 to e910458 Compare May 17, 2024 18:26
AriPerkkio
AriPerkkio commented May 18, 2024
View reviewed changes
@AriPerkkio AriPerkkio force-pushed the feat/merge-reports-coverage branch 2 times, most recently from a8b475d to 90a9848 Compare May 18, 2024 09:34
@AriPerkkio AriPerkkio mentioned this pull request May 18, 2024
Closed
6 tasks
@AriPerkkio
Copy link
Member Author

AriPerkkio commented May 18, 2024

We might need to extend vitest --merge-reports to support specifying separate directory for coverage, e.g. --merge-reports.coverage=./some/directory. By default results are read from .vites-reports/coverage, but if user defines custom directory for --coverage.reporter=blob, it difficult to use path of --merge-reports=<path>. I would like to avoid adding completely separate command --merge-coverage-reports.

@sheremet-va
Copy link
Member

sheremet-va commented May 18, 2024

We might need to extend vitest --merge-reports to support specifying separate directory for coverage, e.g. --merge-reports.coverage=./some/directory. By default results are read from .vites-reports/coverage, but if user defines custom directory for --coverage.reporter=blob, it difficult to use path of --merge-reports=<path>. I would like to avoid adding completely separate command --merge-coverage-reports.

I would like the blob report to just be a single blob file with anything needed for merging. Can't we record everything to a directory (even tmp directory is fine by me), then archive it and unarchive during mering?

@AriPerkkio
Copy link
Member Author

AriPerkkio commented May 18, 2024

That would be great. Should we only allow user to specify directory and nothing else? Then store both test and coverage reports there. When merging the reports, we would respect reporters' outputFile, reportsDirectory etc. flags as usual.

@AriPerkkio AriPerkkio force-pushed the feat/merge-reports-coverage branch from 90a9848 to 4c913db Compare May 18, 2024 10:31
@sheremet-va
Copy link
Member

sheremet-va commented May 18, 2024

That would be great. Should we only allow user to specify directory and nothing else? Then store both test and coverage reports there. When merging the reports, we would respect reporters' outputFile, reportsDirectory etc. flags as usual.

I mean we can still allow specifying the blob name, why not? In the end it should output a single file

@AriPerkkio
Copy link
Member Author

AriPerkkio commented May 18, 2024

In the end it should output a single file

Does this mean that coverage reports should be included in the same file as test results?

@sheremet-va
Copy link
Member

sheremet-va commented May 18, 2024

In the end it should output a single file

Does this mean that coverage reports should be included in the same file as test results?

Yes, I would expect everything to be inside that blob

@AriPerkkio AriPerkkio changed the title feat: --merge-reports to support coverage feat!: --merge-reports to support coverage May 19, 2024
@AriPerkkio AriPerkkio force-pushed the feat/merge-reports-coverage branch 7 times, most recently from 73fb841 to dad15a1 Compare May 20, 2024 17:59
@AriPerkkio AriPerkkio force-pushed the feat/merge-reports-coverage branch from dad15a1 to 626edee Compare May 21, 2024 14:18
AriPerkkio
AriPerkkio commented May 21, 2024
View reviewed changes
@AriPerkkio AriPerkkio force-pushed the feat/merge-reports-coverage branch from 626edee to 0d2f152 Compare May 22, 2024 13:20
@AriPerkkio AriPerkkio force-pushed the feat/merge-reports-coverage branch from 0d2f152 to 34d3706 Compare May 22, 2024 13:25
@AriPerkkio AriPerkkio marked this pull request as ready for review May 22, 2024 13:47
sheremet-va
sheremet-va reviewed May 22, 2024
View reviewed changes
This was referenced Aug 1, 2024
Open
Open
Open
Open
Open
This was referenced Aug 12, 2024
Open
Open
Open
Open
This was referenced Aug 21, 2024
Open
Open
Open
Open
This was referenced Aug 28, 2024
Open
Open
This was referenced Sep 10, 2024
Merged
Merged
This was referenced Sep 23, 2024
Open
Open
Open
Open
@gr00nd gr00nd mentioned this pull request Sep 23, 2024
Open
@leonardoadame leonardoadame mentioned this pull request Sep 28, 2024
Open
This was referenced Oct 28, 2024
Merged
Open
Open
Merged
Closed
Merged
@renovate renovate bot mentioned this pull request Nov 11, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hi-ogawa hi-ogawa hi-ogawa left review comments

@sheremet-va sheremet-va sheremet-va approved these changes

Assignees

No one assigned

Labels

breaking change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Improve support for parallel runs and coverage report merging when --shard option is used [Docs] Explain how to use sharding

3 participants

@AriPerkkio @sheremet-va @hi-ogawa