feat: allow custom pools by sheremet-va · Pull Request #4417 · vitest-dev/vitest

sheremet-va · 2023-10-31T22:35:56Z

Description

This PR adds a possibility to specify a filepath or package name in pool option. Vitest expects that specified module will have a default export which is a function that accepts Vitest as its parameter. This function should return on object that satisfies this interface:

import { ProcessPool, WorkspaceProject } from 'vitest/node'

export interface ProcessPool {
  name: string
  runTests: (files: [project: WorkspaceProject, testFile: string][], invalidates?: string[]) => Promise<void>
  close?: () => Promise<void>
}

runTests is called when user runs vitest command and then each time watch mode is triggered. This API is primarily designed to be used by library authors because it requires deep knowledge of how Vitest works internally. More about this option can be found in documentation.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
Ideally, include a test that fails without this PR but passes with it.
Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

Run the tests with pnpm test:ci.

Documentation

If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

bolt-new-by-stackblitz · 2023-10-31T22:35:59Z

Run & review this pull request in StackBlitz Codeflow.

netlify · 2023-10-31T22:36:01Z

✅ Deploy Preview for fastidious-cascaron-4ded94 canceled.

Name	Link
🔨 Latest commit	`1ae44c9`
🔍 Latest deploy log	https://app.netlify.com/sites/fastidious-cascaron-4ded94/deploys/65588376049e19000801d813

docs/advanced/pool.md

sheremet-va · 2023-11-02T12:32:37Z

Note: this PR just opens up a possibility to do this, we still need to improve how to run tests in the new runtime - like exposing the logic of runtime/{worker,child,vmThreads}.ts. But I don't think we can do this without exposing this option first.

AriPerkkio

Tested a custom pool in locally linked project and seems to work nice.

AriPerkkio · 2023-11-02T14:16:36Z

docs/advanced/pool.md

+
+## API
+
+The file specified in `pool` option should export a function (can be async) that accepts `Vitest` interface as its first option. This functions needs to return an object matching `ProcessPool` interface:


Suggested change

The file specified in `pool` option should export a function (can be async) that accepts `Vitest` interface as its first option. This functions needs to return an object matching `ProcessPool` interface:

The file specified in `pool` option should export a function (can be async) that accepts `Vitest` interface as its first option. This function needs to return an object matching `ProcessPool` interface:

AriPerkkio · 2023-11-02T14:20:32Z

docs/advanced/pool.md

+
+Vitest will wait until `runTests` is executed before finishing a run (i.e., it will emit [`onFinished`](/guide/reporters) only after `runTests` is resolved).
+
+If you are using a custom pool, you will have to provide test files and their results yourself - you can reference [`vitest.state`](https://github.com/vitest-dev/vitest/blob/feat/custom-pool/packages/vitest/src/node/state.ts) for that (mose important are `collectFiles` and `updateTasks`). Vitest uses `startTests` function from `@vitest/runner` package to do that.


Suggested change

If you are using a custom pool, you will have to provide test files and their results yourself - you can reference [`vitest.state`](https://github.com/vitest-dev/vitest/blob/feat/custom-pool/packages/vitest/src/node/state.ts) for that (mose important are `collectFiles` and `updateTasks`). Vitest uses `startTests` function from `@vitest/runner` package to do that.

If you are using a custom pool, you will have to provide test files and their results yourself - you can reference [`vitest.state`](https://github.com/vitest-dev/vitest/blob/feat/custom-pool/packages/vitest/src/node/state.ts) for that (most important are `collectFiles` and `updateTasks`). Vitest uses `startTests` function from `@vitest/runner` package to do that.

And the URL should eventually point at main instead of work branch.

AriPerkkio · 2023-11-02T14:36:24Z

packages/vitest/src/node/pool.ts

+      return sequencer.sort(specs)
+    }
+
+    await Promise.all(Object.entries(filesByPool).map(async (entry) => {


This issue seems to be in main branch too, but all pools are executed parallel here, right?

If user has threads pool and vmThreads pools set, and each are using default max-threads, the execution will spawn too many workers at the same time.

Yes, this is how it works. That's why I wanted agnostic implementation of tinypool.

Tinypool can run worker_threads and child_process parallel.

https://github.com/tinylibs/tinypool/blob/999778ed8acccf18303760558f807f0e208c8b3b/test/runtime.test.ts#L151-L170

It's possible to push all threads tasks in the pool, then wait for queue to be empty (remaining tasks are running, nothing in queue), and then change the runtime and push more tasks in the queue to wait for previous ones to finish. Like here is done:

vitest/packages/vitest/src/node/pools/threads.ts

Lines 171 to 174 in ab55637

// Once all tasks are running or finished, recycle worker for isolation.

// On-going workers will run in the previous environment.

await new Promise<void>(resolve => pool.queueSize === 0 ? resolve() : pool.once('drain', resolve))

await pool.recycleWorkers()

Ideally, I would like an API that we can expose to reuse the same Tinypool instance in different pools:

function createThreadsPool(ctx, { pool }) { function runTests(testFile) { const context = { config: ctx.config, files: [testFile], executeFile: resolve('./dist/worker.js') } await pool.run(context, { runtime: 'worker_threads', name: 'run' }) } }

Maybe instead of having separate worker.js/child.js/vmWorker.js files we have a single file where inside of a context we pass down information on how we want to run it? So, we don't need to change filename option in Tinypool. This should also make it easier for external pools to reuse it

I like the idea. From Tinypool's side I think that kind of API should already be possible - some kind of wrappers around it might be needed though.
There's also some need for refactoring in threads and forks pools to reduce code duplication. While doing that it might be good to look into this kind of API. But all this can be left out of this PR as it's not relevant to these changes.

docs/advanced/pool.md

AriPerkkio reviewed Nov 1, 2023

View reviewed changes

docs/advanced/pool.md Outdated Show resolved Hide resolved

AriPerkkio reviewed Nov 1, 2023

View reviewed changes

docs/advanced/pool.md Show resolved Hide resolved

sheremet-va marked this pull request as ready for review November 2, 2023 11:50

sheremet-va requested a review from AriPerkkio November 2, 2023 11:50

AriPerkkio approved these changes Nov 2, 2023

View reviewed changes

sheremet-va added 13 commits November 18, 2023 09:47

feat: allow custom pools

7b618cc

chore: fix types

3bca509

docs: add custom pool entry to docs

36aa1a5

chore: add custom-pool test

6ffabd3

chore: typo

e1fe2b4

chore: fix pool test

02373bd

chore: cleanup

719b634

chore: cleanup

1db5190

chore: fix types

e20059a

chore: cleanup

d60db58

chore: cleanup

49d3afe

chore: fix windows test

bc92eef

chore: fix docs

fba232a

sheremet-va force-pushed the feat/custom-pool branch from fe67eee to fba232a Compare November 18, 2023 08:48

sheremet-va added 2 commits November 18, 2023 09:55

fix: call .close on pool

0e0f517

chore: normalize test path

1ae44c9

sheremet-va merged commit a3fd5f8 into vitest-dev:main Nov 18, 2023

sheremet-va deleted the feat/custom-pool branch November 18, 2023 09:42

This was referenced Nov 27, 2023

apply sequencer to browser tests #4444

Closed

Support shard in browser mode #4443

Closed

mrbbot mentioned this pull request Jan 19, 2024

Implement Vitest testing environment for Miniflare 3 cloudflare/workers-sdk#4795

Merged

6 tasks


		## API

		The file specified in `pool` option should export a function (can be async) that accepts `Vitest` interface as its first option. This functions needs to return an object matching `ProcessPool` interface:


		Vitest will wait until `runTests` is executed before finishing a run (i.e., it will emit [`onFinished`](/guide/reporters) only after `runTests` is resolved).

		If you are using a custom pool, you will have to provide test files and their results yourself - you can reference [`vitest.state`](https://github.com/vitest-dev/vitest/blob/feat/custom-pool/packages/vitest/src/node/state.ts) for that (mose important are `collectFiles` and `updateTasks`). Vitest uses `startTests` function from `@vitest/runner` package to do that.

	// Once all tasks are running or finished, recycle worker for isolation.
	// On-going workers will run in the previous environment.
	await new Promise<void>(resolve => pool.queueSize === 0 ? resolve() : pool.once('drain', resolve))
	await pool.recycleWorkers()

Uh oh!

Conversation

sheremet-va commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

Tests

Documentation

Changesets

Uh oh!

bolt-new-by-stackblitz bot commented Oct 31, 2023

Uh oh!

netlify bot commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for fastidious-cascaron-4ded94 canceled.

Uh oh!

Uh oh!

Uh oh!

sheremet-va commented Nov 2, 2023

Uh oh!

AriPerkkio left a comment

Choose a reason for hiding this comment

Uh oh!

AriPerkkio Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

AriPerkkio Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

AriPerkkio Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

sheremet-va Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

AriPerkkio Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

sheremet-va Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AriPerkkio Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sheremet-va commented Oct 31, 2023 •

edited

Loading

netlify bot commented Oct 31, 2023 •

edited

Loading

sheremet-va Nov 2, 2023 •

edited

Loading

AriPerkkio Nov 2, 2023 •

edited

Loading