matrix for evals by brookemosby · Pull Request #15247 · vercel/vercel

brookemosby · 2026-02-24T22:21:25Z

Note

Low Risk Change

This PR adds setup/destroy hooks and matrix support for CLI evals, modifying only test infrastructure files (workflow config, eval runner, test hooks) with no changes to production code, auth logic, or database schemas.

CI workflow: increased timeout and added NODE_OPTIONS for memory limits
New hooks.ts/hooks.test.ts for eval setup/teardown with ephemeral project creation
Refactored run.ts to support project-mode variants and improved error handling

^{Risk assessment for commit 8886453.}

changeset-bot · 2026-02-24T22:21:31Z

🦋 Changeset detected

Latest commit: 8886453

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
vercel	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-02-24T22:21:45Z

📦 CLI Tarball Ready

The Vercel CLI tarball for this PR is now available!

Quick Test

You can test this PR's CLI directly by running:

npx https://vercel-9rffbf3yw.vercel.sh/tarballs/vercel.tgz --help

Use in vercel.json

To use this CLI version in your project builds, add to your vercel.json:

{
  "build": {
    "env": {
      "VERCEL_CLI_VERSION": "vercel@https://vercel-9rffbf3yw.vercel.sh/tarballs/vercel.tgz"
    }
  }
}

Python Runtime Wheel

A vercel-runtime wheel was also built for this PR.
To use in your Python project builds, also set this environment variable:

VERCEL_RUNTIME_PYTHON="vercel-runtime @ https://vercel-9rffbf3yw.vercel.sh/tarballs/vercel_runtime-0.6.0.dev1772055350+8886453-py3-none-any.whl"

Python Workers Wheel

A vercel-workers wheel was also built for this PR.
To use in your Python project builds, also set this environment variable:

VERCEL_WORKERS_PYTHON="vercel-workers @ https://vercel-9rffbf3yw.vercel.sh/tarballs/vercel_workers-0.1.0.dev1772055350+8886453-py3-none-any.whl"

github-actions · 2026-02-24T22:22:57Z

🧪 Unit Test Strategy

Comparing: 0ca7895 → 8886453 (view diff)

Strategy: Code changed outside of a package - running all unit tests

⚠️ All unit tests will run because global code changes could impact all packages.

Affected packages - 15 (38%)

vercel
@vercel/build-utils
@vercel/client
@vercel/firewall
@vercel/fs-detectors
@vercel/go
@vercel/hydrogen
@vercel/next
@vercel/node
@vercel/python
@vercel/remix-builder
@vercel/ruby
@vercel/rust
@vercel/static-build
examples

Unaffected packages - 25 (63%)

@vercel-internals/get-package-json
@vercel/backends
@vercel/cervel
@vercel/cli-auth
@vercel/config
@vercel/detect-agent
@vercel/edge
@vercel/elysia
@vercel/error-utils
@vercel/express
@vercel/fastify
@vercel/frameworks
@vercel/functions
@vercel/gatsby-plugin-vercel-builder
@vercel/h3
@vercel/hono
@vercel/koa
@vercel/nestjs
@vercel/oidc
@vercel/oidc-aws-credentials-provider
@vercel/python-analysis
@vercel/redwood
@vercel/related-projects
@vercel/routing-utils
@vercel/static-config

Results

Unit tests: All affected packages will run unit tests
E2E tests: Running in parallel via E2E Tests workflow
Type checks: All affected packages will run type checks

This comment is automatically generated based on the affected testing strategy

packages/cli/evals/hooks.ts

packages/cli/evals/experiments/cli.ts

packages/cli/evals/setup/auth-and-config.ts

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @vercel/routing-utils@6.0.0 ### Major Changes - Allow `routes` to coexist with `rewrites`, `headers`, `redirects`, `cleanUrls`, and `trailingSlash`. ([#15020](#15020)) This is part of our efforts to undeprecate `routes`, which had previously been deprecated for a few years and replaced by the properties above. Now that we are undeprecating `routes` in favor of a more focused deprecation of properties within it, it is now allowed to coexist with the new properties. When used together, `routes` are inserted after `cleanUrls` and `trailingSlash` but before `redirects`, `headers`, and `rewrites` in the routing order. - Deprecate `handle`, `important`, and `override` properties in `routes` schema ([#15015](#15015)) ## @vercel/build-utils@13.6.0 ### Minor Changes - Find entrypoints for django projects. ([#15167](#15167)) ### Patch Changes - Rename fetch to nodeFetch in cases where it is an import from node-fetch ([#15234](#15234)) - [@vercel/build-utils] fix uncaught exception in streamToBuffer when stream exceeds max Buffer size ([#15276](#15276)) - Remove source and destination typo suggestions for routes schema ([#15014](#15014)) - Support function overrides in backends builder ([#15214](#15214)) - Updated dependencies \[[`b3a96cc4f276ce8d16c695eabd499d3a17e73aa8`](b3a96cc)]: - @vercel/python-analysis@0.8.0 ## vercel@50.25.0 ### Minor Changes - Launch auto-provision flow as default for `vercel integration add` and `vercel install`. The `FF_AUTO_PROVISION_INSTALL` flag is now a kill-switch (`=0` to revert) instead of opt-in. ([#15265](#15265)) - [services] make the `experimentalServices` config the primary marker for the services flow ([#15246](#15246)) - Added command to fetch a development OIDC token ([#14785](#14785)) ### Patch Changes - Rename fetch to nodeFetch when importing from node-fetch ([#15232](#15232)) - Add setup/destroy hooks and context handling to the CLI eval runner. ([#15247](#15247)) - [services] fix dev server hang when FastAPI CLI is used ([#15274](#15274)) - Fix `--help` flag to return exit code 0 instead of 2 for the `metrics` command, aligning with standard CLI conventions. ([#15259](#15259)) - Rename fetch to nodeFetch in cases where it is an import from node-fetch ([#15234](#15234)) - Change `vercel metrics` default query output from CSV to rich text with metadata, summary tables, sparklines, and optional raw values via `--raw-values`. ([#15178](#15178)) - Removes the legacy SSO redirect flow code. ([#15228](#15228)) - Updated dependencies \[[`ad86425c3e0a1d1738cef8464dd767dd1d9a0e9a`](ad86425), [`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269), [`d9e3a570a9c6ee47afa3446ec611b6a9e4b4feec`](d9e3a57), [`2db57840669ca2fa8dadf492bd38c21bf9e3cfeb`](2db5784), [`0d42acfd22e51e51cb0e4a5aea7c19f3886ed496`](0d42acf), [`8404a7315260851495d708f1c7c6e870d7ffa007`](8404a73), [`fbf649223e6fa8b509295a85784347dcebd4a0b0`](fbf6492), [`b3a96cc4f276ce8d16c695eabd499d3a17e73aa8`](b3a96cc), [`9a6358529f0737415110cf03d626dc9ee8e99126`](9a63585)]: - @vercel/remix-builder@5.6.0 - @vercel/python@6.17.0 - @vercel/next@4.15.35 - @vercel/node@5.6.8 - @vercel/static-build@2.8.42 - @vercel/build-utils@13.6.0 - @vercel/go@3.4.3 - @vercel/backends@0.0.38 - @vercel/elysia@0.1.41 - @vercel/express@0.1.50 - @vercel/fastify@0.1.44 - @vercel/h3@0.1.50 - @vercel/hono@0.2.44 - @vercel/koa@0.1.24 - @vercel/nestjs@0.2.45 - @vercel/redwood@2.4.9 - @vercel/rust@1.0.5 - @vercel/hydrogen@1.3.5 - @vercel/ruby@2.3.2 ## @vercel/python@6.17.0 ### Minor Changes - Find entrypoints for django projects. ([#15167](#15167)) ### Patch Changes - Rename fetch to nodeFetch when importing from node-fetch ([#15232](#15232)) - [services] fix dev server hang when FastAPI CLI is used ([#15274](#15274)) - [python] fix dev server crash on relative imports ([#15269](#15269)) - Updated dependencies \[[`b3a96cc4f276ce8d16c695eabd499d3a17e73aa8`](b3a96cc)]: - @vercel/python-analysis@0.8.0 ## @vercel/python-analysis@0.8.0 ### Minor Changes - Find entrypoints for django projects. ([#15167](#15167)) ## @vercel/remix-builder@5.6.0 ### Minor Changes - [remix] Support future.v8_middleware option ([#15189](#15189)) ## @vercel/backends@0.0.38 ### Patch Changes - Support function overrides in backends builder ([#15214](#15214)) - Updated dependencies \[[`2db57840669ca2fa8dadf492bd38c21bf9e3cfeb`](2db5784), [`0d42acfd22e51e51cb0e4a5aea7c19f3886ed496`](0d42acf), [`8404a7315260851495d708f1c7c6e870d7ffa007`](8404a73), [`b3a96cc4f276ce8d16c695eabd499d3a17e73aa8`](b3a96cc), [`9a6358529f0737415110cf03d626dc9ee8e99126`](9a63585)]: - @vercel/build-utils@13.6.0 ## @vercel/cervel@0.0.25 ### Patch Changes - Updated dependencies \[[`9a6358529f0737415110cf03d626dc9ee8e99126`](9a63585)]: - @vercel/backends@0.0.38 ## @vercel/client@17.2.48 ### Patch Changes - rename fetch to fetchApi ([#15231](#15231)) - Updated dependencies \[[`e64a85daec784b4fe571abe7405a80aca150007a`](e64a85d), [`2db57840669ca2fa8dadf492bd38c21bf9e3cfeb`](2db5784), [`0d42acfd22e51e51cb0e4a5aea7c19f3886ed496`](0d42acf), [`8404a7315260851495d708f1c7c6e870d7ffa007`](8404a73), [`b3a96cc4f276ce8d16c695eabd499d3a17e73aa8`](b3a96cc), [`9a6358529f0737415110cf03d626dc9ee8e99126`](9a63585), [`af6a651229cad4ddb7fcc15bd14d06355c2b77ee`](af6a651)]: - @vercel/routing-utils@6.0.0 - @vercel/build-utils@13.6.0 ## @vercel/config@0.0.35 ### Patch Changes - Updated dependencies \[[`e64a85daec784b4fe571abe7405a80aca150007a`](e64a85d), [`af6a651229cad4ddb7fcc15bd14d06355c2b77ee`](af6a651)]: - @vercel/routing-utils@6.0.0 ## @vercel/elysia@0.1.41 ### Patch Changes - Updated dependencies \[[`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269)]: - @vercel/node@5.6.8 ## @vercel/express@0.1.50 ### Patch Changes - Updated dependencies \[[`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269)]: - @vercel/node@5.6.8 - @vercel/cervel@0.0.25 ## @vercel/fastify@0.1.44 ### Patch Changes - Updated dependencies \[[`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269)]: - @vercel/node@5.6.8 ## @vercel/frameworks@3.19.1 ### Patch Changes - Rename fetch to nodeFetch when importing from node-fetch ([#15232](#15232)) ## @vercel/fs-detectors@5.8.12 ### Patch Changes - Support function overrides in backends builder ([#15214](#15214)) - Updated dependencies \[[`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269), [`e64a85daec784b4fe571abe7405a80aca150007a`](e64a85d), [`af6a651229cad4ddb7fcc15bd14d06355c2b77ee`](af6a651)]: - @vercel/frameworks@3.19.1 - @vercel/routing-utils@6.0.0 ## @vercel/gatsby-plugin-vercel-builder@2.0.140 ### Patch Changes - Updated dependencies \[[`2db57840669ca2fa8dadf492bd38c21bf9e3cfeb`](2db5784), [`0d42acfd22e51e51cb0e4a5aea7c19f3886ed496`](0d42acf), [`8404a7315260851495d708f1c7c6e870d7ffa007`](8404a73), [`b3a96cc4f276ce8d16c695eabd499d3a17e73aa8`](b3a96cc), [`9a6358529f0737415110cf03d626dc9ee8e99126`](9a63585)]: - @vercel/build-utils@13.6.0 ## @vercel/go@3.4.3 ### Patch Changes - Rename fetch to nodeFetch in cases where it is an import from node-fetch ([#15234](#15234)) ## @vercel/h3@0.1.50 ### Patch Changes - Updated dependencies \[[`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269)]: - @vercel/node@5.6.8 ## @vercel/hono@0.2.44 ### Patch Changes - Updated dependencies \[[`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269)]: - @vercel/node@5.6.8 ## @vercel/koa@0.1.24 ### Patch Changes - Updated dependencies \[[`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269)]: - @vercel/node@5.6.8 ## @vercel/nestjs@0.2.45 ### Patch Changes - Updated dependencies \[[`3d872692c601abeec98743d2af3a3ea19587e799`](3d87269)]: - @vercel/node@5.6.8 ## @vercel/next@4.15.35 ### Patch Changes - Rename fetch to nodeFetch when importing from node-fetch ([#15232](#15232)) ## @vercel/node@5.6.8 ### Patch Changes - Rename fetch to nodeFetch when importing from node-fetch ([#15232](#15232)) - Updated dependencies \[[`2db57840669ca2fa8dadf492bd38c21bf9e3cfeb`](2db5784), [`0d42acfd22e51e51cb0e4a5aea7c19f3886ed496`](0d42acf), [`8404a7315260851495d708f1c7c6e870d7ffa007`](8404a73), [`b3a96cc4f276ce8d16c695eabd499d3a17e73aa8`](b3a96cc), [`9a6358529f0737415110cf03d626dc9ee8e99126`](9a63585)]: - @vercel/build-utils@13.6.0 ## @vercel/static-build@2.8.42 ### Patch Changes - Rename fetch to nodeFetch in cases where it is an import from node-fetch ([#15234](#15234)) - Updated dependencies \[]: - @vercel/gatsby-plugin-vercel-builder@2.0.140 ## @vercel/python-runtime@0.5.2 ### Patch Changes - fix ASGI lifecycle events in non-IPC codepath ([#15268](#15268)) ## @vercel/python-workers@0.0.11 ### Patch Changes - Add a version bump for `@vercel/python-workers` so previously merged changes are included in the next release. ([#15254](#15254))  > [!NOTE] > Low Risk Change > > This PR is an automated Changesets release that only updates version numbers in package.json files and CHANGELOG.md files across multiple packages, with no logic or schema changes. > > - Version bumps across ~30 packages (package.json version fields) > - CHANGELOG.md updates documenting previously merged changes > - Deletion of .changeset/*.md files consumed by release automation > > <sup>Risk assessment for [commit 9b78e59](https://github.com/vercel/vercel/commit/9b78e59c7c4716d10beac3a0228e767c8565f65a).</sup>  Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

matrix for tests

ac55965

brookemosby requested review from a team as code owners February 24, 2026 22:21

brookemosby marked this pull request as draft February 24, 2026 22:21

brookemosby changed the title ~~matrix for tests~~ matrix for evals Feb 24, 2026

vercel bot reviewed Feb 24, 2026

View reviewed changes

packages/cli/evals/hooks.ts Outdated Show resolved Hide resolved

matrix for tests

afc9936

vercel bot deployed to Preview February 24, 2026 22:27 View deployment

matrix for tests

9279269

vercel bot deployed to Preview February 24, 2026 22:33 View deployment

matrix for tests

b22e0a5

vercel bot deployed to Preview February 24, 2026 22:57 View deployment

matrix for tests

1441cf7

vercel bot deployed to Preview February 24, 2026 23:09 View deployment

matrix for tests

72f798e

vercel bot deployed to Preview February 24, 2026 23:25 View deployment

matrix for tests

52882c1

vercel bot deployed to Preview February 24, 2026 23:42 View deployment

cleanup

1c924dd

vercel bot deployed to Preview February 25, 2026 16:14 View deployment

cleanup

ede9e6c

vercel bot deployed to Preview February 25, 2026 16:17 View deployment

cleanup

753f2e0

vercel bot deployed to Preview February 25, 2026 16:18 View deployment

Merge branch 'main' into bm/evalsMatrix

5a393f0

vercel bot deployed to Preview February 25, 2026 16:19 View deployment

cleanup

79cbabd

vercel bot deployed to Preview February 25, 2026 16:29 View deployment

cleanup

5676137

vercel bot deployed to Preview February 25, 2026 18:02 View deployment

Merge branch 'main' into bm/evalsMatrix

431e15d

vercel bot deployed to Preview February 25, 2026 18:04 View deployment

cleanup

4238de0

vercel bot deployed to Preview February 25, 2026 18:38 View deployment

Merge branch 'main' into bm/evalsMatrix

180ea78

vercel bot deployed to Preview February 25, 2026 19:38 View deployment

cleanup

1082a83

vercel bot deployed to Preview February 25, 2026 20:22 View deployment

brookemosby marked this pull request as ready for review February 25, 2026 20:33

brookemosby requested a review from ijjk as a code owner February 25, 2026 20:33

jeffsee55 approved these changes Feb 25, 2026

View reviewed changes

packages/cli/evals/experiments/cli.ts Outdated Show resolved Hide resolved

packages/cli/evals/setup/auth-and-config.ts Show resolved Hide resolved

packages/cli/evals/setup/auth-and-config.ts Show resolved Hide resolved

cleanup

c1436ba

vercel bot deployed to Preview February 25, 2026 21:34 View deployment

Merge branch 'main' into bm/evalsMatrix

8886453

vercel bot deployed to Preview February 25, 2026 21:36 View deployment

brookemosby enabled auto-merge (squash) February 25, 2026 21:40

brookemosby merged commit c37d6fd into main Feb 25, 2026
190 checks passed

brookemosby deleted the bm/evalsMatrix branch February 25, 2026 21:55

vercel-release-bot mentioned this pull request Feb 25, 2026

Version Packages #15253

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

matrix for evals#15247

matrix for evals#15247
brookemosby merged 19 commits intomainfrom
bm/evalsMatrix

brookemosby commented Feb 24, 2026 •

edited by vercel bot

Loading

Uh oh!

changeset-bot bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

brookemosby commented Feb 24, 2026 • edited by vercel bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 CLI Tarball Ready

Quick Test

Use in vercel.json

Python Runtime Wheel

Python Workers Wheel

Uh oh!

github-actions bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 Unit Test Strategy

Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

brookemosby commented Feb 24, 2026 •

edited by vercel bot

Loading

changeset-bot bot commented Feb 24, 2026 •

edited

Loading

github-actions bot commented Feb 24, 2026 •

edited

Loading

github-actions bot commented Feb 24, 2026 •

edited

Loading