Add support for running specific test suites from the CLI with "npm run test" #5607

DanTup · 2025-07-25T15:11:20Z

and some tidying up of the test output to make it less noisy.

gemini-code-assist

Summary of Changes

Hello @DanTup, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the developer experience for running tests by introducing the ability to specify individual test files or globs from the command line. It also includes various improvements to reduce noise in the test output and refactors the underlying test execution scripts for better maintainability and clarity.

Highlights

Test Filtering via CLI: I've added support for running specific test suites directly from the command line using npm run test <...test-file-globs>. This allows developers to quickly run a subset of tests relevant to their current work, significantly speeding up the development cycle.
Improved Test Output & Experience: The test output has been tidied up to be less noisy. This includes suppressing verbose download logs for VS Code test environments and removing redundant console messages during test execution. Additionally, the npm scripts for testing have been modularized into pretest, test, and posttest phases for better organization and clarity.
Refactored Test Runner Logic: The core test runner logic in src/test/test_all.ts and src/test/test_runner.ts has been updated to handle test filtering. A new getTestSuites utility function was introduced to discover and filter test files based on the provided command-line arguments, ensuring only relevant tests are executed.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

Copilot

Pull Request Overview

This PR adds support for running specific test suites from the CLI using npm run test with optional test file filters, while also tidying up test output to reduce noise.

Introduces test filtering capability through command-line arguments and environment variables
Refactors test runner to extract file discovery logic into a reusable function
Modifies package.json scripts to improve test workflow and reduce build output verbosity

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
src/test/test_runner.ts	Adds `getTestSuites` function with filtering logic and integrates filter support into the main test runner
src/test/test_all.ts	Updates to use new filtering functionality and reduces console output noise
package.json	Restructures test scripts to support filtered testing and quieter build output
AGENTS.md	Updates documentation to reflect new test command usage patterns

Comments suppressed due to low confidence (1)

src/test/test_all.ts:41

[nitpick] The empty arrow function should have a more descriptive name or comment explaining why it's intentionally empty.

		report: () => { },

Copilot · 2025-07-25T15:12:24Z

src/test/test_runner.ts

+	// If there are filters, return those that match any of them.
+	const files = new Set<string>();
+	for (let filter of filters.map(normalizeTestFilter)) {
+		filter = `**/*${filter}*`;


The filter pattern construction could be more robust. Consider handling edge cases where the filter might already contain wildcards or be an absolute path.

Suggested change

filter = `**/*${filter}*`;

// Check if the filter contains wildcards or is an absolute path.

const hasWildcards = /[*?]/.test(filter);

const isAbsolutePath = path.isAbsolute(filter);

// Construct the filter pattern based on its type.

if (!hasWildcards && !isAbsolutePath) {

filter = `**/*${filter}*`;

}

// Apply the filter to the list of files.

Copilot · 2025-07-25T15:12:25Z

src/test/test_all.ts

 		try {
-			console.log(`Attempting to download VS Code attempt #${currentAttempt}`);
-			await vstest.downloadAndUnzipVSCode(codeVersion);
+			// console.log(`Attempting to download VS Code attempt #${currentAttempt}`);


Consider removing commented-out code rather than leaving it in place. If this logging might be needed for debugging, consider using a debug flag or logger level instead.

Suggested change

// console.log(`Attempting to download VS Code attempt #${currentAttempt}`);

if (process.env.DEBUG) {

console.log(`Attempting to download VS Code attempt #${currentAttempt}`);

}

Copilot · 2025-07-25T15:12:25Z

AGENTS.md

 - `npm run build`: Builds the extension.
- `npm test`: Runs all automatic tests. This should first be run with the `BOT` env variable set to `"dart"` to run only a subset of basic Dart tests. If all of those tests, the whole test suite should be run.
+- `npm run test <...test-file-globs>`: Runs tests for the given glob (or file path). This should be used while working on an individual feature because it is faster than running all tests.
+- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is larged and this can be slow.


There's a typo: 'larged' should be 'large'. Also consider adding 'is' before 'large': 'The test suite is large and this can be slow.'

Suggested change

- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is larged and this can be slow.

- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is large and this can be slow.

gemini-code-assist

Code Review

This pull request introduces a feature for running specific test suites from the CLI, improving developer workflow. The changes to the npm scripts using pretest and posttest are a good improvement. The efforts to reduce noise in the test output are also appreciated. I've left suggestions to improve the documentation in AGENTS.md and add robustness to the test runner by handling potential JSON parsing errors.

gemini-code-assist · 2025-07-25T15:12:56Z

AGENTS.md

+- `npm run test <...test-file-globs>`: Runs tests for the given glob (or file path). This should be used while working on an individual feature because it is faster than running all tests.
+- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is larged and this can be slow.


The documentation for npm run test is updated, but the term glob might be misleading[^1]. The implementation performs a substring match, not a full glob match. Also, there's a typo: larged should be large. Consider the following suggestion.

Suggested change

- `npm run test <...test-file-globs>`: Runs tests for the given glob (or file path). This should be used while working on an individual feature because it is faster than running all tests.

- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is larged and this can be slow.

- `npm run test <...test-filters>`: Runs tests for file paths containing any of the given filter strings. This should be used while working on an individual feature because it is faster than running all tests.

- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is large and this can be slow.

gemini-code-assist · 2025-07-25T15:12:56Z

src/test/test_runner.ts

+				const testFilter = process.env.DART_CODE_TEST_FILTER;
+				const filters: string[] | undefined = testFilter ? JSON.parse(testFilter) : undefined;


The JSON.parse call on testFilter could throw an exception if the environment variable contains malformed JSON. Add a try...catch block to handle potential parsing errors[^1].

const testFilter = process.env.DART_CODE_TEST_FILTER; let filters: string[] | undefined; if (testFilter) { try { filters = JSON.parse(testFilter); } catch (e) { return reject(new Error(`Failed to parse DART_CODE_TEST_FILTER environment variable: ${e}`)); } }

…un test" + some tidying up of the test output to make it less noisy.

DanTup added this to the v3.116.0 milestone Jul 25, 2025

DanTup added is enhancement is testing Relates to the testing (either manual or automated) of Dart Code labels Jul 25, 2025

DanTup requested a review from Copilot July 25, 2025 15:11

gemini-code-assist bot reviewed Jul 25, 2025

View reviewed changes

Copilot AI reviewed Jul 25, 2025

View reviewed changes

gemini-code-assist bot reviewed Jul 25, 2025

View reviewed changes

Add support for running specific test suites from the CLI with "npm r…

b1011fe

…un test" + some tidying up of the test output to make it less noisy.

DanTup force-pushed the better-test-running branch from 665faf7 to b1011fe Compare July 25, 2025 15:39

DanTup removed the is enhancement label Jul 25, 2025

DanTup merged commit 00483ee into master Jul 27, 2025
21 checks passed

DanTup deleted the better-test-running branch July 27, 2025 12:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for running specific test suites from the CLI with "npm run test" #5607

Add support for running specific test suites from the CLI with "npm run test" #5607

Uh oh!

DanTup commented Jul 25, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 25, 2025

Uh oh!

Copilot AI Jul 25, 2025

Uh oh!

Copilot AI Jul 25, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jul 25, 2025

Uh oh!

gemini-code-assist bot Jul 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-		filter = `**/*${filter}*`;
+		// Check if the filter contains wildcards or is an absolute path.
+		const hasWildcards = /[*?]/.test(filter);
+		const isAbsolutePath = path.isAbsolute(filter);
+		// Construct the filter pattern based on its type.
+		if (!hasWildcards && !isAbsolutePath) {
+			filter = `**/*${filter}*`;
+		}
+		// Apply the filter to the list of files.

	- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is larged and this can be slow.
	- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is large and this can be slow.

		- `npm run test <...test-file-globs>`: Runs tests for the given glob (or file path). This should be used while working on an individual feature because it is faster than running all tests.
		- `npm run test`: Runs _all_ automatic tests for _all_ bots. This can be run with the `BOT` env variable set to (as defined in `test_all.ts`, for example to `"dart"`) to run only a single bots tests. The test suite is larged and this can be slow.

		const testFilter = process.env.DART_CODE_TEST_FILTER;
		const filters: string[] \| undefined = testFilter ? JSON.parse(testFilter) : undefined;

Add support for running specific test suites from the CLI with "npm run test" #5607

Add support for running specific test suites from the CLI with "npm run test" #5607

Uh oh!

Conversation

DanTup commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DanTup commented Jul 25, 2025 •

edited

Loading