feat(linter): Add test to enforce schema documentation for rules with configuration options by connorshea · Pull Request #15234 · oxc-project/oxc

connorshea · 2025-11-02T20:53:15Z

This was implemented with considerable help from GitHub Copilot using Claude Sonnet 4.5, and advice from camc in the discord.

It should be noted that this technically also does not enforce that the config options be documented, it currently only checks that there are config options in the debug output for the Rule's emitted default value. We could potentially evaluate the emitted schema more deeply to ensure all options have documentation info as well, but that would require more work and should maybe be a separate test.

This test enforces the config schema usage by taking the following steps:

Get all rules in the repo
Establish an 'exceptions' list for rules we have not converted yet
Go through each rule, for each:
- If the rule is in the exceptions list, make sure it has no schema (if it does, fail) and then skip to the next rule.
- Check if the rule has a schema, and skip to the next rule if it does.
- Generate the debug output for the rule's default shape, and then check if it has characters which would suggest it has config options (any of {}[] or more than one layer of nested ()).
Return failures at the end.

It also ensures that all rules in the exceptions array are defined rules.

Note that react/state-in-constructor and vue/define-emits-declaration are missed by this, they should error if removed from exceptions, but do not get caught as rules that have configuration options. This is because they are using enum-only configs (e.g. "plugin/rule": ["error", "never"]), and so don't emit a debug string with {} or [] in them. Currently, if you remove the exceptions array entirely, this will find 27 of the 29 known violations in the codebase. So it's probably sufficient for now. I fixed this problem, so it now discovers 29/29 violations.

The output for a failure looks like this:

running 1 test
test test_rules_with_custom_configuration_have_schema ... FAILED

failures:

---- test_rules_with_custom_configuration_have_schema stdout ----

thread 'test_rules_with_custom_configuration_have_schema' (44628423) panicked at crates/oxc_linter/tests/rule_configuration_documentation_test.rs:151:5:
Found 3 configuration documentation issues:

Rule 'react/forbid-elements' accepts configuration options but has no schema defined.
Please see the oxc website for info on adding config option schemas and docs to this rule.
https://oxc.rs/docs/contribute/linter/adding-rules.html

Rule 'react/jsx-handler-names' accepts configuration options but has no schema defined.
Please see the oxc website for info on adding config option schemas and docs to this rule.
https://oxc.rs/docs/contribute/linter/adding-rules.html

Rule 'react/prefer-es6-class' accepts configuration options but has no schema defined.
Please see the oxc website for info on adding config option schemas and docs to this rule.
https://oxc.rs/docs/contribute/linter/adding-rules.html
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

graphite-app · 2025-11-02T20:53:21Z

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

0-merge - adds this PR to the back of the merge queue
hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

codspeed-hq · 2025-11-02T21:09:43Z

CodSpeed Performance Report

Merging #15234 will not alter performance

_{Comparing connorshea:add-rule-config-enforcer (6cfe552) with main (b2af6b5)¹}

Summary

✅ 4 untouched
⏩ 33 skipped²

No successful run was found on main (aee9be1) during the generation of this report, so b2af6b5 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
33 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Copilot

Pull Request Overview

This PR introduces a test to ensure rules with configuration options have proper schema documentation, and improves the documentation for the node/no-process-env rule by providing a better naming convention for its configuration struct.

Adds a new test file that validates all configurable rules have schema documentation
Refactors the no-process-env rule configuration struct with a more descriptive name
Creates an exception list for 37 existing rules that need schema documentation updates

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
crates/oxc_linter/tests/rule_configuration_documentation_test.rs	New test file that validates all configurable linter rules have proper schema documentation, includes exception list for legacy rules
crates/oxc_linter/src/rules/node/no_process_env.rs	Renames configuration struct from `ConfigElement0` to `NoProcessEnvConfig` with proper documentation and schema annotations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/oxc_linter/src/rules/node/no_process_env.rs

crates/oxc_linter/tests/rule_configuration_documentation_test.rs

…on options. This was mostly built using GitHub Copilot w/ Claude Sonnet 4.5. This is primarily a port of the existing check logic I had written in JavaScript for detecting rules that violated, but somewhat more robust. Due to how this is checked, it runs the website generation as part of this, and also uses `git` in order to run the website generation command. This is _probably_ not desirable and there is likely a much better way to programmatically detect whether the rule is set up correctly *without* needing to generate the full docs as part of the test. But this does work for now and the other attempts I tried weren't detecting all the problematic rules, so I'll push it up as a draft. It's also using a somewhat-naive regex to implement the check for whether the schema is documented in the rule's documentation, which should probably also be improved by doing the check programmatically instead.

…ig options properly. This wasn't configured quite right, now it is. Caught this problem thanks to this test :)

Rather than doing this via a janky string search, actually use the rule attributes to verify that the config is being set up properly for this test.

This isn't necessary and will just need to be removed later anyway.

This should work consistently now.

It currently catches 27/29 of the exclusions correctly. Only two are missed by this regex solution when the exceptions array is removed.

This will ensure it doesn't fail silently if the format of the output changes in the future.

crates/oxc_linter/tests/rule_configuration_documentation_test.rs

…ocs. (#15525) Part of #14743. Extracted from #15234. Generated docs: ```md ## Configuration This rule accepts a configuration object with the following properties: ### allowedVariables type: `string[]` default: `[]` Variable names which are allowed to be accessed on `process.env`. ```

camc314

Nice work!

crates/oxc_linter/tests/rule_configuration_documentation_test.rs

# Oxlint ### 🚀 Features - c199e38 linter: Implement config for all tsgolint rules supporting options (#15659) (camchenry) - 7598b3e linter: Allow configuring tsgolint rules (#15286) (camchenry) - e77f2ce vscode: Support `oxlint --lsp` (#15680) (Sysix) - e01c551 oxlint: Add `--lsp` flag to run the language server (#15611) (Sysix) - ac5c2ab lsp/fmt: Support ignore files & patterns (#15647) (Sysix) - e50a9bf linter: Add unicorn/prefer-response-static-json rule (#15692) (Mikhail Baev) - f5d9abb oxlint: Add enabled? column to --rules cli output (#15213) (Wren) - 6b5205c linter/plugins: Implement deprecated `SourceCode#getJSDocComment` method (#15653) (overlookmotel) - 5eccff1 linter/plugins: Implement deprecated `SourceCode` tokens methods (#15645) (overlookmotel) - 4c62ffa language_server: Add `run_server` helper for CLI tools (#15609) (Sysix) - 0d52a5e linter/plugins: Implement `Context#parserOptions` getter (#15632) (overlookmotel) - 287a538 linter/plugins: Implement `Context#get*` deprecated methods (#15606) (overlookmotel) - 84de1ca oxlint,oxfmt: Allow comments and also commas for vscode-json-ls (#15612) (leaysgur) - e763919 linter: Add test to enforce schema documentation for rules with configuration options (#15234) (Connor Shea) - 6cd0b5f linter/no-callback-in-promise: Add support for `timeoutsErr` option (#15507) (Li Wei) ### 🐛 Bug Fixes - 1decd57 linter/no-empty-named-blocks: Only search within node for comma token (#15751) (camc314) - 17c3d6d linter: Improve docs, diagnostic message, and implementation of typescript/consistent-indexed-object-style rule. (#15750) (connorshea) - 78a6df2 linter/no-empty-named-blocks: Skip comments when searching for token (#15735) (camc314) - 042befd linter/require-await: Skip comments when searching for token (#15734) (camc314) - 7a78220 linter/no-async-await: Skip comments when searching for token (#15733) (camc314) - 23b9ad1 linter/consistent-type-specifier-style: Skip comments when searching for token (#15732) (camc314) - 440a977 ast: Include rest properties when using `get_binding_identifiers` (#15710) (camc314) - 516d14e linter/no-namespace: Skip comments when searching for token (#15716) (camc314) - b5aaace linter/prefer-namespace-keyword: Skip comments when searching for token (#15715) (camc314) - 4eb9fa6 editor: Execute `oxc.path.server` in win32 with shell (#14203) (Sysix) - 9f991a4 linter: Reverse extends overrides priority (#14939) (Peter Wagenet) - 7c4a916 linter: Restores `oxlint --rules -f=json` functionality. (#15689) (Wren) - 24d00f4 linter/plugins: Add types for suggested fixes (#15636) (overlookmotel) - 257360f linter/plugins: Fill in TS type def for `RuleMeta` (#15629) (overlookmotel) - 81e179c linter: Allow file extensions without a dot in react/jsx-filename-extension rule (#15574) (Connor Shea) - 7a0e931 linter: Update the unicorn/prefer-add-event-listener rule with new JavaScript APIs (#15581) (connorshea) - bb5f8ca oxlint: Fix type annotation for big-endian (#15561) (Sysix) ### ⚡ Performance - e2a0997 linter/plugins: Recycle empty visitor object in ESLint compat mode (#15693) (overlookmotel) - 7528db4 language_server: Pass file content as a referenced `String` (#15568) (Sysix) - 54061e9 linter/plugins: Avoid implicit boolean coercion in `initLines` (#15641) (overlookmotel) - ccfd935 language_server: Avoid `Mutex` in `ServerLinter` (#15516) (Sysix) ### 📚 Documentation - 3ab750a linter: Clarify react-in-jsx-scope rule docs. (#15749) (connorshea) - a5feebc linter: `oxlint-disable` not `eslint-disable` (#15672) (overlookmotel) - 3d15805 linter: Reformat doc comments (#15670) (overlookmotel) - 16fcba6 linter: Remove "experimental" from description of stable features (#15669) (overlookmotel) - e62fd98 linter: Correct comment on what `EnablePlugins` does (#15668) (overlookmotel) - a25d31e linter: Fix grammar (#15666) (overlookmotel) - f5f452f linter: Add missing `perf` category (#15667) (overlookmotel) - 2c58952 linter: Add config docs for no-restricted-globals rule. (#15662) (connorshea) - a210b12 linter/plugins: Improve JSDoc comment for `RuleOptionsSchema` (#15642) (overlookmotel) - 3aabfac linter/plugins: Alter comments on `FILE_CONTEXT` used in ESLint-compat `Context` shim (#15605) (overlookmotel) - fd58aea linter: Fix a typo in the docs for react/no-is-mounted. (#15575) (Connor Shea) # Oxfmt ### 🚀 Features - 84de1ca oxlint,oxfmt: Allow comments and also commas for vscode-json-ls (#15612) (leaysgur) - 99823ad oxfmt: Print nothing for default(write) mode (#15583) (leaysgur) - 25a0163 formatter/sort_imports: Sort imports by `Array<Array<string>>` groups (#15578) (leaysgur) ### 🐛 Bug Fixes - bf20cf5 formatter: `CRLF` issue in the member chain (#15764) (Dunqing) - 5d688a0 formatter: Measuring multiline text in `fits_text` is incorrect (#15762) (Dunqing) - e306958 formatter: Regression case for test call (#15760) (Dunqing) - c42d983 formatter: Re-fix all cases that fail after `AstNode::Argument` was removed (#15676) (Dunqing) ### ⚡ Performance - d99a83f oxfmt: Use simdutf8 based read_to_string (#15614) (leaysgur) - 128e186 formatter/sort_imports: Precompute import metadata (#15580) (leaysgur) - cd31cc1 formatter/sort_imports: Use `Vec::with_capacity` for `next_elements` (#15579) (leaysgur) ### 📚 Documentation - 3d15805 linter: Reformat doc comments (#15670) (overlookmotel) Co-authored-by: Boshen <1430279+Boshen@users.noreply.github.com>

# Oxlint ### 🚀 Features - c199e38 linter: Implement config for all tsgolint rules supporting options (oxc-project#15659) (camchenry) - 7598b3e linter: Allow configuring tsgolint rules (oxc-project#15286) (camchenry) - e77f2ce vscode: Support `oxlint --lsp` (oxc-project#15680) (Sysix) - e01c551 oxlint: Add `--lsp` flag to run the language server (oxc-project#15611) (Sysix) - ac5c2ab lsp/fmt: Support ignore files & patterns (oxc-project#15647) (Sysix) - e50a9bf linter: Add unicorn/prefer-response-static-json rule (oxc-project#15692) (Mikhail Baev) - f5d9abb oxlint: Add enabled? column to --rules cli output (oxc-project#15213) (Wren) - 6b5205c linter/plugins: Implement deprecated `SourceCode#getJSDocComment` method (oxc-project#15653) (overlookmotel) - 5eccff1 linter/plugins: Implement deprecated `SourceCode` tokens methods (oxc-project#15645) (overlookmotel) - 4c62ffa language_server: Add `run_server` helper for CLI tools (oxc-project#15609) (Sysix) - 0d52a5e linter/plugins: Implement `Context#parserOptions` getter (oxc-project#15632) (overlookmotel) - 287a538 linter/plugins: Implement `Context#get*` deprecated methods (oxc-project#15606) (overlookmotel) - 84de1ca oxlint,oxfmt: Allow comments and also commas for vscode-json-ls (oxc-project#15612) (leaysgur) - e763919 linter: Add test to enforce schema documentation for rules with configuration options (oxc-project#15234) (Connor Shea) - 6cd0b5f linter/no-callback-in-promise: Add support for `timeoutsErr` option (oxc-project#15507) (Li Wei) ### 🐛 Bug Fixes - 1decd57 linter/no-empty-named-blocks: Only search within node for comma token (oxc-project#15751) (camc314) - 17c3d6d linter: Improve docs, diagnostic message, and implementation of typescript/consistent-indexed-object-style rule. (oxc-project#15750) (connorshea) - 78a6df2 linter/no-empty-named-blocks: Skip comments when searching for token (oxc-project#15735) (camc314) - 042befd linter/require-await: Skip comments when searching for token (oxc-project#15734) (camc314) - 7a78220 linter/no-async-await: Skip comments when searching for token (oxc-project#15733) (camc314) - 23b9ad1 linter/consistent-type-specifier-style: Skip comments when searching for token (oxc-project#15732) (camc314) - 440a977 ast: Include rest properties when using `get_binding_identifiers` (oxc-project#15710) (camc314) - 516d14e linter/no-namespace: Skip comments when searching for token (oxc-project#15716) (camc314) - b5aaace linter/prefer-namespace-keyword: Skip comments when searching for token (oxc-project#15715) (camc314) - 4eb9fa6 editor: Execute `oxc.path.server` in win32 with shell (oxc-project#14203) (Sysix) - 9f991a4 linter: Reverse extends overrides priority (oxc-project#14939) (Peter Wagenet) - 7c4a916 linter: Restores `oxlint --rules -f=json` functionality. (oxc-project#15689) (Wren) - 24d00f4 linter/plugins: Add types for suggested fixes (oxc-project#15636) (overlookmotel) - 257360f linter/plugins: Fill in TS type def for `RuleMeta` (oxc-project#15629) (overlookmotel) - 81e179c linter: Allow file extensions without a dot in react/jsx-filename-extension rule (oxc-project#15574) (Connor Shea) - 7a0e931 linter: Update the unicorn/prefer-add-event-listener rule with new JavaScript APIs (oxc-project#15581) (connorshea) - bb5f8ca oxlint: Fix type annotation for big-endian (oxc-project#15561) (Sysix) ### ⚡ Performance - e2a0997 linter/plugins: Recycle empty visitor object in ESLint compat mode (oxc-project#15693) (overlookmotel) - 7528db4 language_server: Pass file content as a referenced `String` (oxc-project#15568) (Sysix) - 54061e9 linter/plugins: Avoid implicit boolean coercion in `initLines` (oxc-project#15641) (overlookmotel) - ccfd935 language_server: Avoid `Mutex` in `ServerLinter` (oxc-project#15516) (Sysix) ### 📚 Documentation - 3ab750a linter: Clarify react-in-jsx-scope rule docs. (oxc-project#15749) (connorshea) - a5feebc linter: `oxlint-disable` not `eslint-disable` (oxc-project#15672) (overlookmotel) - 3d15805 linter: Reformat doc comments (oxc-project#15670) (overlookmotel) - 16fcba6 linter: Remove "experimental" from description of stable features (oxc-project#15669) (overlookmotel) - e62fd98 linter: Correct comment on what `EnablePlugins` does (oxc-project#15668) (overlookmotel) - a25d31e linter: Fix grammar (oxc-project#15666) (overlookmotel) - f5f452f linter: Add missing `perf` category (oxc-project#15667) (overlookmotel) - 2c58952 linter: Add config docs for no-restricted-globals rule. (oxc-project#15662) (connorshea) - a210b12 linter/plugins: Improve JSDoc comment for `RuleOptionsSchema` (oxc-project#15642) (overlookmotel) - 3aabfac linter/plugins: Alter comments on `FILE_CONTEXT` used in ESLint-compat `Context` shim (oxc-project#15605) (overlookmotel) - fd58aea linter: Fix a typo in the docs for react/no-is-mounted. (oxc-project#15575) (Connor Shea) # Oxfmt ### 🚀 Features - 84de1ca oxlint,oxfmt: Allow comments and also commas for vscode-json-ls (oxc-project#15612) (leaysgur) - 99823ad oxfmt: Print nothing for default(write) mode (oxc-project#15583) (leaysgur) - 25a0163 formatter/sort_imports: Sort imports by `Array<Array<string>>` groups (oxc-project#15578) (leaysgur) ### 🐛 Bug Fixes - bf20cf5 formatter: `CRLF` issue in the member chain (oxc-project#15764) (Dunqing) - 5d688a0 formatter: Measuring multiline text in `fits_text` is incorrect (oxc-project#15762) (Dunqing) - e306958 formatter: Regression case for test call (oxc-project#15760) (Dunqing) - c42d983 formatter: Re-fix all cases that fail after `AstNode::Argument` was removed (oxc-project#15676) (Dunqing) ### ⚡ Performance - d99a83f oxfmt: Use simdutf8 based read_to_string (oxc-project#15614) (leaysgur) - 128e186 formatter/sort_imports: Precompute import metadata (oxc-project#15580) (leaysgur) - cd31cc1 formatter/sort_imports: Use `Vec::with_capacity` for `next_elements` (oxc-project#15579) (leaysgur) ### 📚 Documentation - 3d15805 linter: Reformat doc comments (oxc-project#15670) (overlookmotel) Co-authored-by: Boshen <1430279+Boshen@users.noreply.github.com>

github-actions bot added A-linter Area - Linter C-enhancement Category - New feature or request labels Nov 2, 2025

connorshea force-pushed the add-rule-config-enforcer branch 2 times, most recently from e49bb9b to d4f63fe Compare November 8, 2025 17:59

connorshea marked this pull request as ready for review November 8, 2025 20:56

connorshea requested a review from camc314 as a code owner November 8, 2025 20:56

Copilot AI review requested due to automatic review settings November 8, 2025 20:56

Copilot AI reviewed Nov 8, 2025

View reviewed changes

crates/oxc_linter/src/rules/node/no_process_env.rs Show resolved Hide resolved

crates/oxc_linter/tests/rule_configuration_documentation_test.rs Outdated Show resolved Hide resolved

graphite-app bot reviewed Nov 8, 2025

View reviewed changes

crates/oxc_linter/tests/rule_configuration_documentation_test.rs Outdated Show resolved Hide resolved

connorshea added 15 commits November 8, 2025 14:23

Resolve linter violations.

a2ff921

Remove completed rules from the exceptions list.

a9ff85f

docs(linter): Fix node/no-process-en rule to generate docs for conf…

0e5205f

…ig options properly. This wasn't configured quite right, now it is. Caught this problem thanks to this test :)

feat(linter): Update test to properly check for rule config setup.

e516c5e

Rather than doing this via a janky string search, actually use the rule attributes to verify that the config is being set up properly for this test.

test(linter): Remove extra test.

9617044

This isn't necessary and will just need to be removed later anyway.

Update configuration docs test to simplify code a bit/add comments.

46ff58a

Ensure a default value is set for no-process-env rule.

f1d6dfa

Ensure exceptions list is up to date with rule schemas.

1b18a45

Tweak docs

04def58

Simplify docs check.

1ad9d67

fix(linter): Fix rule configuration documentation test to fully work.

fa29b0d

This should work consistently now.

Add a note about limitations of this current test.

3cac449

It currently catches 27/29 of the exclusions correctly. Only two are missed by this regex solution when the exceptions array is removed.

Use lazy regex outside loop to resolve feedback.

353eead

Improve the error message.

f08836e

connorshea force-pushed the add-rule-config-enforcer branch from a578d43 to f08836e Compare November 8, 2025 21:23

connorshea added 3 commits November 8, 2025 14:49

Improve the regex to capture all 29 rules.

c1e0d62

Minor refactoring to finish tidying up.

690b828

One last tweak.

527c271

This will ensure it doesn't fail silently if the format of the output changes in the future.

graphite-app bot reviewed Nov 8, 2025

View reviewed changes

crates/oxc_linter/tests/rule_configuration_documentation_test.rs Show resolved Hide resolved

connorshea mentioned this pull request Nov 9, 2025

docs(linter): Update node/no-process-env rule to have proper config docs. #15525

Merged

Merge branch 'main' into add-rule-config-enforcer

d36ba5a

camc314 approved these changes Nov 11, 2025

View reviewed changes

crates/oxc_linter/tests/rule_configuration_documentation_test.rs Outdated Show resolved Hide resolved

Merge branch 'main' into add-rule-config-enforcer

6cfe552

camc314 merged commit e763919 into oxc-project:main Nov 11, 2025
19 checks passed

connorshea deleted the add-rule-config-enforcer branch November 12, 2025 00:40

Boshen mentioned this pull request Nov 17, 2025

release(apps): oxlint v1.29.0 && oxfmt v0.14.0 #15772

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(linter): Add test to enforce schema documentation for rules with configuration options#15234

feat(linter): Add test to enforce schema documentation for rules with configuration options#15234
camc314 merged 20 commits intooxc-project:mainfrom
connorshea:add-rule-config-enforcer

connorshea commented Nov 2, 2025 •

edited

Loading

Uh oh!

graphite-app bot commented Nov 2, 2025

Uh oh!

codspeed-hq bot commented Nov 2, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

camc314 left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

connorshea commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

graphite-app bot commented Nov 2, 2025

How to use the Graphite Merge Queue

Uh oh!

codspeed-hq bot commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #15234 will not alter performance

Summary

Footnotes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

camc314 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

connorshea commented Nov 2, 2025 •

edited

Loading

codspeed-hq bot commented Nov 2, 2025 •

edited

Loading