Skip to content

feat: Introduce ecosystem tests for popular plugins#127

Merged
nzakas merged 27 commits into
eslint:mainfrom
JoshuaKGoldberg:repo-ecosystem-plugin-tests
Mar 26, 2025
Merged

feat: Introduce ecosystem tests for popular plugins#127
nzakas merged 27 commits into
eslint:mainfrom
JoshuaKGoldberg:repo-ecosystem-plugin-tests

Conversation

@JoshuaKGoldberg

Copy link
Copy Markdown
Contributor

Summary

Adding an CI job to the eslint/eslint repo that checks changes against a small selection of third-party plugins.

Related Issues

eslint/eslint#19139

@fasttime fasttime added the Initial Commenting This RFC is in the initial feedback stage label Nov 26, 2024
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
@voxpelli

Copy link
Copy Markdown

That none of the eslint-community plugins are included feels a bit odd? Since it's somewhat operated under the same umbrella as ESLint itself?

At least one of eg eslint-plugin-n, eslint-plugin-promise or eslint-plugin-security would be good to include?

I also note that the selection criteria is similar to those outlined in the suggested eslint-community governance, and which have gotten feedback / objections / concerns there: eslint-community/governance#1 (comment)

Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md

@nzakas nzakas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My apologies, I started a review last week, left a bunch of comments, and then forgot to hit "Submit Review".

I think this outlines a good infrastructure, just looking for some clarifications.

Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated

@aladdin-add aladdin-add left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me! This is an excellent starting point. We can proceed to run it and see how it performs. 🚀🚀🚀

Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
nzakas
nzakas previously approved these changes Feb 6, 2025

@nzakas nzakas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the direction of this makes sense now. Nice work.

Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Comment thread designs/2024-repo-ecosystem-plugin-tests/README.md Outdated
Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com>
The new CI job will, for each plugin:

1. Clone the plugin into a directory named `test/ecosystem/${plugin}`
2. Run the plugin's package installation and build commands with [ni](https://github.com/antfu-collective/ni)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How will ni be installed? And will ni automatically install npm (or another package manager) if it's not found locally, and does this also work in a CI environment?

@JoshuaKGoldberg JoshuaKGoldberg Mar 3, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

installed

npm i -g ni,

automatically install

The nci command will.

I'll make those two points explicit.

Aside: I filed antfu-collective/ni#263 as a docs issue on ni to support this.

2. Run the plugin's package installation and build commands with [ni](https://github.com/antfu-collective/ni)
3. Link the plugin to the current eslint installation
- This will have to be done manually, as ni does not support linking ([ni#85](https://github.com/antfu-collective/ni/issues/85 "ni issue 85: Maybe xxx link can join ni project"))
4. Run the plugin's `test:eslint-compat` script with [ni](https://github.com/antfu-collective/ni)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does ni also install Node.js in the correct version for the package?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but last I checked, these packages are all fine with running on approximately the same Node.js versions as ESLint's CI. As in, they might specify preferred packageManager / .nvmrc versions, but they should pass CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we should be fine using actions/setup-node@v4 with Node.js lts/* like it's done in other jobs.

      - uses: actions/setup-node@v4
        with:
          node-version: "lts/*"

(e.g. https://github.com/eslint/eslint/blob/c99db89141f1601abe6f9d398a4b6c126e3a0bdb/.github/workflows/ci.yml#L17-L19).

This will also make sure that npm is installed so we can run npm i -g ni.


See [Plugin Selection](#plugin-selection) below for specifics on which plugins will be included.

> ⚠️ Plugins are currently being asked for feedback on the `test:eslint-compat` script.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's good. If there are any questions I think we can also discuss them in this RFC.

@mdjermanovic

Copy link
Copy Markdown
Member

@JoshuaKGoldberg just a reminder there are a few questions left.

@JoshuaKGoldberg

Copy link
Copy Markdown
Contributor Author

👍 I'd been hoping for an answer on the Discord messaging, #127 (comment). I just went ahead now and pushed it as part of the RFC for review.

@nzakas nzakas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving.

@nzakas

nzakas commented Mar 18, 2025

Copy link
Copy Markdown
Member

@mdjermanovic @fasttime any other comments on this?

I feel like there's enough here now to move forward with a PoC.

@fasttime fasttime left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks.

@nzakas nzakas added Final Commenting This RFC is in the final week of commenting and removed Initial Commenting This RFC is in the initial feedback stage labels Mar 19, 2025
@nzakas

nzakas commented Mar 19, 2025

Copy link
Copy Markdown
Member

Moving to Final Commenting based on two TSC approvals

@mdjermanovic mdjermanovic left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks!

@nzakas

nzakas commented Mar 26, 2025

Copy link
Copy Markdown
Member

Final commenting period has ended, merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature Final Commenting This RFC is in the final week of commenting

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants