chore: add initial ecosystem plugin tests workflow by JoshuaKGoldberg · Pull Request #19643 · eslint/eslint

JoshuaKGoldberg · 2025-04-21T19:47:40Z

Prerequisites checklist

I have read the contributing guidelines.

What is the purpose of this pull request? (put an "X" next to an item)

[ ] Documentation update
[ ] Bug fix (template)
[ ] New rule (template)
[ ] Changes an existing rule (template)
[ ] Add autofix to a rule
[ ] Add a CLI option
[ ] Add something to the core
[x] Other, please explain: internal CI change; implements eslint/rfcs#127

What changes did you make? (Give an overview)

Fixes #19139.

Adds an initial .github/workflows/ecosystem.yml and package script.

Is there anything you'd like reviewers to focus on?

netlify · 2025-04-21T19:47:57Z

✅ Deploy Preview for docs-eslint canceled.

Name	Link
🔨 Latest commit	`78095d7`
🔍 Latest deploy log	https://app.netlify.com/projects/docs-eslint/deploys/69da7814a3e73300089b8610

nzakas

Thanks for getting started on this. Left some comments throughout.

github-actions · 2025-05-10T22:33:50Z

Hi everyone, it looks like we lost track of this pull request. Please review and see what the next steps are. This pull request will auto-close in 7 days without an update.

Co-authored-by: Nicholas C. Zakas <nicholas@humanwhocodes.com>

JoshuaKGoldberg · 2025-05-12T17:16:04Z

Todo for self: noting https://github.com/eslint/eslint/pull/19643/files#r2068835929 at root level so I don't forget to file the issue.

JoshuaKGoldberg · 2026-04-06T17:37:10Z

Oh, right! Sorry - after all the back-and-forth around direction I'd lost track of this. I'll get it up-to-date, address feedback, and re-request review once it's ready again.

Edit: I remember now, I'd been waiting for resolution on #19643 (comment) too. I'll just resolve that thread since it's been a while.

JoshuaKGoldberg · 2026-04-07T01:19:29Z

Well, since last review activity, several packages have started failing. Specifically:

@eslint/markdown
@typescript-eslint/typescript-eslint

These still pass:

@eslint/css
@eslint/json
@eslint-community/eslint-plugin-eslint-comments
eslint-plugin-unicorn
eslint-plugin-vue

Proposal: would the team be ok with me removing the currently-failing packages, with followup issues to add them back in? Failures keep popping up between long review periods and it's slow wack-a-mole fixing them.

fasttime · 2026-04-07T14:50:25Z

Well, since last review activity, several packages have started failing. Specifically:
* `@eslint/markdown`
* `@typescript-eslint/typescript-eslint`
These still pass:
* `@eslint/css`
* `@eslint/json`
* `@eslint-community/eslint-plugin-eslint-comments`
* `eslint-plugin-unicorn`
* `eslint-plugin-vue`
Proposal: would the team be ok with me removing the currently-failing packages, with followup issues to add them back in? Failures keep popping up between long review periods and it's slow wack-a-mole fixing them.

I think that's perfectly okay 👍 For @eslint/markdown, the problem seems to be just a missing "types": [] entry in tsconfig.json, which causes TypeScript to use the outdated @types/eslint-scope package in the node_modules of the eslnt repo. For @typescript-eslint/typescript-eslint, we'd need to take a closer look (I suspect that yarn and npm don't fit well together), but that could be done later.

fasttime · 2026-04-07T15:39:35Z

+	runCommand(["git", "checkout", pluginSettings.commit]);
+
+	// 3. Install the plugin's dependencies
+	runCommand(["pwd"]);


I'm testing on Windows and this doesn't seem to work because, in PowerShell pwd is a cmdlet alias rather than an executable. This is what I'm seeing:

Testing @eslint/css in ...\eslint\ecosystem\eslint-css [@eslint/css] git clone https://github.com/eslint/css ...\eslint\ecosystem\eslint-css --depth 1 [@eslint/css] git fetch origin 5c1c6251a359d26d117171de3a1e258e2c15ded9 [@eslint/css] git checkout 5c1c6251a359d26d117171de3a1e258e2c15ded9 [@eslint/css] pwd Failed: @eslint/css Errors occurred while testing plugins: @eslint/css: Error: spawnSync pwd ENOENT at notFoundError (...\eslint\node_modules\cross-spawn\lib\enoent.js:6:26) at Object.verifyENOENTSync (...\eslint\node_modules\cross-spawn\lib\enoent.js:48:16) at spawn.spawnSync [as sync] (...\eslint\node_modules\cross-spawn\index.js:29:43) at runCommand (file:///.../eslint/tools/test-ecosystem/index.mjs:62:24) at runTests (file:///.../eslint/tools/test-ecosystem/index.mjs:90:2) at async file:///.../eslint/tools/test-ecosystem/index.mjs:127:3

Not a blocker since the ecosystem tests run on Unix in CI, but it would still be nice to fix this for Windows compatibility if possible.

Oh wait, cross-spawn uses the old cmd.exe shell on Windows, so instead of pwd, this could be just cd without arguments to print the current directory.

I would definitely like to leave any Windows-specific work as a followup. 🫠

Co-authored-by: Francesco Trotta <github@fasttime.org> Co-authored-by: 루밀LuMir <rpfos@naver.com>

fasttime

LGTM, thanks! Leaving open for @lumirlumir to verify.

lumirlumir

I'm sorry for the delay. Everything looks good to me.

Thank you for your time and effort on this PR!

JoshuaKGoldberg · 2026-04-28T13:26:41Z

Whoop! Thanks everyone for reviewing - this was a lot of back-and-forth, and I'm really excited to see it finally land!

github-project-automation Bot added this to Triage Apr 21, 2025

github-project-automation Bot moved this to Needs Triage in Triage Apr 21, 2025

eslint-github-bot Bot added the chore This change is not user-facing label Apr 21, 2025

github-actions Bot added the github actions label Apr 21, 2025

github-advanced-security AI found potential problems Apr 21, 2025

View reviewed changes

Comment thread tools/test-ecosystem.mjs Fixed

JoshuaKGoldberg force-pushed the ecosystem-tests branch 5 times, most recently from 578c12d to baa84df Compare April 22, 2025 15:35

chore: add initial ecosystem plugin tests workflow

baa84df

JoshuaKGoldberg marked this pull request as ready for review April 22, 2025 15:48

JoshuaKGoldberg requested a review from a team as a code owner April 22, 2025 15:48

Merge branch 'main' into ecosystem-tests

b53fc7a

JoshuaKGoldberg commented Apr 22, 2025

View reviewed changes

Comment thread .github/workflows/ecosystem.yml Outdated

Comment thread .github/workflows/ecosystem.yml Outdated

Comment thread package.json Outdated

Comment thread .github/workflows/ecosystem.yml

Comment thread tools/test-ecosystem.mjs Outdated

mdjermanovic moved this from Needs Triage to Implementing in Triage Apr 22, 2025

mdjermanovic added the accepted There is consensus among the team that this change meets the criteria for inclusion label Apr 22, 2025

nzakas reviewed Apr 30, 2025

View reviewed changes

Comment thread tools/test-ecosystem.mjs Outdated

Comment thread tools/test-ecosystem.mjs Outdated

Comment thread tools/test-ecosystem.mjs Outdated

github-actions Bot added the Stale label May 10, 2025

fisker reviewed May 11, 2025

View reviewed changes

Comment thread tools/test-ecosystem.mjs Outdated

fisker reviewed May 11, 2025

View reviewed changes

Comment thread tools/test-ecosystem.mjs Outdated

github-actions Bot removed the Stale label May 11, 2025

Apply suggestions from code review

4bb2f02

Co-authored-by: Nicholas C. Zakas <nicholas@humanwhocodes.com>

JoshuaKGoldberg added 3 commits May 12, 2025 13:26

Comments and structure

f17af7d

Switch to nano-spawn, with plugins.json

30404a4

Merge branch 'main'

d822b05

JoshuaKGoldberg requested a review from fisker May 19, 2025 19:23

Merge branch 'main' into ecosystem-tests

d90144f

JoshuaKGoldberg added 2 commits April 6, 2026 14:23

feat: granular failing

8f3b0f1

fix: remove erroneous build scripts

4480759

JoshuaKGoldberg requested a review from fasttime April 7, 2026 01:19

Bump to latest commits, and add quick summaries

a367527

fasttime reviewed Apr 7, 2026

View reviewed changes

JoshuaKGoldberg added 2 commits April 8, 2026 08:52

remove antfu/ni now

9fd5ba1

remove markdown, typescript-eslint

750c104

JoshuaKGoldberg requested a review from fasttime April 8, 2026 13:06

fasttime reviewed Apr 9, 2026

View reviewed changes

Comment thread .github/workflows/ecosystem-updates.yml Outdated

Comment thread tools/test-ecosystem/index.mjs Outdated

Comment thread tools/test-ecosystem/index.mjs Outdated

lumirlumir reviewed Apr 9, 2026

View reviewed changes

Comment thread .github/workflows/ecosystem-tests.yml Outdated

Comment thread .github/workflows/ecosystem-updates.yml Outdated

Comment thread .github/workflows/ecosystem-updates.yml Outdated

lumirlumir reviewed Apr 9, 2026

View reviewed changes

Comment thread .gitignore

JoshuaKGoldberg and others added 3 commits April 11, 2026 12:32

Apply suggestions from code review

c7be37a

Co-authored-by: Francesco Trotta <github@fasttime.org> Co-authored-by: 루밀LuMir <rpfos@naver.com>

recursive

d4eaff0

correction: result(s); add to ignores

78095d7

JoshuaKGoldberg requested review from fasttime and lumirlumir April 11, 2026 16:35

fasttime approved these changes Apr 14, 2026

View reviewed changes

lumirlumir approved these changes Apr 27, 2026

View reviewed changes

lumirlumir merged commit 77add7f into eslint:main Apr 27, 2026
38 checks passed

github-project-automation Bot moved this from Second Review Needed to Complete in Triage Apr 27, 2026

JoshuaKGoldberg deleted the ecosystem-tests branch April 27, 2026 12:17

This was referenced Apr 28, 2026

Repo: Add @eslint/markdown and typescript-eslint ecosystem plugin tests #20816

Closed

Repo: roll ecosystem tests out to main #20819

Closed

JoshuaKGoldberg mentioned this pull request May 18, 2026

Repo: Improve failure message for ecosystem tests failure #20886

Open

fasttime mentioned this pull request Jun 8, 2026

Improve ecosystem tests compatibility on Windows #20960

Open

3 tasks

Uh oh!

Uh oh!

Conversation

JoshuaKGoldberg commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Prerequisites checklist

What is the purpose of this pull request? (put an "X" next to an item)

What changes did you make? (Give an overview)

Is there anything you'd like reviewers to focus on?

Uh oh!

netlify Bot commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for docs-eslint canceled.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nzakas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 10, 2025

Uh oh!

Uh oh!

Uh oh!

JoshuaKGoldberg commented May 12, 2025

Uh oh!

JoshuaKGoldberg commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshuaKGoldberg commented Apr 7, 2026

Uh oh!

fasttime commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fasttime Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

fasttime Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

JoshuaKGoldberg Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fasttime left a comment

Choose a reason for hiding this comment

Uh oh!

lumirlumir left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JoshuaKGoldberg commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

JoshuaKGoldberg commented Apr 21, 2025 •

edited

Loading

netlify Bot commented Apr 21, 2025 •

edited

Loading

JoshuaKGoldberg commented Apr 6, 2026 •

edited

Loading

fasttime commented Apr 7, 2026 •

edited

Loading