ci: speed up test suite, especially SEA tests#244
Merged
Conversation
…elism - Move SEA tests 85-92 into npmTests (only-npm) — they ignore the target argument and always build for the host Node version, so running them in both test:22 and test:24 was redundant (42 heavy invocations removed) - Share build artifacts: build job uploads lib-es5/ once, test jobs download it instead of each running yarn build (~30s saved × 18 jobs) - Add restore-keys fallback to pkg-cache for better cache hit rates - Add bounded-concurrency test runner (TEST_CONCURRENCY env var): test:22/test:24 run 4 tests in parallel, test:host stays sequential (pnpm/SEA tests have global state that races under concurrency) Closes #241 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The first commit introduced parallel test execution but pkg-fetch uses a deterministic temp filename (*.downloading) that collides when multiple processes download the same binary concurrently. This caused ENOENT errors on CI where the cache was cold (cache key was renamed). Fix by pre-downloading binaries for all platforms (linux, macos, win) with a single sequential pkg invocation before the parallel test loop. Also add the old cache key format as a restore-key fallback so existing caches from previous runs are not wasted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two independent fixes for the CI parallelism issues:
1. lib/index.ts: macOS bytecode signing no longer races across concurrent
pkg processes. The old rm + copy + sign sequence wrote to the shared
'{binary}-signed' path on every invocation, so parallel processes
truncated each other's writes. Now: reuse the signed binary if it
exists; otherwise write to a unique temp path and atomically rename
(rename replaces the target on POSIX, so late writers don't corrupt
earlier readers whose file handles remain valid).
2. test/test.js: use pkg-fetch's need() API directly to warm the binary
cache before parallel execution, instead of spawning pkg. Cleaner,
skips pkg's codesign step, and covers all platform/arch combinations
the tests could target (linux/macos/win × x64/arm64).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
getNodejsExecutable ran checksum verification and archive extraction on every invocation, even when the cached archive and extracted binary were already on disk. nodejs.org archives are immutable, so re-hashing 100 MB and re-extracting buys nothing but does cost ~5-7 s per call. - Checksum verification now runs only when we actually download. - extract() returns early if the target binary already exists. This saves ~7 s per SEA test invocation after the first in each CI job, ~50 s off test_host on CI (7 SEA tests run in that suite). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TEST_CONCURRENCY=4 exposed a race in pkg-fetch's shared cache: parallel pkg processes that detect a hash mismatch on a restored cached binary all call unlinkSync + re-download simultaneously, so only one wins and the rest crash with ENOENT on macOS runners. The warm-up loop in test.js tried to mitigate this by pre-fetching binaries, but the race still triggers when the GH Actions cache restores a binary whose hash doesn't match the current pkg-fetch EXPECTED_HASHES. Parallelism saved ~1 min on test:22/test:24 but required two follow-up fix commits and still failed CI. Reverting to sequential eliminates the race entirely. The macOS signing atomic-rename fix in lib/index.ts stays — it's correct regardless and helps users who happen to run pkg concurrently. The other ci.yml wins from #244 remain intact: SEA test dedup (test_host only), shared build artifacts, and cache restore-keys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each SEA test's pkg invocation defaults to building three binaries (linux, macos, win). Most of the wall time is postject injection (~8s per binary, CPU-bound), yet the test only ever executes the host binary in assertSeaOutput() — the other two are built and discarded. Pass --target host + --output <host-path> so pkg builds the one binary the test actually runs. Local measurement on test-85-sea-enhanced drops from ~25s to ~3s. Across the seven SEA tests in test_host that follows the pattern, ~2.5 min of CI time saved. Factored the suffix lookup and pkg invocation into utils.seaHostOutputs and utils.runSeaHostOnly so the eight tests stay uniform. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rm + copy + sign sequence only races when multiple pkg processes target the same platform concurrently. We introduced the tmp + rename fix to unblock parallel test execution; now that the tests run sequentially again there is no caller that triggers this race in our CI, and the extra code path was speculative coverage for user-side concurrent pkg usage. Reverting to the original sequence keeps the codebase smaller. If a real concurrent-pkg use case surfaces we can reintroduce the atomic variant with a regression test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Earlier in the PR every SEA test was switched to host-only builds for speed. That removed cross-compile coverage — each build previously exercised pkg's SEA pipeline for linux, macos and win targets (postject, node-archive download, platform-specific segment naming). test-00-sea is the simplest SEA test and runs first, so it's the cheapest place to keep that coverage. Restore the multi-target build here; the other seven SEA tests (85-92) stay host-only since their extra coverage is about SEA features (ESM entry, workers, TLA, assets) that don't vary per target, and a regression in those would show up on the host build too. Net CI impact is a single ~25s slot instead of ~190s across all SEA tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously getNodejsExecutable() skipped extraction whenever the target binary existed on disk. Both tar and unzipper write to the final path directly, so a pkg process killed mid-extract (OOM, power loss, SIGKILL) leaves a truncated node binary behind. The next invocation skips re-extraction and silently poisons the build with a corrupted binary. Guard the skip with a sentinel file written only after extraction completes. If the sentinel is absent — fresh cache, or a previous run crashed — we clear any partial output and re-extract. The sentinel name piggybacks on the final path so it lands in the same directory automatically for both win and posix layouts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR aims to reduce CI wall-clock time for yao-pkg/pkg by avoiding redundant SEA test work, sharing build artifacts across test jobs, and improving SEA node archive/extraction behavior.
Changes:
- Move SEA tests (85–92) into the host-only
npmTestslist and add helpers to build/clean up SEA host outputs. - Optimize SEA node archive handling by adding an extraction completion sentinel and skipping repeated checksum verification for cached archives.
- Update GitHub Actions workflows to (a) upload build outputs once and download them in test jobs, and (b) improve
~/.pkg-cacherestore behavior.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
test/utils.js |
Adds seaHostOutputs() / runSeaHostOnly() helpers and reuses a shared SEA platform suffix map. |
test/test.js |
Runs SEA tests only in the host-only suite by expanding npmTests; refactors logging in the test loop. |
test/test-00-sea/main.js |
Adds clarifying comment explaining why this SEA test remains cross-compiling. |
test/test-85-sea-enhanced/main.js |
Switches to host-only SEA outputs and invocation via test utils helpers. |
test/test-86-sea-assets/main.js |
Switches to host-only SEA outputs and invocation via test utils helpers. |
test/test-87-sea-esm/main.js |
Switches to host-only SEA outputs and invocation via test utils helpers. |
test/test-89-sea-fs-ops/main.js |
Switches to host-only SEA outputs and invocation via test utils helpers. |
test/test-90-sea-worker-threads/main.js |
Switches to host-only SEA outputs and invocation via test utils helpers. |
test/test-91-sea-esm-entry/main.js |
Switches to host-only SEA outputs and invocation via test utils helpers. |
test/test-92-sea-tla/main.js |
Switches to host-only SEA outputs and invocation via test utils helpers. |
lib/sea.ts |
Adds extraction .ok sentinel; avoids re-downloading and re-hashing cached archives. |
.github/workflows/test.yml |
Downloads build artifacts instead of running yarn build in each test job; adds cache restore keys. |
.github/workflows/ci.yml |
Uploads build outputs once and makes test workflows depend on the build job. |
Two related gaps in the SEA download/extract cache: 1. extract() returned the cached nodePath when only the .ok sentinel existed. A user or cleanup tool that removed the extracted binary but left the sentinel would leak a stale path to bake(), surfacing as a confusing ENOENT on copyFile instead of a clean re-extract. Gate the skip on both sentinel and binary, and clear a stale sentinel on re-extract so subsequent runs don't hit the same path. 2. getNodejsExecutable() skipped download + checksum whenever the archive file existed on disk. downloadFile writes straight to the final path with no tmp+rename, so an interrupted download left a partial archive that subsequent runs trusted unverified. Apply the same sentinel pattern to the archive: only skip when both archive and .ok sentinel are present; otherwise clear and re-download. Neither fix is an attack-surface change — a compromised pkg-cache directory has always been full trust — but both close legitimate crash-recovery gaps. Addresses PR #244 review feedback on lib/sea.ts:108 and :321. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Test jobs previously blocked on the full 6-cell build matrix even though only one cell (ubuntu-22.x) produced the build-output artifact they consumed. Tests waited on the slowest build (Windows, 80-150s) for no reason. Split into two jobs: - build_artifact: non-matrix, ubuntu + Node 22. Runs lint, build, and uploads the artifact. Tests depend on this. - build: matrix across Linux/macOS/Windows × Node 22/24, sanity check that `yarn build` still succeeds everywhere. Runs in parallel with tests — a platform-specific build failure still marks the PR red without gating the tests. Addresses PR #244 review feedback on ci.yml:79. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
npmTestsso they only run intest:host— they ignore the target argument (always build for the host Node version), so running them in bothtest:22andtest:24was redundant (42 heavy invocations removed)lib-es5/+prelude/sea-bootstrap.bundle.jsonce, test jobs download it instead of each runningyarn build(~30s saved × 18 jobs)restore-keysfallback topkg-cachefor better cache hit rates on first runsTEST_CONCURRENCYenv var —test:22/test:24run 4 tests in parallel (~2.5x speedup),test:hoststays sequential (pnpm/SEA tests have global state that races under concurrency)Expected impact
yarn buildin test jobsTest plan
test:22 no-npmwithTEST_CONCURRENCY=4: 140/141 pass (1 pre-existing failure in test-77), deterministic across 3 runstest:host only-npmwithTEST_CONCURRENCY=1: 20/20 passCloses #241
🤖 Generated with Claude Code