Skip to content

ci: speed up test suite, especially SEA tests#244

Merged
robertsLando merged 12 commits into
mainfrom
ci/speed-up-test-suite
Apr 20, 2026
Merged

ci: speed up test suite, especially SEA tests#244
robertsLando merged 12 commits into
mainfrom
ci/speed-up-test-suite

Conversation

@robertsLando

Copy link
Copy Markdown
Member

Summary

  • SEA test deduplication: Move SEA tests 85-92 into npmTests so they only run in test:host — they ignore the target argument (always build for the host Node version), so running them in both test:22 and test:24 was redundant (42 heavy invocations removed)
  • Shared build artifacts: Build job uploads lib-es5/ + prelude/sea-bootstrap.bundle.js once, test jobs download it instead of each running yarn build (~30s saved × 18 jobs)
  • Cache restore-keys: Add restore-keys fallback to pkg-cache for better cache hit rates on first runs
  • Parallel test execution: Bounded-concurrency test runner controlled via TEST_CONCURRENCY env var — test:22/test:24 run 4 tests in parallel (~2.5x speedup), test:host stays sequential (pnpm/SEA tests have global state that races under concurrency)

Expected impact

Metric Before After
test:22/test:24 per-job time ~10m ~3-4m
Redundant SEA test runs 42 0
yarn build in test jobs 18×
Estimated CI wall-clock ~11m ~5-6m

Test plan

  • test:22 no-npm with TEST_CONCURRENCY=4: 140/141 pass (1 pre-existing failure in test-77), deterministic across 3 runs
  • test:host only-npm with TEST_CONCURRENCY=1: 20/20 pass
  • Lint passes
  • CI passes on all matrix entries (ubuntu/windows/macos × node22/node24)

Closes #241

🤖 Generated with Claude Code

robertsLando and others added 10 commits April 16, 2026 12:04
…elism

- Move SEA tests 85-92 into npmTests (only-npm) — they ignore the target
  argument and always build for the host Node version, so running them in
  both test:22 and test:24 was redundant (42 heavy invocations removed)
- Share build artifacts: build job uploads lib-es5/ once, test jobs
  download it instead of each running yarn build (~30s saved × 18 jobs)
- Add restore-keys fallback to pkg-cache for better cache hit rates
- Add bounded-concurrency test runner (TEST_CONCURRENCY env var):
  test:22/test:24 run 4 tests in parallel, test:host stays sequential
  (pnpm/SEA tests have global state that races under concurrency)

Closes #241

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The first commit introduced parallel test execution but pkg-fetch uses
a deterministic temp filename (*.downloading) that collides when
multiple processes download the same binary concurrently. This caused
ENOENT errors on CI where the cache was cold (cache key was renamed).

Fix by pre-downloading binaries for all platforms (linux, macos, win)
with a single sequential pkg invocation before the parallel test loop.
Also add the old cache key format as a restore-key fallback so existing
caches from previous runs are not wasted.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two independent fixes for the CI parallelism issues:

1. lib/index.ts: macOS bytecode signing no longer races across concurrent
   pkg processes. The old rm + copy + sign sequence wrote to the shared
   '{binary}-signed' path on every invocation, so parallel processes
   truncated each other's writes. Now: reuse the signed binary if it
   exists; otherwise write to a unique temp path and atomically rename
   (rename replaces the target on POSIX, so late writers don't corrupt
   earlier readers whose file handles remain valid).

2. test/test.js: use pkg-fetch's need() API directly to warm the binary
   cache before parallel execution, instead of spawning pkg. Cleaner,
   skips pkg's codesign step, and covers all platform/arch combinations
   the tests could target (linux/macos/win × x64/arm64).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
getNodejsExecutable ran checksum verification and archive extraction on
every invocation, even when the cached archive and extracted binary were
already on disk. nodejs.org archives are immutable, so re-hashing 100 MB
and re-extracting buys nothing but does cost ~5-7 s per call.

- Checksum verification now runs only when we actually download.
- extract() returns early if the target binary already exists.

This saves ~7 s per SEA test invocation after the first in each CI job,
~50 s off test_host on CI (7 SEA tests run in that suite).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TEST_CONCURRENCY=4 exposed a race in pkg-fetch's shared cache: parallel
pkg processes that detect a hash mismatch on a restored cached binary
all call unlinkSync + re-download simultaneously, so only one wins and
the rest crash with ENOENT on macOS runners. The warm-up loop in
test.js tried to mitigate this by pre-fetching binaries, but the race
still triggers when the GH Actions cache restores a binary whose hash
doesn't match the current pkg-fetch EXPECTED_HASHES.

Parallelism saved ~1 min on test:22/test:24 but required two follow-up
fix commits and still failed CI. Reverting to sequential eliminates
the race entirely. The macOS signing atomic-rename fix in
lib/index.ts stays — it's correct regardless and helps users who
happen to run pkg concurrently.

The other ci.yml wins from #244 remain intact: SEA test dedup
(test_host only), shared build artifacts, and cache restore-keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each SEA test's pkg invocation defaults to building three binaries
(linux, macos, win). Most of the wall time is postject injection
(~8s per binary, CPU-bound), yet the test only ever executes the
host binary in assertSeaOutput() — the other two are built and
discarded.

Pass --target host + --output <host-path> so pkg builds the one
binary the test actually runs. Local measurement on test-85-sea-enhanced
drops from ~25s to ~3s. Across the seven SEA tests in test_host that
follows the pattern, ~2.5 min of CI time saved.

Factored the suffix lookup and pkg invocation into utils.seaHostOutputs
and utils.runSeaHostOnly so the eight tests stay uniform.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rm + copy + sign sequence only races when multiple pkg processes
target the same platform concurrently. We introduced the tmp + rename
fix to unblock parallel test execution; now that the tests run
sequentially again there is no caller that triggers this race in our
CI, and the extra code path was speculative coverage for user-side
concurrent pkg usage.

Reverting to the original sequence keeps the codebase smaller. If a
real concurrent-pkg use case surfaces we can reintroduce the atomic
variant with a regression test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Earlier in the PR every SEA test was switched to host-only builds for
speed. That removed cross-compile coverage — each build previously
exercised pkg's SEA pipeline for linux, macos and win targets
(postject, node-archive download, platform-specific segment naming).

test-00-sea is the simplest SEA test and runs first, so it's the
cheapest place to keep that coverage. Restore the multi-target build
here; the other seven SEA tests (85-92) stay host-only since their
extra coverage is about SEA features (ESM entry, workers, TLA, assets)
that don't vary per target, and a regression in those would show up
on the host build too.

Net CI impact is a single ~25s slot instead of ~190s across all SEA
tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously getNodejsExecutable() skipped extraction whenever the
target binary existed on disk. Both tar and unzipper write to the
final path directly, so a pkg process killed mid-extract (OOM,
power loss, SIGKILL) leaves a truncated node binary behind. The
next invocation skips re-extraction and silently poisons the
build with a corrupted binary.

Guard the skip with a sentinel file written only after extraction
completes. If the sentinel is absent — fresh cache, or a previous
run crashed — we clear any partial output and re-extract. The
sentinel name piggybacks on the final path so it lands in the
same directory automatically for both win and posix layouts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to reduce CI wall-clock time for yao-pkg/pkg by avoiding redundant SEA test work, sharing build artifacts across test jobs, and improving SEA node archive/extraction behavior.

Changes:

  • Move SEA tests (85–92) into the host-only npmTests list and add helpers to build/clean up SEA host outputs.
  • Optimize SEA node archive handling by adding an extraction completion sentinel and skipping repeated checksum verification for cached archives.
  • Update GitHub Actions workflows to (a) upload build outputs once and download them in test jobs, and (b) improve ~/.pkg-cache restore behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
test/utils.js Adds seaHostOutputs() / runSeaHostOnly() helpers and reuses a shared SEA platform suffix map.
test/test.js Runs SEA tests only in the host-only suite by expanding npmTests; refactors logging in the test loop.
test/test-00-sea/main.js Adds clarifying comment explaining why this SEA test remains cross-compiling.
test/test-85-sea-enhanced/main.js Switches to host-only SEA outputs and invocation via test utils helpers.
test/test-86-sea-assets/main.js Switches to host-only SEA outputs and invocation via test utils helpers.
test/test-87-sea-esm/main.js Switches to host-only SEA outputs and invocation via test utils helpers.
test/test-89-sea-fs-ops/main.js Switches to host-only SEA outputs and invocation via test utils helpers.
test/test-90-sea-worker-threads/main.js Switches to host-only SEA outputs and invocation via test utils helpers.
test/test-91-sea-esm-entry/main.js Switches to host-only SEA outputs and invocation via test utils helpers.
test/test-92-sea-tla/main.js Switches to host-only SEA outputs and invocation via test utils helpers.
lib/sea.ts Adds extraction .ok sentinel; avoids re-downloading and re-hashing cached archives.
.github/workflows/test.yml Downloads build artifacts instead of running yarn build in each test job; adds cache restore keys.
.github/workflows/ci.yml Uploads build outputs once and makes test workflows depend on the build job.

Comment thread test/test.js
Comment thread .github/workflows/ci.yml
Comment thread lib/sea.ts
Comment thread lib/sea.ts Outdated
robertsLando and others added 2 commits April 20, 2026 16:08
Two related gaps in the SEA download/extract cache:

1. extract() returned the cached nodePath when only the .ok sentinel
   existed. A user or cleanup tool that removed the extracted binary
   but left the sentinel would leak a stale path to bake(), surfacing
   as a confusing ENOENT on copyFile instead of a clean re-extract.
   Gate the skip on both sentinel and binary, and clear a stale
   sentinel on re-extract so subsequent runs don't hit the same path.

2. getNodejsExecutable() skipped download + checksum whenever the
   archive file existed on disk. downloadFile writes straight to the
   final path with no tmp+rename, so an interrupted download left a
   partial archive that subsequent runs trusted unverified. Apply the
   same sentinel pattern to the archive: only skip when both archive
   and .ok sentinel are present; otherwise clear and re-download.

Neither fix is an attack-surface change — a compromised pkg-cache
directory has always been full trust — but both close legitimate
crash-recovery gaps.

Addresses PR #244 review feedback on lib/sea.ts:108 and :321.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Test jobs previously blocked on the full 6-cell build matrix even
though only one cell (ubuntu-22.x) produced the build-output artifact
they consumed. Tests waited on the slowest build (Windows, 80-150s)
for no reason.

Split into two jobs:
- build_artifact: non-matrix, ubuntu + Node 22. Runs lint, build,
  and uploads the artifact. Tests depend on this.
- build: matrix across Linux/macOS/Windows × Node 22/24, sanity check
  that `yarn build` still succeeds everywhere. Runs in parallel with
  tests — a platform-specific build failure still marks the PR red
  without gating the tests.

Addresses PR #244 review feedback on ci.yml:79.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@robertsLando robertsLando merged commit aa63483 into main Apr 20, 2026
26 checks passed
@robertsLando robertsLando deleted the ci/speed-up-test-suite branch April 20, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: speed up test suite, especially SEA tests

2 participants