test(e2e): retire docker-unreachable gateway script#5119
Conversation
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
📝 WalkthroughWalkthroughThis PR retires the shell-based ChangesDocker-unreachable test retirement and Vitest migration
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
E2E Advisor RecommendationRequired E2E: None Dispatch hint: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
E2E Scenario Advisor RecommendationRequired scenario E2E: None Full scenario advisor summaryE2E Scenario AdvisorBase: Required scenario E2E
Optional scenario E2E
Relevant changed files
|
PR Review AdvisorFindings: 0 needs attention, 0 worth checking, 0 nice ideas Consider writing more tests for
This is an automated advisory review. A human maintainer must make the final merge decision. |
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Direction on migration evidence and
|
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
|
Addressed this direction in 33e9601: the deleted shell script is no longer kept as a retired direct-entry row in legacy-inventory.json, and the inventory test is back to requiring direct legacy entries to exist. The deletion rationale now lives in the PR body / linked issue trail, while the branch keeps the focused Vitest caller-level regression and the workflow-contract test for the removed lane. |
Renames test/fixtures/strict-tool-call-probe-driver.cjs to .ts and spawns it via tsx to satisfy the codebase-growth guardrail that forbids newly added .js/.cjs/.mjs files. The driver body stays JS-shaped (CommonJS-style) because the embedded `node -e` strings must remain plain JS for the spawned child processes, and dist/lib/* targets are CJS modules. Uses `createRequire(import.meta.url)` plus `import.meta.url`-derived __dirname/__filename so tsx's ESM-default execution can still require() the built artifacts. `@ts-nocheck` keeps the surface identical to the retired bash heredoc. Verified: `tsx test/fixtures/strict-tool-call-probe-driver.ts` prints all four `[PASS]` markers; `npx vitest run --project cli test/strict-tool-call-probe.test.ts test/regression-e2e-workflow.test.ts` passes 3/3; `npx biome check` clean; `tsc --noEmit -p tsconfig.cli.json` clean for touched files; file-size budget passes. Refs #5098, #4537, #4349, #5119.
## Summary Retires the legacy `test/e2e/test-strict-tool-call-probe.sh` regression and replaces it with a focused Vitest process test that drives the same caller-level Local Ollama strict Chat Completions tool-call validation behavior against a hermetic mock endpoint. The legacy script was process-level mock-driven (`node -e ...` against in-process module mocks plus a localhost stub) — the same shape `nemoclaw-e2e-legacy-migrate` Phase 0 explicitly refuses, with a recommendation to follow #5119's retirement pattern: author the replacement directly as a Vitest test in `test/`, delete the bash script, remove the regression-e2e job. ## Related Issue Refs #5098 Refs #4537 Refs #4349 Refs #5119 ## Changes - Adds `test/strict-tool-call-probe.test.ts` — a Vitest test that spawns `test/fixtures/strict-tool-call-probe-driver.ts` via `tsx` and asserts the four legacy `[PASS]` markers (success, onboarding caller, transient-502 retry, plain-text fail-closed). - Adds `test/fixtures/strict-tool-call-probe-driver.ts` containing the former inline `node -e` block from the bash script. The driver is authored as TypeScript (rather than `.cjs`) per the codebase-growth guardrail forbidding newly added `.js`/`.cjs`/`.mjs` files; body is JS-shaped because the embedded `node -e` strings must remain plain CommonJS for the spawned children, and the dist/lib/* targets are CJS modules. Uses `createRequire(import.meta.url)` plus `fileURLToPath`-derived `__dirname` so tsx's ESM-default execution can still load the CJS dist artifacts. `// @ts-nocheck` keeps the surface identical to the retired bash heredoc. - Driver runs in a fresh `tsx` child process so its `curl`-based validation subprocess is identical to production runtime conditions and is not affected by Vitest's worker pool, fetch shim, or signal handling. - Deletes `test/e2e/test-strict-tool-call-probe.sh`. - Removes the `strict-tool-call-probe-e2e` job, its selector branch, output declaration, and `Valid:` description entry from `.github/workflows/regression-e2e.yaml`. - Adds a workflow-contract test in `test/regression-e2e-workflow.test.ts` proving the retired job is not advertised or selected (mirrors the docker-unreachable contract test added in #5119). - Removes the deleted shell script's row from `test/e2e-scenario/migration/legacy-inventory.json` (mirrors #5119's inventory hygiene); the retirement rationale is captured in this PR and the linked issue trail. ### Why not live E2E scenario fixtures? This probe asserts payload shape and retry behavior on production caller code against a localhost OpenAI-compatible stub. It does not need a live sandbox lifecycle, scenario registry entry, shared fixture, or matrix runner. Keeping it as a focused process Vitest test preserves the same caller boundary without adding another live E2E surface. ### Simplicity check - Test shape: focused process Vitest test. - New shared helpers: none. - New live E2E fixtures/registry/ledger: none. - Workflow changes: remove the retired regression lane. - Pre-#5126 compatibility: the `legacy-inventory.json` row is removed only because the current deletion gate requires the inventory to match existing `test/e2e/test-*.sh` entry points. ### Build dependency This test consumes built artifacts from `dist/lib/...` (matches the existing CLI-shard convention). CI runs `npm run build:cli` before vitest. For local standalone runs, run `npm run build:cli` first; without it, the wrapper fails fast with a clear `npm run build:cli` diagnostic before launching the driver. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Verification - `npm run build:cli` - `node_modules/.bin/tsx test/fixtures/strict-tool-call-probe-driver.ts` (legacy parity check — all four `[PASS]` markers print; matches `bash test/e2e/test-strict-tool-call-probe.sh` output line-for-line) - `npx vitest run --project cli test/strict-tool-call-probe.test.ts test/regression-e2e-workflow.test.ts test/e2e-scenario/framework-tests/e2e-migration-inventory.test.ts test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts` → 10/10 passing - `npx tsc --noEmit -p tsconfig.cli.json` clean for the touched files - `npx biome check test/strict-tool-call-probe.test.ts test/fixtures/strict-tool-call-probe-driver.ts` clean - `npx tsx scripts/check-test-file-size-budget.ts` passes - YAML syntax check on `.github/workflows/regression-e2e.yaml` passes - [ ] `npx prek run --all-files` passes - [ ] `npm test` passes - [x] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [ ] Docs updated for user-facing behavior changes - [ ] `npm run docs` builds without warnings (doc changes only) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Removed strict tool-call probe regression test lane from the continuous integration workflow * Updated test infrastructure with modernized validation for Chat Completions tool-call enforcement on local integration paths <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary Retires the legacy caller-level onboard inference smoke bash regression now that the same contract is covered by a focused Vitest process test. ## Related Issues Refs #3253 Refs #4349 Refs #5098 Refs #5119 ## Changes - Adds `test/onboard-inference-smoke.test.ts`, which drives `setupInference()` in a subprocess with PATH-shimmed `openshell` and `curl`. - Verifies onboard probes `/chat/completions`, fails closed on upstream HTTP 503, surfaces provider/model/API base/credential diagnostics, and does not print route success after smoke failure. - Deletes `test/e2e/test-onboard-inference-smoke.sh`. - Removes the retired `onboard-inference-smoke-e2e` lane from `regression-e2e.yaml`. - Updates the post-#5126 frozen legacy E2E shell allowlist and regression workflow contract test to keep the retired lane out of dispatch selection. ## Simplicity check - Test shape: focused process Vitest test. - New shared helpers: none. - New framework/registry/ledger: none. - Workflow changes: remove retired regression lane and update frozen shell/workflow contract allowlists. ## Verification - `npm run build:cli` - `npx vitest run test/onboard-inference-smoke.test.ts` - `npx tsc --noEmit -p jsconfig.json` - `npx vitest run --project cli test/onboard-inference-smoke.test.ts test/regression-e2e-workflow.test.ts test/e2e-script-workflow.test.ts --silent=false --reporter=default` - `git diff --check` Notes: - A local full hook/all-files run was attempted and failed on existing environment-dependent failures unrelated to this change (missing nested `nemoclaw/dist` / `nemoclaw/node_modules/json5`, Docker daemon unavailable, and several local timeout-sensitive tests). CI should run the repository's normal validation in the correct environment. ## Type of Change - [x] Test-only change - [x] CI/workflow cleanup for retired legacy coverage - [x] No secrets, API keys, or credentials committed <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Migrated onboard inference smoke test from shell script to Vitest-based test for improved reliability and isolation. * **Chores** * Removed `onboard-inference-smoke-e2e` from regression workflow selectable jobs. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Summary
Retires the legacy Docker-unreachable gateway-start shell regression now that the caller-level contract is covered by a focused Vitest process test. The new test drives
startGateway()through a PATH-shimmedopenshellbinary and proves the fallback exits before health polling, status/info probes, diagnostics, or cleanup.Related Issue
Closes #5113
Refs #5098
Refs #4355
Refs #2347
Changes
test/onboard-gateway-docker-unreachable.test.tswith the former shell trace assertions as a focused Vitest process regression.it.todocoverage-gap block fromsrc/lib/onboard/gateway-start-failure-integration.test.tsand points helper-level coverage to the new caller-level test.test/e2e/test-docker-unreachable-gateway-start.shand removes itsregression-e2e.yamldispatch lane.legacy-inventory.jsonso the inventory continues to track current direct legacy entrypoints; the retirement rationale is captured in this PR and linked issue trail.test/onboard.test.tslegacy line budget after moving the regression out.Type of Change
Verification
npm run build:clinpx vitest run --project cli test/onboard-gateway-docker-unreachable.test.ts src/lib/onboard/gateway-start-failure-integration.test.ts test/e2e-scenario/framework-tests/e2e-migration-inventory.test.ts test/regression-e2e-workflow.test.ts --silent=false --reporter=defaultHOME=/home/cvillela/.cache/nemoclaw-makecheck-nosign XDG_CONFIG_HOME=/home/cvillela/.cache/nemoclaw-makecheck-nosign/.config npx prek run --files .github/workflows/regression-e2e.yaml ci/test-file-size-budget.json test/onboard-gateway-docker-unreachable.test.ts test/onboard.test.ts src/lib/onboard/gateway-start-failure-integration.test.ts test/e2e-scenario/framework-tests/e2e-migration-inventory.test.ts test/e2e-scenario/migration/legacy-inventory.json test/regression-e2e-workflow.test.tsHOME=/home/cvillela/.cache/nemoclaw-makecheck-nosign XDG_CONFIG_HOME=/home/cvillela/.cache/nemoclaw-makecheck-nosign/.config npx prek run --files test/e2e-scenario/framework-tests/e2e-migration-inventory.test.ts test/e2e-scenario/migration/legacy-inventory.jsonHOME=/home/cvillela/.cache/nemoclaw-makecheck-nosign XDG_CONFIG_HOME=/home/cvillela/.cache/nemoclaw-makecheck-nosign/.config npm run test-size:checkgit diff --checknpx prek run --all-filespassesnpm testpassesTests added or updated for new or changed behavior
No secrets, API keys, or credentials committed
Docs updated for user-facing behavior changes
npm run docsbuilds without warnings (doc changes only)Doc pages follow the style guide (doc changes only)
New doc pages include SPDX header and frontmatter (new pages only)
Signed-off-by: Carlos Villela cvillela@nvidia.com