Skip to content

test(e2e): migrate common-egress agent scenario#5360

Merged
cv merged 5 commits into
mainfrom
codex/5098-common-egress-agent-e2e
Jun 13, 2026
Merged

test(e2e): migrate common-egress agent scenario#5360
cv merged 5 commits into
mainfrom
codex/5098-common-egress-agent-e2e

Conversation

@cv

@cv cv commented Jun 12, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Collaborator

Summary

Migrates test/e2e/test-common-egress-agent-e2e.sh into a focused live Vitest scenario. The new test preserves the legacy OpenClaw balanced/open and Hermes open common-egress agent-turn contracts while using existing fixture clients for artifacts, cleanup, redaction, and live skip gates.

Related Issue

Refs #5098

Changes

  • Added test/e2e-scenario/live/common-egress-agent.test.ts with OpenClaw SSH agent-turn checks, Hermes API-server agent checks, policy assertions, live-secret gating, and legacy skip/keep-sandbox knobs.
  • Added common-egress-agent-vitest to the free-standing E2E Vitest workflow, inventory, report fan-in, and selector support tests.
  • Added workflow boundary validation for the new live-secret job shape.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • Tests

    • Added live end-to-end scenarios for the common egress agent, including policy assertions and agent token verification.
    • Expanded support tests to cover workflow dispatch/selector behavior for the new scenario.

  • Chores

    • Added a CI job to run the new Vitest E2E scenarios and upload artifacts; PR report now includes its status.
    • Added workflow validation to enforce job and artifact requirements.

@cv
test(e2e): migrate common-egress agent scenario
f5099e5 
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv self-assigned this Jun 12, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jun 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f3adbbe0-ec75-4d2b-b2da-1f04c0688560

📥 Commits

Reviewing files that changed from the base of the PR and between dd8e86d and 169c14d.

📒 Files selected for processing (3)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
  • tools/e2e-scenarios/workflow-boundary.mts
✅ Files skipped from review due to trivial changes (1)
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • tools/e2e-scenarios/workflow-boundary.mts
📝 Walkthrough

Walkthrough

This PR introduces a new free-standing Vitest E2E job for common-egress-agent scenarios. It adds 1,043 lines total: a comprehensive test suite with OpenClaw and Hermes agent assertions, a GitHub Actions workflow job definition, workflow-boundary validation rules, and dispatch-selector test coverage to ensure the job integrates correctly into the e2e-vitest-scenarios workflow.

Changes

Common Egress Agent E2E Test Suite and CI Integration

Layer / File(s) Summary
Live E2E test suite: parsing, sandbox management, and agent assertions
test/e2e-scenario/live/common-egress-agent.test.ts		 Implements helpers for OpenClaw JSON parsing, Hermes chat content extraction, sandbox-missing detection, and cleanup registration; NemoClaw onboarding with provider-validation skip detection; policy assertion helpers; OpenClaw agent assertions over SSH with retry and scope-upgrade recovery; Hermes agent assertions with HTTP chat completion; and three sequential live scenarios (C1 balanced weather via Open-Meteo, C2 open reference via Wikidata, C3 Hermes open reference with Nous presets) including policy and token verification.
GitHub Actions workflow job wiring and PR result aggregation
.github/workflows/e2e-vitest-scenarios.yaml		 Adds the common-egress-agent-vitest free-standing job gated by workflow inputs; installs dependencies, OpenShell, runs the test, and uploads artifacts; extends report-to-pr to include common-egress-agent-vitest in its needs list so PR result aggregation includes the job's status.
Workflow boundary validation for common-egress-agent-vitest job
tools/e2e-scenarios/workflow-boundary.mts		 Introduces validateCommonEgressAgentVitestJob enforcing job configuration (runner, timeout, required env flags, secret non-exposure with NVIDIA_API_KEY exemption on the run step, required/pinned steps, vitest targeting common-egress-agent.test.ts, artifact upload naming and path constraints) and integrates it into validateE2eVitestScenariosWorkflowBoundary.
Workflow dispatch selector test coverage
test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts		 Adds assertions validating common-egress-agent dispatch selection behavior when provided via scenarios or jobs, confirming correct free-standing job selection.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5243: Adds a free-standing Vitest job and selector wiring related to the same e2e workflow-selector plumbing used here.
  • NVIDIA/NemoClaw#5231: Similar free-standing Vitest job additions (inference-routing) that exercise the same workflow-boundary and report-to-pr patterns.
  • NVIDIA/NemoClaw#5226: Adds another free-standing Vitest job and updates workflow-boundary validation analogous to this PR.

Suggested labels

area: e2e, area: ci

Suggested reviewers

  • prekshivyas

Poem

🐰 A test-hop here, a sandbox there,
I parse the logs with whiskers fair,
OpenClaw sings and Hermes hums,
Tokens found — the CI drums,
Hooray, the agent hops through air!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'test(e2e): migrate common-egress agent scenario' directly and specifically describes the main change—migrating a legacy e2e test into a new Vitest scenario structure.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/5098-common-egress-agent-e2e

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: None

Workflow run

Full advisor summary

E2E Recommendation Advisor

Failed: Could not parse JSON from advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/e2e-advisor/e2e-advisor-raw-output.txt

@github-actions

github-actions Bot commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: None
Optional Vitest E2E scenarios: None

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Failed: Could not parse JSON from advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/e2e-advisor/e2e-scenario-advisor-raw-output.txt

@github-actions

github-actions Bot commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 1 worth checking, 0 nice ideas
Top item: PR review advisor unavailable

Review findings

🛠️ Needs attention

  • None.

🔎 Worth checking

  • PR review advisor unavailable: The automated advisor could not complete: Could not parse JSON from PR review advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/pr-review-advisor/pr-review-advisor-raw-output.txt
    • Recommendation: Re-run the PR Review Advisor or perform a manual review.
    • Evidence: Could not parse JSON from PR review advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/pr-review-advisor/pr-review-advisor-raw-output.txt

🌱 Nice ideas

  • None.
Consider writing more tests for
  • **Runtime validation** — Add or identify targeted runtime/integration validation for the changed behavior; do not report external E2E job pass/fail here.. Runtime/sandbox/infrastructure paths need behavioral runtime validation: .github/workflows/e2e-vitest-scenarios.yaml, tools/e2e-scenarios/workflow-boundary.mts.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

cv added 2 commits June 12, 2026 12:13
@cv
test(e2e): harden common-egress provider skips
7bcc70b 
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv
test(e2e): cover common-egress parser fallback
1f94611 
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27437547928
Workflow ref: codex/5098-common-egress-agent-e2e
Requested scenarios: (default — all supported)
Requested jobs: common-egress-agent-vitest
Summary: 2 passed, 0 failed, 22 skipped

Job Result
common-egress-agent-vitest ✅ success
credential-migration-vitest ⏭️ skipped
credential-sanitization-vitest ⏭️ skipped
double-onboard-vitest ⏭️ skipped
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
hermes-e2e-vitest ⏭️ skipped
hermes-root-entrypoint-smoke-vitest ⏭️ skipped
inference-routing-vitest ⏭️ skipped
issue-4434-tui-unreachable-inference-vitest ⏭️ skipped
launchable-smoke-vitest ⏭️ skipped
live-scenarios ⏭️ skipped
model-router-provider-routed-inference-vitest ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
rebuild-openclaw-vitest ⏭️ skipped
runtime-overrides-vitest ⏭️ skipped
sandbox-rebuild-vitest ⏭️ skipped
sandbox-survival-vitest ⏭️ skipped
shields-config-vitest ⏭️ skipped
skill-agent-vitest ⏭️ skipped
token-rotation-vitest ⏭️ skipped

@cv cv marked this pull request as ready for review June 12, 2026 21:03
@cv cv added the v0.0.65 Release target label Jun 13, 2026
@cv cv merged commit 894d451 into main Jun 13, 2026
42 checks passed
@cv cv deleted the codex/5098-common-egress-agent-e2e branch June 13, 2026 02:25
@cv cv mentioned this pull request Jun 13, 2026
Open
79 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@cv cv

@jyaunches jyaunches

Labels

v0.0.65 Release target

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cv @jyaunches