ci(e2e): clarify scenario dry-run summary by cv · Pull Request #4656 · NVIDIA/NemoClaw

cv · 2026-06-02T07:35:18Z

Summary

Clarifies the GitHub Step Summary emitted by the scenario workflow so maintainers can see that it ran the typed dry-run path. The summary now renders scenario, manifest, expected state, suite, and phase data from the typed run-plan artifact.

Related Issue

Refs #3588.

Changes

Rename the summary step to Append typed dry-run summary.
Add an explicit typed dry-run mode note to the workflow summary.
Render .e2e/run-plan.json as a compact markdown table with fallback text when no plan artifact exists.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Verification

npx prek run --all-files passes
npm test passes
Tests added or updated for new or changed behavior
No secrets, API keys, or credentials committed
Docs updated for user-facing behavior changes
npm run docs builds without warnings (doc changes only)
Doc pages follow the style guide (doc changes only)
New doc pages include SPDX header and frontmatter (new pages only)

Additional verification run:

npx prek run --files .github/workflows/e2e-scenarios.yaml
npx vitest run --project e2e-scenario-framework test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts --silent=false --reporter=default

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

Chores
- Enhanced E2E scenario test reporting in CI workflows. The test summary now displays comprehensive scenario execution details in a structured Markdown table format when available, including scenario identifiers, test suite information, and execution phases. Improved fallback options gracefully handle missing or unavailable structured artifacts.

copy-pr-bot · 2026-06-02T07:35:21Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-06-02T07:35:26Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fbd8d8e9-136c-4635-8f79-3ee8c8987bc2

📥 Commits

Reviewing files that changed from the base of the PR and between a25d5b3 and 0491bad.

📒 Files selected for processing (1)

.github/workflows/e2e-scenarios.yaml

📝 Walkthrough

Walkthrough

The E2E scenario workflow step that appends artifacts to the GitHub step summary was replaced. It now writes a "typed scenario dry-run" section, renders a structured Markdown table from .e2e/run-plan.json when available, falls back to .e2e/plan.txt, and handles missing artifacts gracefully.

Changes

E2E Scenario Dry-Run Summary

Layer / File(s)	Summary
E2E Scenario Dry-Run Summary Step `.github/workflows/e2e-scenarios.yaml`	The workflow step that appends E2E scenario artifacts to the GitHub step summary now supports typed dry-run output. It generates a Markdown table from `.e2e/run-plan.json` (including scenario id, manifest path, expected state, suites, and phase names), falls back to fenced `.e2e/plan.txt` output if the JSON artifact is missing, and emits a "no typed scenario plan artifact" message when neither file exists.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested labels

enhancement

Suggested reviewers

prekshivyas
cjagwani
jyaunches

Poem

🐰 A workflow step hops with joy today,
With typed dry-runs in a structured way!
Tables bloom where JSON flows so clear,
Fallback plans for files that may not appear—
E2E summaries, polished and bright! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'ci(e2e): clarify scenario dry-run summary' directly summarizes the main change—improving clarity of the GitHub Step Summary for scenario workflow dry-runs.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch ci/e2e-scenario-dry-run-summary

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-02T07:35:55Z

PR Review Advisor

Findings: 0 needs attention, 4 worth checking, 0 nice ideas
Since last review: 0 prior items resolved, 3 still apply, 0 new items found

Review findings

🛠️ Needs attention

None.

🔎 Worth checking

Source-of-truth review needed: Append typed dry-run summary fallback from `.e2e/run-plan.json` to `.e2e/plan.txt` or no-artifact message: The advisor marked localized patch analysis as needs_followup.
- Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
- Evidence: The workflow checks `[ -f .e2e/run-plan.json ]`, then `[ -f .e2e/plan.txt ]`, then emits `_No typed scenario plan artifact was produced._`; `compiler.ts` writes both artifacts in normal operation.
Harden run-plan summary parsing and Markdown rendering (.github/workflows/e2e-scenarios.yaml:338): The new always-run summary step parses `.e2e/run-plan.json` and writes plan fields directly into a GitHub Markdown table. Malformed JSON, non-list JSON, non-object plan entries, or non-object phase entries can fail the reporting step after scenario execution. Values containing Markdown table delimiters, newlines, or backticks can also break or misrepresent the Step Summary.
- Recommendation: Validate the JSON shape before iterating, catch parse/type errors and emit a bounded warning or fenced fallback instead of failing the summary step, and escape Markdown table metacharacters before writing plan-derived values to `$GITHUB_STEP_SUMMARY`.
- Evidence: The inline Python uses `json.loads(Path('.e2e/run-plan.json').read_text(...))`, iterates `for plan in plans`, calls `plan.get(...)` and `phase.get('name')`, and formats those values directly into Markdown table cells.
Clarify the source of truth for the plan artifact fallback (.github/workflows/e2e-scenarios.yaml:337): This PR adds localized compatibility/fallback behavior for missing `.e2e/run-plan.json`, legacy `.e2e/plan.txt`, and the no-artifact case, but the codebase already has a canonical writer that emits both artifacts. The invalid state, why it cannot be fixed at the artifact source, and when the fallback can be removed are not established by code or trusted issue context.
- Recommendation: Either make the canonical artifact contract explicit and diagnose source artifact creation failures, or document/test the compatibility reason for the `.e2e/plan.txt` and no-artifact fallback paths, including their removal condition.
- Evidence: `test/e2e-scenario/scenarios/compiler.ts` writes both `.e2e/run-plan.json` and `.e2e/plan.txt` in `writePlanArtifacts()`, while the workflow now checks JSON, then text, then prints `_No typed scenario plan artifact was produced._`.
Add regression coverage for the new summary behavior (test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts:20): The changed behavior is deterministic and testable, but nearby tests only validate workflow boundary properties and plan artifact creation. They do not assert the new summary step, the JSON table rendering, the legacy text fallback, no-artifact output, malformed JSON handling, or Markdown escaping.
- Recommendation: Add deterministic tests or extract a small renderer and cover valid JSON table output, fenced `.e2e/plan.txt` fallback, no-artifact messaging, malformed/unexpected JSON tolerance, Markdown escaping, and a workflow-boundary assertion requiring the `Append typed dry-run summary` step.
- Evidence: `e2e-scenarios-workflow.test.ts` checks triggers, permissions, runner routing, WSL steps, and artifact upload. `e2e-plan-compiler.test.ts` checks that `.e2e/run-plan.json` and `.e2e/plan.txt` are emitted. Neither test file exercises the new summary-rendering branch added to `.github/workflows/e2e-scenarios.yaml`.

🌱 Nice ideas

None.

Consider writing more tests for

**Add regression coverage for the new summary behavior** — Add deterministic tests or extract a small renderer and cover valid JSON table output, fenced `.e2e/plan.txt` fallback, no-artifact messaging, malformed/unexpected JSON tolerance, Markdown escaping, and a workflow-boundary assertion requiring the `Append typed dry-run summary` step.
**Acceptance clause:** Refs Implement layered E2E scenario model #3588. — add test evidence or identify existing coverage. The deterministic context did not include linked issue Implement layered E2E scenario model #3588 body or comments, so literal issue acceptance clauses could not be mapped.
**Acceptance clause:** Render `.e2e/run-plan.json` as a compact markdown table with fallback text when no plan artifact exists. — add test evidence or identify existing coverage. The workflow renders `.e2e/run-plan.json` as a Markdown table and prints `_No typed scenario plan artifact was produced._` when neither JSON nor text exists. However, it also adds an untested `.e2e/plan.txt` fallback and does not handle malformed or unexpected JSON gracefully.
**Append typed dry-run summary fallback from `.e2e/run-plan.json` to `.e2e/plan.txt` or no-artifact message** — Missing for the workflow summary fallback semantics. Existing tests assert artifact creation and workflow boundary safety, but not JSON rendering, `plan.txt` fallback, no-artifact output, malformed JSON tolerance, or Markdown escaping.. The workflow checks `[ -f .e2e/run-plan.json ]`, then `[ -f .e2e/plan.txt ]`, then emits `_No typed scenario plan artifact was produced._`; `compiler.ts` writes both artifacts in normal operation.

Since last review details

Current findings:

Source-of-truth review needed: Append typed dry-run summary fallback from `.e2e/run-plan.json` to `.e2e/plan.txt` or no-artifact message: The advisor marked localized patch analysis as needs_followup.
- Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
- Evidence: The workflow checks `[ -f .e2e/run-plan.json ]`, then `[ -f .e2e/plan.txt ]`, then emits `_No typed scenario plan artifact was produced._`; `compiler.ts` writes both artifacts in normal operation.
Harden run-plan summary parsing and Markdown rendering (.github/workflows/e2e-scenarios.yaml:338): The new always-run summary step parses `.e2e/run-plan.json` and writes plan fields directly into a GitHub Markdown table. Malformed JSON, non-list JSON, non-object plan entries, or non-object phase entries can fail the reporting step after scenario execution. Values containing Markdown table delimiters, newlines, or backticks can also break or misrepresent the Step Summary.
- Recommendation: Validate the JSON shape before iterating, catch parse/type errors and emit a bounded warning or fenced fallback instead of failing the summary step, and escape Markdown table metacharacters before writing plan-derived values to `$GITHUB_STEP_SUMMARY`.
- Evidence: The inline Python uses `json.loads(Path('.e2e/run-plan.json').read_text(...))`, iterates `for plan in plans`, calls `plan.get(...)` and `phase.get('name')`, and formats those values directly into Markdown table cells.
Clarify the source of truth for the plan artifact fallback (.github/workflows/e2e-scenarios.yaml:337): This PR adds localized compatibility/fallback behavior for missing `.e2e/run-plan.json`, legacy `.e2e/plan.txt`, and the no-artifact case, but the codebase already has a canonical writer that emits both artifacts. The invalid state, why it cannot be fixed at the artifact source, and when the fallback can be removed are not established by code or trusted issue context.
- Recommendation: Either make the canonical artifact contract explicit and diagnose source artifact creation failures, or document/test the compatibility reason for the `.e2e/plan.txt` and no-artifact fallback paths, including their removal condition.
- Evidence: `test/e2e-scenario/scenarios/compiler.ts` writes both `.e2e/run-plan.json` and `.e2e/plan.txt` in `writePlanArtifacts()`, while the workflow now checks JSON, then text, then prints `_No typed scenario plan artifact was produced._`.
Add regression coverage for the new summary behavior (test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts:20): The changed behavior is deterministic and testable, but nearby tests only validate workflow boundary properties and plan artifact creation. They do not assert the new summary step, the JSON table rendering, the legacy text fallback, no-artifact output, malformed JSON handling, or Markdown escaping.
- Recommendation: Add deterministic tests or extract a small renderer and cover valid JSON table output, fenced `.e2e/plan.txt` fallback, no-artifact messaging, malformed/unexpected JSON tolerance, Markdown escaping, and a workflow-boundary assertion requiring the `Append typed dry-run summary` step.
- Evidence: `e2e-scenarios-workflow.test.ts` checks triggers, permissions, runner routing, WSL steps, and artifact upload. `e2e-plan-compiler.test.ts` checks that `.e2e/run-plan.json` and `.e2e/plan.txt` are emitted. Neither test file exercises the new summary-rendering branch added to `.github/workflows/e2e-scenarios.yaml`.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

github-actions · 2026-06-02T07:37:04Z

E2E Advisor Recommendation

Required E2E: None
Optional E2E: ubuntu-repo-cloud-openclaw

Dispatch hint: ubuntu-repo-cloud-openclaw

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

None. No merge-blocking E2E is required because the only change is to CI summary/reporting in the scenario workflow after scenario execution; it cannot affect NemoClaw runtime or user assistant flows.

Optional E2E

ubuntu-repo-cloud-openclaw (low): Optional validation of the modified E2E / Scenario Runner workflow path and the new typed dry-run summary rendering on the default Ubuntu runner. This directly exercises the changed workflow step without being merge-blocking for runtime behavior.

New E2E recommendations

None.

Dispatch hint

Workflow: .github/workflows/e2e-scenarios.yaml
jobs input: ubuntu-repo-cloud-openclaw

github-actions · 2026-06-02T07:37:05Z

E2E Scenario Advisor Recommendation

Required scenario E2E: e2e-scenarios-all
Optional scenario E2E: None

Dispatch required scenario E2E:

gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

e2e-scenarios-all: The PR changes the scenario runner workflow itself. Per scenario E2E policy, workflow changes require the full scenario fan-out to validate dispatch, runner routing, summaries, and artifact behavior across all scenario routes.
- Dispatch: gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Optional scenario E2E

None.

Relevant changed files

.github/workflows/e2e-scenarios.yaml

wscurran · 2026-06-03T18:11:15Z

✨
Related open issues:

#3588 Implement layered E2E scenario model

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

cv added 4 commits June 1, 2026 23:42

test(e2e): add scenario aliases for layered plans

dbc672f

docs(e2e): reconcile scenario migration docs

c66d2a3

test(e2e): guard scenario workflow drift

401c631

ci(e2e): clarify scenario dry-run summary

a93d8cb

cv self-assigned this Jun 2, 2026

wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure labels Jun 3, 2026

Base automatically changed from test/e2e-scenario-drift-guards to main June 7, 2026 14:39

merge(main): resolve PR 4656 conflicts

0491bad

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

cv marked this pull request as ready for review June 7, 2026 20:54

Merge branch 'main' into ci/e2e-scenario-dry-run-summary

9fd7046

cv merged commit 13405a5 into main Jun 7, 2026
31 checks passed

cv deleted the ci/e2e-scenario-dry-run-summary branch June 7, 2026 21:22

cv added the v0.0.61 Release target label Jun 8, 2026

coderabbitai Bot mentioned this pull request Jun 8, 2026

test(e2e): execute real shell assertions; delete dry-run, --validate-only, and the bash runner #4380

Merged

wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 8, 2026

Conversation

cv commented Jun 2, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Type of Change

Verification

Summary by CodeRabbit

Uh oh!

copy-pr-bot Bot commented Jun 2, 2026

Uh oh!

coderabbitai Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

Poem

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review Advisor

🛠️ Needs attention

🔎 Worth checking

🌱 Nice ideas

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Advisor Recommendation

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Dispatch hint

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Scenario Advisor Recommendation

E2E Scenario Advisor

Required scenario E2E

Optional scenario E2E

Relevant changed files

Uh oh!

wscurran commented Jun 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cv commented Jun 2, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading