fix(ci): bucket cancelled jobs in Selective E2E Results comment by cjagwani · Pull Request #5246 · NVIDIA/NemoClaw

cjagwani · 2026-06-11T17:02:38Z

Summary

The "Selective E2E Results" comment posted by .github/workflows/nightly-e2e.yaml bucketed job results into passed / failed / skipped but never accounted for cancelled. A job ending in cancelled (e.g. when cancel-in-progress kills a stale run) slipped past all three tallies and fell through to the default "✅ All requested jobs passed" status, with the summary line reading "0 passed, 0 failed, 0 skipped" — masking that the run produced no signal at all.

Repro (in the wild)

PR #5085 commit 026802ba8:

### Selective E2E Results — ✅ All requested jobs passed
Run: 27305033204
Requested jobs: agent-turn-latency-e2e,cloud-inference-e2e
Summary: 0 passed, 0 failed, 0 skipped

| Job                    | Result        |
|------------------------|---------------|
| agent-turn-latency-e2e | ⚠️ cancelled |
| cloud-inference-e2e    | ⚠️ cancelled |

Both requested jobs were cancelled (cancel-in-progress superseding the older run) yet the headline read green. Same pattern earlier on #4610.

Root

.github/workflows/nightly-e2e.yaml:2486-2495 (pre-fix):

const passed  = ran.filter(([, v]) => v.result === 'success');
const failed  = ran.filter(([, v]) => v.result === 'failure');
const skipped = reportedEntries.filter(([, v]) => v.result === 'skipped');

const status =
  failed.length > 0 || missingRequested.length > 0 ? '❌ Some jobs failed'
  : skipped.length > 0 && passed.length === 0    ? '⚠️ No requested jobs ran'
  :                                                  '✅ All requested jobs passed';

A cancelled job is !== 'success', !== 'failure', !== 'skipped' — falls through to the default.

Changes

Add a cancelled bucket derived from ran the same way the others are.
Insert a status branch between the failure case and the no-ran case: when cancelled jobs are present and nothing passed, surface ⚠️ Run cancelled — no signal instead of falsely claiming success.
Include cancelled count in the summary line so the bucket is visible even when the other states are zero.

Successful, failed, and skipped runs continue to render exactly as before — only the cancelled case changes from "false green" to "honest yellow."

cc @jyaunches (flagged this earlier in slack)

Summary by CodeRabbit

Tests
- Improved nightly e2e test reporting in PR comments to better distinguish cancelled jobs from other outcomes. PR comments now display cancelled job counts and provide clearer status messaging when test runs are cancelled.

The Selective E2E Results comment script at .github/workflows/nightly-e2e.yaml bucketed job results into passed / failed / skipped but never accounted for `cancelled`. A job ending in `cancelled` (e.g. when cancel-in-progress kills a stale run) slipped past all three tallies and fell through to the default "✅ All requested jobs passed" status, with the summary line reading "0 passed, 0 failed, 0 skipped" — masking the fact that the run produced no signal. Observed on #5085 (the cancel-in-progress run against `026802ba8` listed two cancelled jobs but still posted the green check) and earlier on #4610. Fix: - Add a `cancelled` bucket derived the same way as the other tallies. - Insert a status branch between the failure case and the no-ran case: when cancelled jobs are present and nothing passed, surface "⚠️ Run cancelled — no signal" instead of falsely claiming success. - Include cancelled count in the summary line so the bucket is visible even when other states are zero. No other behavior changes — successful, failed, and skipped runs continue to render exactly as before. Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

coderabbitai · 2026-06-11T17:02:52Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bdd4a4b3-6d69-467c-b2e5-976f4f8c6e20

📥 Commits

Reviewing files that changed from the base of the PR and between 8dc6826 and 73b747b.

📒 Files selected for processing (1)

.github/workflows/nightly-e2e.yaml

📝 Walkthrough

Walkthrough

The workflow now explicitly classifies cancelled E2E test jobs and updates PR comment generation to report cancellation separately. Status logic routes cancellation outcomes distinctly, and the comment summary includes cancelled counts alongside pass/fail/skip totals.

Changes

E2E Test Cancellation Reporting

Layer / File(s)	Summary
Cancelled job reporting in PR comments `.github/workflows/nightly-e2e.yaml`	Status computation derives a separate `cancelled` count from `ran` results and updates the condition to report "Run cancelled — no signal" when there are no passed jobs but cancelled jobs exist. Summary line adds the cancelled count to the reported totals.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

A rabbit hops through workflows bright,
Cancellations caught in light—
No more lost jobs in the night,
Each status now reports just right! 🐰✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title clearly and concisely summarizes the main change: fixing the handling of cancelled jobs in the E2E Results comment by adding them to the results bucket.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/selective-e2e-cancelled-bucket

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-11T17:03:39Z

E2E Advisor Recommendation

Required E2E: None
Optional E2E: None

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

None. No existing product E2E job is needed. This PR only changes CI reporting behavior in .github/workflows/nightly-e2e.yaml; running NemoClaw E2E scenarios would not validate the cancelled-job tally/comment formatting path and would not provide meaningful runtime confidence.

Optional E2E

None.

New E2E recommendations

None.

github-actions · 2026-06-11T17:03:40Z

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: None
Optional Vitest E2E scenarios: None

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

None. Change is limited to .github/workflows/nightly-e2e.yaml, which is outside the Vitest scenario workflow and test/e2e-scenario surfaces; it does not affect Vitest scenario behavior.

Optional Vitest E2E scenarios

None.

Relevant changed files

None.

github-actions · 2026-06-11T17:05:40Z

PR Review Advisor

Findings: 0 needs attention, 2 worth checking, 0 nice ideas
Top item: Treat any cancelled requested job as incomplete in the selective E2E headline

Review findings

🛠️ Needs attention

None.

🔎 Worth checking

Source-of-truth review needed: .github/workflows/nightly-e2e.yaml selective E2E result classification: The advisor marked localized patch analysis as needs_followup.
- Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
- Evidence: The PR adds a cancelled bucket and count, but the headline still only treats cancellation as non-passing when `passed.length === 0`.
Mixed success and cancelled results still render as all passed (.github/workflows/nightly-e2e.yaml:2496): The new status branch only changes the headline when cancelled jobs exist and no requested jobs passed. For a selective dispatch where one requested job succeeds and another is cancelled, `failed.length` and `missingRequested.length` are zero, `passed.length` is nonzero, and the status still falls through to `✅ All requested jobs passed`. That preserves a misleading green headline for an incomplete run, even though the summary/table now include the cancelled job.
- Recommendation: Make cancellation non-passing in the headline whenever `cancelled.length > 0`, or add a distinct partial/incomplete status for mixed passed+cancelled results before the final success branch.
- Evidence: `.github/workflows/nightly-e2e.yaml:2496-2503` uses `cancelled.length > 0 && passed.length === 0 ? '⚠️ Run cancelled — no signal' : ... : '✅ All requested jobs passed'`, so `[success, cancelled]` reaches the final success status.

🌱 Nice ideas

None.

Consider writing more tests for

**Runtime validation** — Selective E2E comment reports `⚠️ Run cancelled — no signal` and `0 passed, 0 failed, 2 cancelled, 0 skipped` when all requested jobs are cancelled.. The changed behavior is embedded in a GitHub Actions reporting script with write-scoped PR comments. A small targeted fixture/helper test or runtime validation would catch misleading status regressions without relying on external E2E status surfaces.
**Runtime validation** — Selective E2E comment does not report `✅ All requested jobs passed` when any requested job is cancelled alongside successful jobs.. The changed behavior is embedded in a GitHub Actions reporting script with write-scoped PR comments. A small targeted fixture/helper test or runtime validation would catch misleading status regressions without relying on external E2E status surfaces.
**Runtime validation** — Selective E2E comment keeps failure and missing-requested precedence when failed or missing jobs are present with cancelled jobs.. The changed behavior is embedded in a GitHub Actions reporting script with write-scoped PR comments. A small targeted fixture/helper test or runtime validation would catch misleading status regressions without relying on external E2E status surfaces.
**Runtime validation** — Selective E2E comment keeps skipped-only status as `⚠️ No requested jobs ran` when all reported requested jobs are skipped and none are cancelled.. The changed behavior is embedded in a GitHub Actions reporting script with write-scoped PR comments. A small targeted fixture/helper test or runtime validation would catch misleading status regressions without relying on external E2E status surfaces.
**Runtime validation** — Selective E2E comment includes cancelled rows with `⚠️ cancelled` for requested selective dispatch results.. The changed behavior is embedded in a GitHub Actions reporting script with write-scoped PR comments. A small targeted fixture/helper test or runtime validation would catch misleading status regressions without relying on external E2E status surfaces.
**Acceptance clause:** Successful, failed, and skipped runs continue to render exactly as before — only the cancelled case changes from "false green" to "honest yellow." — add test evidence or identify existing coverage. The all-cancelled/no-pass case changes to yellow, but mixed success+cancelled is also a cancelled case and still falls through to `✅ All requested jobs passed`.
**.github/workflows/nightly-e2e.yaml selective E2E result classification** — No targeted regression test is present for the selective E2E comment renderer; mixed success+cancelled behavior remains unprotected.. The PR adds a cancelled bucket and count, but the headline still only treats cancellation as non-passing when `passed.length === 0`.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

cjagwani self-assigned this Jun 11, 2026

cjagwani added bug Something fails against expected or documented behavior area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure v0.0.64 Release target labels Jun 11, 2026

cjagwani requested a review from jyaunches June 11, 2026 17:07

jyaunches approved these changes Jun 11, 2026

View reviewed changes

jyaunches merged commit 3ac71fd into main Jun 11, 2026
42 checks passed

jyaunches deleted the fix/selective-e2e-cancelled-bucket branch June 11, 2026 18:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ci): bucket cancelled jobs in Selective E2E Results comment#5246

fix(ci): bucket cancelled jobs in Selective E2E Results comment#5246
jyaunches merged 1 commit into
mainfrom
fix/selective-e2e-cancelled-bucket

cjagwani commented Jun 11, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 11, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

github-actions Bot commented Jun 11, 2026

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Uh oh!

github-actions Bot commented Jun 11, 2026

Vitest E2E Scenario Advisor

Required Vitest E2E scenarios

Optional Vitest E2E scenarios

Relevant changed files

Uh oh!

github-actions Bot commented Jun 11, 2026

🛠️ Needs attention

🔎 Worth checking

🌱 Nice ideas

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cjagwani commented Jun 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Repro (in the wild)

Root

Changes

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

github-actions Bot commented Jun 11, 2026

E2E Advisor Recommendation

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Uh oh!

github-actions Bot commented Jun 11, 2026

Vitest E2E Scenario Recommendation

Vitest E2E Scenario Advisor

Required Vitest E2E scenarios

Optional Vitest E2E scenarios

Relevant changed files

Uh oh!

github-actions Bot commented Jun 11, 2026

PR Review Advisor

🛠️ Needs attention

🔎 Worth checking

🌱 Nice ideas

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cjagwani commented Jun 11, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 11, 2026 •

edited

Loading