Skip to content

Nightly E2E: triage & fix plan for 3 consistent failures on main #2104

@cjagwani

Description

@cjagwani

The nightly E2E workflow has been failing on main every night for the last 5+ consecutive runs. Auto-created notifier #1992 reports the event; this issue is the fix plan.

Failing jobs

  1. cloud-experimental-e2e04-landlock-readonly.sh checks 1–3 fail: /sandbox, .bashrc, .profile are writable. Root cause tracked in [All Platform][Security]OpenShell 0.0.26 does not enforce Landlock filesystem policy — /sandbox writable on all platforms #1739 (OpenShell 0.0.26 does not apply Landlock ruleset to sandbox process). Policy fix in progress upstream — out of scope for this issue, will clear once the OpenShell version pin in nemoclaw-blueprint/blueprint.yaml and scripts/install-openshell.sh is bumped to the fixed release.

  2. token-rotation-e2e — hits the 15min job timeout (exit 124) during Phase 2 re-onboard. Silent hang — output is buffered in a $(…) capture with no per-step timeout, so the real error isn't visible.

  3. snapshot-commands-e2e — Phase 7 snapshot restore fails due to an ambiguous timestamp prefix-match when two snapshots are created within the same second.

Fix plan (in scope for this issue)

  • Snapshot: add --format json to nemoclaw <sandbox> snapshot list, use exact-ID match in the test, and fix the silent-exit pattern so future failures aren't invisible.
  • Token-rotation: add streaming + bounded timeout around the Phase 2 onboard call so the real error surfaces. Quarantine the job with continue-on-error: true while we diagnose, then remove once the root cause is fixed.

Landlock work is handed off — see #1739.

Links: #1992 (noise tracker) · #1739 (Landlock regression, upstream) · workflow run 24642489117

Metadata

Metadata

Assignees

Labels

area: ciCI workflows, checks, release automation, or GitHub Actionsarea: e2eEnd-to-end tests, nightly failures, or validation infrastructuresecurityPotential vulnerability, unsafe behavior, or access risk

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions