Summary

Four E2E test scripts exist in test/e2e/ but have never been wired into the nightly workflow or any automated CI job. They were created between March 19 and April 14, are actively maintained (last touched Apr 23 for the e2e-timeout refactor and sandbox teardown reliability work), and have never appeared in .github/workflows/nightly-e2e.yaml at any point in git history.

These scripts are fully self-contained — they install NemoClaw, run nemoclaw onboard, execute test cases, and clean up. They require NVIDIA_API_KEY (available in nightly via repository secrets) and NEMOCLAW_NON_INTERACTIVE=1.

Scripts to wire

Context

Analysis of the last 3 weeks of bug fix PRs (Apr 7–27) shows an E2E catch rate of ~17% — for every bug the nightly catches, roughly 5 more reach NV QA or community users first. The rebuild/lifecycle and onboard-resume domains account for 6 bugs found by humans that these existing scripts would have caught if they were running nightly.

In particular:

• Rebuild bugs (fix(rebuild): forward stored --from Dockerfile path to onboard on rebuild #2302, fix(rebuild): make rebuild atomic with credential preflight and recovery (#2273) #2366, nemoclaw rebuild builds the wrong sandbox type and deletes existing sandbox! #2201) — test-sandbox-rebuild.sh tests the exact --from Dockerfile and atomic-rebuild scenarios that broke for users. All 3 bugs were reported by NV QA (zNeill) or community (oparoz).
• Session resume bugs ([NemoClaw][brev][Onboard] Re-running install.sh after failed Ollama onboard auto-resumes stale session — user cannot change provider #2430) — test-onboard-resume.sh tests the interrupted-onboard-then-resume flow. Issue [NemoClaw][brev][Onboard] Re-running install.sh after failed Ollama onboard auto-resumes stale session — user cannot change provider #2430 (stale session blocks provider change) was reported by NV QA and required PR fix(install): refuse to auto-resume a failed onboarding session #2437 to fix.
• Multi-sandbox / registry drift (fix(cli): preserve sandbox registry when active gateway drifts (#2276) #2330) — test-double-onboard.sh tests the gateway-drift scenario where openshell gateway stop && start switches the active gateway, causing nemoclaw connect to destroy the registry entry.

Implementation

Each script needs a new job in .github/workflows/nightly-e2e.yaml following the existing pattern (see cloud-e2e as template):

  double-onboard-e2e:
    if: github.repository == 'NVIDIA/NemoClaw'
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Checkout
        uses: actions/checkout@v6
      - name: Run double-onboard E2E test
        env:
          NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
          NEMOCLAW_NON_INTERACTIVE: "1"
          NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
        run: bash test/e2e/test-double-onboard.sh
      - name: Upload logs on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: double-onboard-log
          path: /tmp/nemoclaw-e2e-*.log
          if-no-files-found: ignore

Repeat for sandbox-rebuild-e2e, onboard-resume-e2e, and onboard-repair-e2e.

Also add all four job names to the notify-on-failure job's needs: list so failures auto-create issues.

Acceptance criteria

• All 4 scripts run as nightly jobs
• All 4 are in the notify-on-failure needs: list
• First nightly run after merge shows all 4 jobs (pass or fail — if they fail, the nightly catches the regression immediately)