E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• gpu-repo-local-ollama-openclaw (high): Required because this is the typed Docker-CDI GPU scenario on the self-hosted GPU runner. It validates repo-current onboarding, Docker-CDI GPU sandbox creation/recreation, local Ollama inference, and Ollama proxy reachability—the exact runtime path changed by the PR.

Optional E2E

• gpu-e2e (high): Useful redundant confidence from the legacy nightly GPU E2E script path. It validates install/onboard plus local Ollama GPU inference on an ephemeral GPU runner, but is less targeted than the typed Docker-CDI scenario.
• gpu-double-onboard-e2e (high): Optional adjacent coverage for repeated GPU onboarding/recreate behavior and persisted Ollama proxy token consistency after re-onboard. It is relevant to lifecycle regressions but not the primary CDI mode-selection fix.

New E2E recommendations

• docker-cdi-gpu-recreate-mode (high): Existing GPU E2E proves the sandbox works, but does not appear to explicitly assert the recreated container used CDI (--device nvidia.com/gpu=all) instead of legacy --gpus all, nor that transient Error phases beyond the old debounce window recover without rollback.
  • Suggested test: Add a focused Docker-CDI GPU recreate regression that captures the patched create option/container inspect output and verifies CDI mode plus successful supervisor reconnect after a transient Error phase.

Dispatch hint

• Workflow: .github/workflows/e2e-scenarios.yaml
• jobs input: gpu-repo-local-ollama-openclaw

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

• gpu-repo-local-ollama-openclaw: Changes affect the Docker GPU patch and supervisor reconnect path, including CDI-first GPU mode selection and reconnect debounce behavior. The dispatchable scenario that exercises Docker CDI GPU onboarding is gpu-repo-local-ollama-openclaw.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=gpu-repo-local-ollama-openclaw

Optional scenario E2E

• None.

Relevant changed files

• src/lib/onboard/docker-gpu-patch.ts
• src/lib/onboard/docker-gpu-supervisor-reconnect.ts

Walkthrough

Reorders GPU patch mode probing to prefer CDI when Docker reports a readable NVIDIA CDI spec, reorganizes related re-exports, and adds tests validating CDI-first selection, fallback to --gpus and nvidia-runtime, and propagation of the CDI device flag into recreate flows.

Changes

CDI-first GPU patch mode candidate selection

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

• [Ubuntu 24.04][Onboard] nemoclaw onboard --gpu always fails with Docker GPU patch supervisor reconnect timeout on aarch64 dual-GPU host #4950: Changes to prefer CDI first and to supervisor-reconnect re-exports affect GPU-mode selection and reconnect behavior referenced by this issue.

Possibly related PRs

• NVIDIA/NemoClaw#4407: Shares related diagnostics and docker-gpu-patch plumbing changes (getSandboxFailurePhase, captureDockerGpuPatchSandboxSnapshot, classifyDockerGpuPatchFailure).

Suggested labels

Docker, platform: container, fix, Sandbox, bug-fix

Suggested reviewers

• cv
• prekshivyas

Poem

🐰 I found a CDI flag bright and small,
I nudged it forward, now it leads the call.
No legacy flags trailing behind the cart,
The sandbox wakes with GPU in its heart.
Hooray — CDI first, a hoppity new start!

❌ Failed checks (1 warning)

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

🛠️ Needs attention

• Linked issue runtime acceptance is still not proven (src/lib/onboard/docker-gpu-patch-mode-selection.test.ts:108): Issue [Ubuntu 24.04][Onboard] onboard cannot create a GPU-enabled sandbox on Docker-driver GPU host #4948 requires a normal first onboard on an affected Docker-CDI GPU host to create a GPU-enabled sandbox, reconnect the OpenShell supervisor, run the GPU proof, reach Ready, and complete onboard. The changed tests prove deterministic CDI-first selection, fallback ordering, CDI argv propagation, and a mocked long Error-phase reconnect recovery, but they still stop at mocked Docker/OpenShell boundaries and do not observe the full first-onboard runtime path.
  • Recommendation: Add or identify targeted runtime/integration validation for a normal first onboard on an affected Ubuntu Docker-CDI GPU host with no existing NemoClaw sandbox, asserting the recreated container uses `--device nvidia.com/gpu=all` rather than `--gpus all`, the supervisor reconnects, the GPU proof runs, the sandbox reaches Ready, and onboard completes.
  • Evidence: The new recreate test calls `recreateOpenShellDockerSandboxWithGpu()` directly with mocked `dockerRunDetached` and mocked `runOpenshell`; the reconnect test simulates `runOpenshell`/`runCaptureOpenshell` responses. The linked issue Expected Result says: "The sandbox is created with GPU access, the OpenShell supervisor reconnects to the GPU-enabled container, the GPU proof runs, the sandbox reaches Ready, and the first onboard completes."

🔎 Worth checking

• Sandbox lifecycle security confidence still depends on real Docker-CDI validation (src/lib/onboard/docker-gpu-patch.ts:509): No direct sandbox escape, credential leak, SSRF bypass, policy bypass, workflow trusted-code-boundary issue, or unsafe shell-string execution issue was found. However, this is a security-sensitive sandbox lifecycle path: the PR changes which GPU device-passthrough mode is used for an OpenShell-managed container and expands the transient Error-phase debounce from about 30 seconds to about 2 minutes. Unit tests validate selection and mocked reconnect behavior, but not the real Docker-CDI container/supervisor/policy lifecycle.
  • Recommendation: Cover the real Docker-CDI host behavior in runtime validation: after CDI recreate, the supervisor reconnects before timeout, the sandbox does not remain in terminal Error during GPU enablement, policy expectations are preserved, the GPU proof runs under the expected container wiring, and persistent failures still roll back to the pre-patch sandbox.
  • Evidence: `buildDockerGpuModeCandidates()` now prepends CDI when `cdiAvailable` is true, and `DOCKER_GPU_SUPERVISOR_RECONNECT_ERROR_PHASE_DEFAULT_DEBOUNCE_POLLS` is now 60. The changed tests assert `dockerRunDetached` receives `--device nvidia.com/gpu=all` and mock supervisor reconnect success/recovery rather than exercising real Docker/OpenShell lifecycle behavior.

🌱 Nice ideas

• None.

• **Runtime validation** — Runtime validation: normal first onboard on an affected Ubuntu Docker-CDI GPU host with no existing NemoClaw sandbox selects `--device nvidia.com/gpu=all`, not `--gpus all`, and onboard completes.. Unit coverage is strong for deterministic mode selection, fallback ordering, Docker argv propagation, and mocked supervisor debounce. The linked bug is a runtime/sandbox infrastructure path involving Docker-CDI hardware, OpenShell supervisor reconnect, GPU proof, Ready phase, and first-onboard completion.
• **Runtime validation** — Runtime validation: after CDI recreate, the OpenShell supervisor reconnects before timeout and `openshell sandbox list` reaches Ready.. Unit coverage is strong for deterministic mode selection, fallback ordering, Docker argv propagation, and mocked supervisor debounce. The linked bug is a runtime/sandbox infrastructure path involving Docker-CDI hardware, OpenShell supervisor reconnect, GPU proof, Ready phase, and first-onboard completion.
• **Runtime validation** — Runtime validation: the direct sandbox GPU proof runs successfully after CDI recreate and records verified CUDA usability.. Unit coverage is strong for deterministic mode selection, fallback ordering, Docker argv propagation, and mocked supervisor debounce. The linked bug is a runtime/sandbox infrastructure path involving Docker-CDI hardware, OpenShell supervisor reconnect, GPU proof, Ready phase, and first-onboard completion.
• **Runtime validation** — Runtime validation: during CDI GPU-enable, a transient Error phase recovers without aborting, while a persistent Error still rolls back to the pre-patch sandbox.. Unit coverage is strong for deterministic mode selection, fallback ordering, Docker argv propagation, and mocked supervisor debounce. The linked bug is a runtime/sandbox infrastructure path involving Docker-CDI hardware, OpenShell supervisor reconnect, GPU proof, Ready phase, and first-onboard completion.
• **Runtime validation** — Composed/unit validation: the `createDockerGpuSandboxCreatePatch` first-onboard create-time path wires CDI detection through `recreateOpenShellDockerSandboxWithGpu`, not only direct calls to the recreate helper.. Unit coverage is strong for deterministic mode selection, fallback ordering, Docker argv propagation, and mocked supervisor debounce. The linked bug is a runtime/sandbox infrastructure path involving Docker-CDI hardware, OpenShell supervisor reconnect, GPU proof, Ready phase, and first-onboard completion.
• **Acceptance clause:** On a Docker-driver GPU host (NVIDIA GPU auto-detected), `nemoclaw onboard` cannot bring up a GPU-enabled sandbox. — add test evidence or identify existing coverage. The production change targets Docker GPU patch mode selection, but no changed test runs `nemoclaw onboard` on a Docker-driver GPU host.
• **Acceptance clause:** While creating the sandbox, onboard enables GPU passthrough — this is the standard create-then-GPU-enable path that runs on a normal FIRST onboard whenever a GPU is present on a Docker-driver gateway (gated by NEMOCLAW_DOCKER_GPU_PATCH); — add test evidence or identify existing coverage. The new CDI test directly calls `recreateOpenShellDockerSandboxWithGpu()` and proves CDI args reach `dockerRunDetached`; it does not exercise the full first-onboard create-time orchestration with no existing NemoClaw sandbox.
• **Acceptance clause:** The OpenShell supervisor never reconnects to the GPU-enabled container, — add test evidence or identify existing coverage. The reconnect test simulates a transient Error phase followed by mocked supervisor success, but no changed test observes a real OpenShell supervisor reconnecting to a real Docker-CDI GPU-enabled container.

Current findings:

• Linked issue runtime acceptance is still not proven (src/lib/onboard/docker-gpu-patch-mode-selection.test.ts:108): Issue [Ubuntu 24.04][Onboard] onboard cannot create a GPU-enabled sandbox on Docker-driver GPU host #4948 requires a normal first onboard on an affected Docker-CDI GPU host to create a GPU-enabled sandbox, reconnect the OpenShell supervisor, run the GPU proof, reach Ready, and complete onboard. The changed tests prove deterministic CDI-first selection, fallback ordering, CDI argv propagation, and a mocked long Error-phase reconnect recovery, but they still stop at mocked Docker/OpenShell boundaries and do not observe the full first-onboard runtime path.
  • Recommendation: Add or identify targeted runtime/integration validation for a normal first onboard on an affected Ubuntu Docker-CDI GPU host with no existing NemoClaw sandbox, asserting the recreated container uses `--device nvidia.com/gpu=all` rather than `--gpus all`, the supervisor reconnects, the GPU proof runs, the sandbox reaches Ready, and onboard completes.
  • Evidence: The new recreate test calls `recreateOpenShellDockerSandboxWithGpu()` directly with mocked `dockerRunDetached` and mocked `runOpenshell`; the reconnect test simulates `runOpenshell`/`runCaptureOpenshell` responses. The linked issue Expected Result says: "The sandbox is created with GPU access, the OpenShell supervisor reconnects to the GPU-enabled container, the GPU proof runs, the sandbox reaches Ready, and the first onboard completes."
• Sandbox lifecycle security confidence still depends on real Docker-CDI validation (src/lib/onboard/docker-gpu-patch.ts:509): No direct sandbox escape, credential leak, SSRF bypass, policy bypass, workflow trusted-code-boundary issue, or unsafe shell-string execution issue was found. However, this is a security-sensitive sandbox lifecycle path: the PR changes which GPU device-passthrough mode is used for an OpenShell-managed container and expands the transient Error-phase debounce from about 30 seconds to about 2 minutes. Unit tests validate selection and mocked reconnect behavior, but not the real Docker-CDI container/supervisor/policy lifecycle.
  • Recommendation: Cover the real Docker-CDI host behavior in runtime validation: after CDI recreate, the supervisor reconnects before timeout, the sandbox does not remain in terminal Error during GPU enablement, policy expectations are preserved, the GPU proof runs under the expected container wiring, and persistent failures still roll back to the pre-patch sandbox.
  • Evidence: `buildDockerGpuModeCandidates()` now prepends CDI when `cdiAvailable` is true, and `DOCKER_GPU_SUPERVISOR_RECONNECT_ERROR_PHASE_DEFAULT_DEBOUNCE_POLLS` is now 60. The changed tests assert `dockerRunDetached` receives `--device nvidia.com/gpu=all` and mock supervisor reconnect success/recovery rather than exercising real Docker/OpenShell lifecycle behavior.