E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• sandbox-operations-e2e (high): Required because this PR changes the core sandbox status command used in live sandbox operations. This existing E2E creates/manages real sandboxes and includes the sandbox operations validation path that checks nemoclaw list and nemoclaw <sandbox> status against a running sandbox, catching regressions in the refactored text status path and gateway/sandbox lookup behavior.

Optional E2E

• diagnostics-e2e (medium): Useful adjacent coverage because diagnostics validates host-side nemoclaw <sandbox> status output includes model/provider information in a live onboarded environment. This can catch regressions in status snapshot model/provider resolution, but it is not as directly targeted as sandbox-operations-e2e.
• inference-routing-e2e (medium): Optional confidence for the inference-health side of the JSON report. The changed status snapshot calls provider health probes and exposes inference health; inference-routing-e2e exercises provider routing and health/error-classification behavior, but the PR does not directly change route configuration.
• docs-validation-e2e (low): Optional validation for the updated command reference and command metadata. PR-level docs/CLI parity should already cover much of this, but this E2E can provide extra confidence that installed CLI docs validation still passes.

New E2E recommendations

• sandbox status JSON live contract (high): Existing E2E coverage appears to exercise text nemoclaw <sandbox> status, not the new --json machine-readable contract against a live sandbox. Add a focused E2E validation step that runs nemoclaw <sandbox> status --json and nemoclaw sandbox status <sandbox> --json, parses JSON, asserts schemaVersion, found:true, gatewayState:present, model/provider fields, string openshellDriver/openshellVersion, no text renderer output, and expected exit codes.
  • Suggested test: Extend sandbox-operations-e2e with a new validation step such as test/e2e/validation_suites/sandbox/operations/02-status-json.sh, or add equivalent assertions to test/e2e/test-sandbox-operations.sh.
• status JSON missing-sandbox automation contract (medium): The documented canonical form should emit JSON with found:false and exit non-zero for an unknown sandbox. Unit tests cover this with mocked OpenShell, but no live/installed CLI E2E appears to verify the automation contract after install.
  • Suggested test: Add a sandbox-operations E2E assertion for nemoclaw sandbox status definitely-missing --json that checks parseable JSON, found:false, gatewayState != "present", and exit code 1.
• status JSON redaction boundary (high): The change explicitly routes JSON through centralized redaction to prevent token-shaped values in inferenceHealth endpoint/detail/subprobes. Existing credential-sanitization E2E does not appear to target status JSON output.
  • Suggested test: Add an E2E or regression script that configures a mock/local provider health failure containing token-shaped strings and asserts nemoclaw <sandbox> status --json redacts them before output.

Dispatch hint

• Workflow: E2E / Nightly
• jobs input: sandbox-operations-e2e

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: medium

Required scenario E2E

• ubuntu-repo-cloud-openclaw: The PR changes the sandbox status command/action implementation. The ubuntu repo cloud OpenClaw scenario is the smallest standard scenario that exercises nemoclaw <sandbox> status through the baseline-onboarding suite against a real onboarded sandbox.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Optional scenario E2E

• ubuntu-repo-cloud-hermes: Optional adjacent coverage for the same sandbox status path with a Hermes-agent sandbox, including the Hermes-specific status-side setup path.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-hermes
• brev-launchable-cloud-openclaw: Optional remote/launchable coverage for the same sandbox status behavior in the Brev launchable scenario; kept optional because it uses special remote provisioning and is not the primary path.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=brev-launchable-cloud-openclaw

Relevant changed files

• src/commands/sandbox/status.ts
• src/lib/actions/sandbox/status.ts

Walkthrough

Adds structured JSON output to per-sandbox nemoclaw <name> status: a SandboxStatusReport and getSandboxStatusReport() that collect gateway/inference/runtime metadata; CLI wiring to enable --json and return the report with exit-code semantics; tests for JSON shape and failure cases; and docs updated to document --json behavior.

Changes

Per-sandbox status JSON output support

Sequence Diagram

sequenceDiagram
  participant CLI
  participant SandboxStatusCommand
  participant StatusLib
  participant Gateway
  participant OpenShell
  participant Registry

  CLI->>SandboxStatusCommand: nemoclaw <name> status --json
  SandboxStatusCommand->>StatusLib: getSandboxStatusReport(name)
  StatusLib->>Gateway: reconcile gateway state
  Gateway-->>StatusLib: gatewayState / gateway metadata
  alt gateway present
    StatusLib->>OpenShell: inference.get() probe
    OpenShell-->>StatusLib: inference output or rpcIssue
    StatusLib->>Registry: lookup model/provider fallback
    Registry-->>StatusLib: registry metadata
  end
  StatusLib-->>SandboxStatusCommand: SandboxStatusReport
  SandboxStatusCommand->>CLI: emit JSON (redacted) and set exitCode if found=false or gatewayState != "present" or rpcIssue present

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• NVIDIA/NemoClaw#4180: Touches src/lib/actions/sandbox/status.ts and adds Docker health probing/output that overlaps status code paths.

Suggested labels

documentation, Sandbox

Suggested reviewers

• ericksoa
• cv
• jyaunches
• prekshivyas

🐰
I hop with code and carrot cheer,
Sandbox status now speaks clear.
GPUs, drivers, health in view,
JSON ready — tidy and true.
Parse away and automate here!

❌ Failed checks (1 warning)

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

🛠️ Needs attention

• Status monolith still grows past the hotspot threshold (src/lib/actions/sandbox/status.ts:64): The PR adds the shared snapshot type, collector, JSON report shaping, and local-provider gateway-chain subprobe wiring inside the already large and high-churn sandbox status action module. The deterministic drift check reports this file grew from 439 to 514 lines (+75), above the repository hotspot threshold for sandbox/gateway lifecycle code.
  • Recommendation: Extract the shared status snapshot/report collection into a focused module, or offset the growth by moving rendering/report-shaping helpers out of src/lib/actions/sandbox/status.ts. Keep text and JSON renderers as thin consumers of the same validated snapshot.
  • Evidence: The diff adds SandboxStatusReport, SandboxStatusSnapshot, collectSandboxStatusSnapshot(), getSandboxStatusReport(), and JSON-specific shaping in src/lib/actions/sandbox/status.ts; deterministic monolithDeltas reports baseLines=439, headLines=514, delta=75.

🔎 Worth checking

• Source-of-truth review needed: collectSandboxStatusSnapshot() gateway reconciler fallback: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: src/lib/actions/sandbox/status.ts catches err around getReconciledSandboxGatewayState() and synthesizes { state: 'gateway_error', output: ` Could not probe live gateway state: ${message}` }.
• Source-of-truth review needed: collectSandboxStatusSnapshot() inference capture fallback: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: src/lib/actions/sandbox/status.ts catches inference capture errors with catch { liveResult = null; }, then falls back to registry model/provider and may compute inferenceHealth.
• Gateway reconciliation throw fallback is now a JSON contract without a pinned source-of-truth test (src/lib/actions/sandbox/status.ts:103): collectSandboxStatusSnapshot() catches any exception from getReconciledSandboxGatewayState() and synthesizes gatewayState='gateway_error'. That preserved the legacy text-output behavior, but this PR exposes the synthesized state through machine-readable JSON without documenting why the reconciler cannot return a typed state itself, what exact invalid state is expected, or when the localized fallback can be removed.
  • Recommendation: Prefer normalizing expected probe failures inside the gateway-state layer. If this localized fallback remains, add a per-sandbox --json regression test that forces the reconciler/probe boundary to throw and asserts gatewayState='gateway_error', found/model/provider behavior, and the exit code.
  • Evidence: src/lib/actions/sandbox/status.ts catches err around getReconciledSandboxGatewayState() and builds { state: 'gateway_error', output: ` Could not probe live gateway state: ${message}` }; added tests cover missing and non-present gateway states but do not force this catch path.
• Inference capture failure can emit registry-backed JSON without an explicit unverified marker (src/lib/actions/sandbox/status.ts:114): When the gateway is present but captureOpenshellForStatus(["inference", "get"]) throws, the collector silently sets liveResult=null, falls back to registry model/provider, and can still compute provider health. JSON consumers may treat those fields as live gateway-verified status even though live inference configuration could not be read.
  • Recommendation: Define the JSON contract for inference capture failure. Prefer an explicit live-inference-unavailable marker or set inferenceHealth=null when live inference capture fails, then add a regression test proving the body and exit code for this path.
  • Evidence: The collector has catch { liveResult = null; }, then computes currentModel/currentProvider from live || registry || 'unknown' and calls getSandboxStatusInferenceHealth(lookup.state === 'present', currentProvider, currentModel). No added test forces captureOpenshellForStatus(["inference", "get"]) to throw.
• Per-sandbox diagnostic redaction is implemented but not pinned for nested health fields (src/commands/sandbox/status.ts:27): The JSON path now routes the report through redactForLog(), which addresses the main credential-leakage risk. However, this new machine-readable output can include inferenceHealth.endpoint, detail, and nested subprobes that automation may persist, and the added tests do not inject token-shaped values into those per-sandbox health fields to prove the redaction contract stays intact.
  • Recommendation: Add a focused per-sandbox status --json test that injects token-like values into inferenceHealth.endpoint, inferenceHealth.detail, and nested subprobes, then asserts the output is parseable JSON and contains only redacted values.
  • Evidence: SandboxStatusCommand.run() returns redactForLog(report), and SandboxStatusReport includes inferenceHealth: ProviderHealthStatus | null. Existing credential-redaction coverage is for the global status --json path, not the new per-sandbox report with health subprobes.

🌱 Nice ideas

• None.

Current findings:

• Source-of-truth review needed: collectSandboxStatusSnapshot() gateway reconciler fallback: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: src/lib/actions/sandbox/status.ts catches err around getReconciledSandboxGatewayState() and synthesizes { state: 'gateway_error', output: ` Could not probe live gateway state: ${message}` }.
• Source-of-truth review needed: collectSandboxStatusSnapshot() inference capture fallback: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: src/lib/actions/sandbox/status.ts catches inference capture errors with catch { liveResult = null; }, then falls back to registry model/provider and may compute inferenceHealth.
• Status monolith still grows past the hotspot threshold (src/lib/actions/sandbox/status.ts:64): The PR adds the shared snapshot type, collector, JSON report shaping, and local-provider gateway-chain subprobe wiring inside the already large and high-churn sandbox status action module. The deterministic drift check reports this file grew from 439 to 514 lines (+75), above the repository hotspot threshold for sandbox/gateway lifecycle code.
  • Recommendation: Extract the shared status snapshot/report collection into a focused module, or offset the growth by moving rendering/report-shaping helpers out of src/lib/actions/sandbox/status.ts. Keep text and JSON renderers as thin consumers of the same validated snapshot.
  • Evidence: The diff adds SandboxStatusReport, SandboxStatusSnapshot, collectSandboxStatusSnapshot(), getSandboxStatusReport(), and JSON-specific shaping in src/lib/actions/sandbox/status.ts; deterministic monolithDeltas reports baseLines=439, headLines=514, delta=75.
• Gateway reconciliation throw fallback is now a JSON contract without a pinned source-of-truth test (src/lib/actions/sandbox/status.ts:103): collectSandboxStatusSnapshot() catches any exception from getReconciledSandboxGatewayState() and synthesizes gatewayState='gateway_error'. That preserved the legacy text-output behavior, but this PR exposes the synthesized state through machine-readable JSON without documenting why the reconciler cannot return a typed state itself, what exact invalid state is expected, or when the localized fallback can be removed.
  • Recommendation: Prefer normalizing expected probe failures inside the gateway-state layer. If this localized fallback remains, add a per-sandbox --json regression test that forces the reconciler/probe boundary to throw and asserts gatewayState='gateway_error', found/model/provider behavior, and the exit code.
  • Evidence: src/lib/actions/sandbox/status.ts catches err around getReconciledSandboxGatewayState() and builds { state: 'gateway_error', output: ` Could not probe live gateway state: ${message}` }; added tests cover missing and non-present gateway states but do not force this catch path.
• Inference capture failure can emit registry-backed JSON without an explicit unverified marker (src/lib/actions/sandbox/status.ts:114): When the gateway is present but captureOpenshellForStatus(["inference", "get"]) throws, the collector silently sets liveResult=null, falls back to registry model/provider, and can still compute provider health. JSON consumers may treat those fields as live gateway-verified status even though live inference configuration could not be read.
  • Recommendation: Define the JSON contract for inference capture failure. Prefer an explicit live-inference-unavailable marker or set inferenceHealth=null when live inference capture fails, then add a regression test proving the body and exit code for this path.
  • Evidence: The collector has catch { liveResult = null; }, then computes currentModel/currentProvider from live || registry || 'unknown' and calls getSandboxStatusInferenceHealth(lookup.state === 'present', currentProvider, currentModel). No added test forces captureOpenshellForStatus(["inference", "get"]) to throw.
• Per-sandbox diagnostic redaction is implemented but not pinned for nested health fields (src/commands/sandbox/status.ts:27): The JSON path now routes the report through redactForLog(), which addresses the main credential-leakage risk. However, this new machine-readable output can include inferenceHealth.endpoint, detail, and nested subprobes that automation may persist, and the added tests do not inject token-shaped values into those per-sandbox health fields to prove the redaction contract stays intact.
  • Recommendation: Add a focused per-sandbox status --json test that injects token-like values into inferenceHealth.endpoint, inferenceHealth.detail, and nested subprobes, then asserts the output is parseable JSON and contains only redacted values.
  • Evidence: SandboxStatusCommand.run() returns redactForLog(report), and SandboxStatusReport includes inferenceHealth: ProviderHealthStatus | null. Existing credential-redaction coverage is for the global status --json path, not the new per-sandbox report with health subprobes.