Walkthrough

This PR refines agent sandbox policy messages to prevent model hallucination of resource restrictions. It updates network and filesystem policy context strings to emphasize verification-by-action and distinguishes real failures from policy denials. Behavior guidance is clarified to instruct agents to attempt actions before asserting blocks and to report actual failure modes rather than speculated ones.

Changes

Runtime Context Policy Clarity

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

• NVIDIA/NemoClaw#4037: Refactored injection of <nemoclaw-runtime> block via prependSystemContext, which directly relates to this PR's updates to the injected policy and behavior guidance.

Suggested labels

bug-fix, area: policy, enhancement: policy

Suggested reviewers

• cv

Poem

🐰 Agents were dreaming of walls that weren't there,
claiming blocked /tmp with decidedly no care,
Now we ask them to try before saying "nope,"
distinguish real failures—don't just hallucinate hope!
Attempt first, report true, sandbox scripted with care. 🏗️

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: medium

Required E2E

• common-egress-agent-e2e (high; provisions real sandboxes and runs model-backed agent turns): Closest existing E2E for the changed behavior: runs real OpenClaw agent turns that must use web_fetch against policy-allowed endpoints. This validates that NemoClaw/OpenClaw runtime guidance does not cause the assistant to preemptively refuse allowed network access and that real user network-tool flows still work.

Optional E2E

• network-policy-e2e (high; full sandbox/network policy exercise): Useful adjacent confidence because the changed text describes deny-by-default egress, allowed endpoints, proxy/policy denials, and operator approval. This job validates the underlying network-policy enforcement and allowed/blocked endpoint behavior, although it does not directly assert the injected prompt context.
• agent-turn-latency-e2e (high; provisions OpenClaw and Hermes sandboxes and runs real model-backed turns): Optional broad real-turn smoke for OpenClaw plugin/runtime integration after changing the prompt context hook. It confirms a model-backed OpenClaw turn still works, but it is less targeted than common-egress-agent-e2e because it does not exercise network-tool decisions.

New E2E recommendations

• OpenClaw runtime context injection (high): Existing E2E coverage does not appear to assert that the real OpenClaw prompt contains the block or that the before_prompt_build hook is active in an onboarded sandbox. Add a lightweight real-sandbox E2E that triggers an OpenClaw turn with a debug/mock provider or prompt-capture hook and verifies the NemoClaw runtime context is prepended.
  • Suggested test: real-openclaw-runtime-context-injection-e2e
• Restricted endpoint approval behavior (medium): The PR changes the agent directive from preemptive refusal to attempting restricted endpoints and distinguishing proxy/policy denials from DNS/timeout/TLS failures. Add an E2E with a controlled restricted URL and mock/observable approval path that verifies the assistant attempts the request and reports the actual policy-denial/approval semantics instead of refusing before trying.
  • Suggested test: restricted-egress-attempt-and-approval-request-e2e

Dispatch hint

• Workflow: nightly-e2e.yaml
• jobs input: common-egress-agent-e2e

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

• ubuntu-repo-cloud-openclaw: The source change modifies NemoClaw's OpenClaw runtime context injected into prompts. The standard Ubuntu repo-current cloud OpenClaw scenario is the smallest ROUTES-backed scenario that installs the current branch with OpenClaw and exercises baseline OpenClaw startup/sandbox/inference behavior.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Optional scenario E2E

• wsl-repo-cloud-openclaw: Optional adjacent coverage for the same OpenClaw repo-current surface on WSL; special-runner scenario, so not primary unless WSL-specific behavior is suspected.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=wsl-repo-cloud-openclaw
• macos-repo-cloud-openclaw: Optional adjacent coverage for repo-current OpenClaw installation on macOS; special-runner/platform-only scenario and not the primary path for this runtime-context change.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=macos-repo-cloud-openclaw

Relevant changed files

• nemoclaw/src/runtime-context.ts

🛠️ Needs attention

• Linked issue expects missing-tool honesty, but the prompt only says to try the operation (nemoclaw/src/runtime-context.ts:21): Issue [Ubuntu 24.04][Agent&Skills] Nano Omni 30B hallucinates policy restrictions when no tools configured — claims /tmp blocked, network blocked #4850 identifies the actual failure as absent file-write/bash/fetch tools and expects the model to say it lacks the tool instead of fabricating a policy denial. The new filesystem text says the agent can create/edit/run files "using your file and shell tools" and should try the operation, but it never tells the agent what to do when those tools are not available. That leaves the acceptance-critical no-tool case only partially addressed.
  • Recommendation: Add explicit runtime-context guidance such as: if the needed file, shell, web/fetch, or execution tool is not available in the current session, say the tool is unavailable and offer code/instructions instead of describing a sandbox policy denial. Add a unit assertion for that wording.
  • Evidence: Issue [Ubuntu 24.04][Agent&Skills] Nano Omni 30B hallucinates policy restrictions when no tools configured — claims /tmp blocked, network blocked #4850 expected: "I don't have a file-writing tool available in this session..." and states the real issue was no file-write/bash-execute tools. The diff adds line 21: "within the sandbox you can create, edit, and run files ... using your file and shell tools; try the operation and report if it fails rather than assuming it is unavailable".

🔎 Worth checking

• Source-of-truth review needed: Runtime prompt workaround for hallucinated sandbox policy denials: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: Issue [Ubuntu 24.04][Agent&Skills] Nano Omni 30B hallucinates policy restrictions when no tools configured — claims /tmp blocked, network blocked #4850 states the actual issue was no file-write or bash-execute tools. The new prompt text says to try operations using file and shell tools, but does not address the missing-tool state.
• Network-attempt guidance should be bounded to task-relevant, policy-mediated requests (nemoclaw/src/runtime-context.ts:16): The new system context intentionally nudges the agent to make restricted endpoint requests so OpenShell can raise an approval prompt. The policy boundary is not bypassed by this code, but the wording can encourage probing arbitrary user-supplied URLs before considering whether they are task-relevant, sensitive, internal, or metadata-style targets.
  • Recommendation: Qualify the directive so attempts are limited to task-relevant requests through normal sandbox tools and do not encourage probing internal, metadata, credential-bearing, or otherwise sensitive destinations unless explicitly required and mediated by policy.
  • Evidence: Line 16 says: "attempting a restricted endpoint is productive... so make the request rather than refusing preemptively". Line 67 similarly says not to assert a URL or host is blocked unless it has been attempted this turn.
• Prompt tests do not guard the safety and no-tool behaviors this change relies on (nemoclaw/src/runtime-context.test.ts:135): The updated tests check that the grounding directive and approval-request wording appear, but they do not assert the acceptance-critical missing-tool guidance, nor do they guard that the prompt still denies unrestricted host/internet access while encouraging real attempts.
  • Recommendation: Add behavior-focused assertions for missing tool reporting, sandbox-writable `/tmp` and `/sandbox` with no host-path access, no unrestricted internet/host claims, and policy-denial reporting without requiring a literal 403.
  • Evidence: The test currently asserts general section headers and substrings including "unless you have actually attempted it this turn" and "raises an operator approval request", but there is no assertion about unavailable file/shell/fetch tools or preserving the restrictive host/internet posture.

🌱 Nice ideas

• None.

• **Runtime validation** — injects guidance to report missing file, shell, execution, or fetch tools instead of fabricating sandbox policy denials. The changed production logic is deterministic prompt construction and can be guarded with unit tests, but the linked bug is model behavior under a no-tool runtime configuration, so prompt-string tests alone cannot prove the hallucination is resolved.
• **Runtime validation** — keeps `/tmp` and `/sandbox` described as sandbox-writable while still denying host-path access. The changed production logic is deterministic prompt construction and can be guarded with unit tests, but the linked bug is model behavior under a no-tool runtime configuration, so prompt-string tests alone cannot prove the hallucination is resolved.
• **Runtime validation** — keeps unrestricted internet and host access prohibited while encouraging task-relevant policy-mediated attempts. The changed production logic is deterministic prompt construction and can be guarded with unit tests, but the linked bug is model behavior under a no-tool runtime configuration, so prompt-string tests alone cannot prove the hallucination is resolved.
• **Runtime validation** — reports policy denials without requiring a literal proxy 403 status. The changed production logic is deterministic prompt construction and can be guarded with unit tests, but the linked bug is model behavior under a no-tool runtime configuration, so prompt-string tests alone cannot prove the hallucination is resolved.
• **Runtime validation** — validates the no-file-write/no-shell/no-fetch tool configuration does not produce `/tmp blocked by operator` or total network-blocked wording. The changed production logic is deterministic prompt construction and can be guarded with unit tests, but the linked bug is model behavior under a no-tool runtime configuration, so prompt-string tests alone cannot prove the hallucination is resolved.
• **Prompt tests do not guard the safety and no-tool behaviors this change relies on** — Add behavior-focused assertions for missing tool reporting, sandbox-writable `/tmp` and `/sandbox` with no host-path access, no unrestricted internet/host claims, and policy-denial reporting without requiring a literal 403.
• **Acceptance clause:** When `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` is asked to write a file or fetch a URL, it invents false policy restrictions rather than reporting that no tool is available. — add test evidence or identify existing coverage. The diff discourages speculative blocked/unreachable claims, but it does not add guidance for reporting missing file-write, shell, or fetch tools.
• **Acceptance clause:** It claims `/tmp` is "blocked by operator" and "network access is blocked" — both incorrect. — add test evidence or identify existing coverage. The new runtime context says `/tmp` and `/sandbox` are usable inside the sandbox and says not to assert a URL/host is blocked without attempting it. Tests do not specifically guard against the `/tmp blocked by operator` wording recurring.