Skip to content

[All Platforms][Policy&Network] Multi-sandbox gateway: inference through shared inference.local blocked by scope upgrade pending approval even though gateway is reused and both sandboxes are Ready #4263

@PrachiShevate-nv

Description

@PrachiShevate-nv

Platform

DGX / Linux host

Symptom

In a two-sandbox setup, both sandboxes can start agents concurrently, but gateway-backed inference fails for both with scope upgrade pending approval / pairing required, causing OpenClaw to fall back to the embedded agent instead of using the shared inference.local gateway route. This violates condition 2 of the test: "Both sandboxes can run inference concurrently against the shared gateway."

Component area

Gateway / Scopes & Pairing / Multi-sandbox Inference

Steps to reproduce

Preconditions

  • NemoClaw CLI installed and working.

  • At least one sandbox already onboarded and Ready, e.g. ollama-resume ("alpha" in the test), with:

    • Provider: ollama-local
    • Model: qwen2.5:32b
    • Policies: npm, pypi, huggingface, brew, brave, local-inference
  • Gateway and first sandbox confirmed via:

    nemoclaw ollama-resume status
    nemoclaw status

    showing ollama-resume Ready and a running gateway/dash at :18789.

Repro

  1. Run NemoClaw onboarding for a second sandbox ("beta") on the same host:

    nemoclaw onboard --name prachi-gemini
  2. Accept reuse of the existing gateway when prompted (or allow it to auto-detect and reuse).

  3. Choose provider gemini-api with model gemini-2.5-flash-lite.

  4. Let onboarding finish so that both sandboxes are Ready.

  5. Confirm both sandboxes and gateway state:

    nemoclaw list
    nemoclaw status

    Example status:

    Sandboxes:
      ollama-resume      (qwen2.5:32b)             :18789
        Inference:       ollama-local / qwen2.5:32b
        Connected:       no
      prachi-gemini *    (gemini-2.5-flash-lite)   :18790
        Inference:       gemini-api / gemini-2.5-flash-lite
        Connected:       no
    
  6. In Terminal 1, connect to the first sandbox and run a test agent:

    nemoclaw ollama-resume connect
    # inside sandbox:
    openclaw agent --agent main -m "Test from ollama-resume" --session-id debug
  7. In Terminal 2, connect to the second sandbox and run a test agent:

    nemoclaw prachi-gemini connect
    # inside sandbox:
    openclaw agent --agent main -m "Test from prachi-gemini" --session-id debug
  8. Observe logs in both sandboxes.

Expected vs Actual result

Expected (per test spec)

  • The second nemoclaw onboard MUST reuse the existing gateway (shared network) without recreating it or invalidating the first sandbox's gateway TLS certs.
  • Both sandboxes should be able to run inference concurrently via the shared inference.local managed route, without falling back to embedded mode:
    • Gateway WebSocket connects successfully.
    • Requests to https://inference.local/v1/... succeed for both sandboxes.
    • nemoclaw status lists both sandboxes and shows a single gateway/runtime (no duplicate gateway or separate cluster per sandbox).

Actual

Gateway reuse and status

Onboarding of the second sandbox logs:

[2/8] Starting OpenShell gateway
[reuse] Skipping gateway (running)
Reusing healthy NemoClaw gateway.

nemoclaw list and nemoclaw status show:

Sandboxes:
  ollama-resume      (qwen2.5:32b)             :18789
    Inference:       ollama-local / qwen2.5:32b
    Connected:       no
  prachi-gemini *    (gemini-2.5-flash-lite)   :18790
    Inference:       gemini-api / gemini-2.5-flash-lite
    Connected:       no

Only one gateway/runtime is present; both sandboxes are Ready with correct provider configs.

Inference from first sandbox (ollama-resume)

From inside ollama-resume:

openclaw agent --agent main -m "Test from ollama-resume" --session-id debug

Logs show:

NemoClaw registered
  Endpoint:  Managed Inference Route (inference.local)
  Provider:  Local Ollama
  Model:     qwen2.5:32b
  Slash:     /nemoclaw

gateway connect failed: GatewayClientRequestError:
  scope upgrade pending approval
  (requestId: 1d872890-c821-439e-8fdc-9176857a1aa5)

EMBEDDED FALLBACK: Gateway agent failed; running embedded agent:
  GatewayTransportError: gateway closed (1008):
  pairing required: device is asking for more scopes than currently approved
  (requestId: 1d872890-c821-439e-8fdc-9176857a1aa5)

Gateway target:   ws://127.0.0.1:18789
Source:           env OPENCLAW_GATEWAY_URL
Config:           /sandbox/.openclaw/openclaw.json
...
[agent/embedded] tool-search: cataloged 28 tools behind compact prompt surface
I understand you're running a test from an environment ...

Inference from second sandbox (prachi-gemini)

From inside prachi-gemini:

openclaw agent --agent main -m "Test from prachi-gemini" --session-id debug

Logs show:

NemoClaw registered
  Endpoint:  Managed Inference Route (inference.local)
  Provider:  Google Gemini
  Model:     gemini-2.5-flash-lite
  Slash:     /nemoclaw

gateway connect failed: GatewayClientRequestError:
  scope upgrade pending approval
  (requestId: 92b6a455-02f1-4400-bce9-16a280011959)

EMBEDDED FALLBACK: Gateway agent failed; running embedded agent:
  GatewayTransportError: gateway closed (1008):
  pairing required: device is asking for more scopes than currently approved
  (requestId: 92b6a455-02f1-4400-bce9-16a280011959)

Gateway target:   ws://127.0.0.1:18790
Source:           env OPENCLAW_GATEWAY_URL
Config:           /sandbox/.openclaw/openclaw.json
...
[agent/embedded] tool-search: cataloged 28 tools behind compact prompt surface
I'm sorry, but I cannot find any tools related to "prachi-gemini". ...

Observations

  • Both sandboxes successfully load the NemoClaw plugin and attempt to use the Managed Inference Route (inference.local) via the gateway WebSocket ports (:18789 and :18790).
  • In both cases, the gateway rejects the session with scope upgrade pending approval / pairing required: device is asking for more scopes than currently approved, and OpenClaw falls back to the embedded agent instead of using the shared gateway.
  • This happens even when both sessions are run concurrently in separate terminals and both dashboards are connected.

Failing condition

  • Condition 1 (gateway reuse): Pass – second nemoclaw onboard correctly reuses the existing gateway; no TLS breakage for the first sandbox.
  • Condition 3 (two sandboxes, one gateway in status): Passnemoclaw status lists both sandboxes and a single runtime.
  • Condition 2 (both sandboxes can run inference concurrently via shared gateway): Fail – inference via inference.local is blocked in both sandboxes by gateway scope approval and always falls back to embedded mode.

Environment versions

  • NemoClaw CLI: v0.0.50
  • OpenShell CLI: 0.0.44
  • OpenClaw: v2026.5.18
  • Sandboxes:
    • ollama-resume – provider ollama-local, model qwen2.5:32b, policies include local-inference
    • prachi-gemini – provider gemini-api, model gemini-2.5-flash-lite

Metadata

Metadata

Assignees

Labels

NV QABugs found by the NVIDIA QA Teamneeds: triageAwaiting maintainer classification

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions