E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• hermes-inference-switch-e2e (medium): Covers the existing Hermes inference set path to ensure default OpenAI-compatible switching, config.yaml patching, hashes, registry/session updates, and live requests still work.
• openclaw-inference-switch-e2e (medium): Covers the existing OpenClaw inference set path to ensure openclaw.json patching, hashes, registry/session updates, and live requests still work.
• hermes-anthropic-inference-switch-e2e (medium): Directly validates the new Hermes compatible Anthropic Messages switch path, including model.api_mode, inference.local /v1/messages traffic, and the new mock Anthropic provider helper.
• openclaw-anthropic-inference-switch-e2e (medium): Directly validates the new OpenClaw compatible Anthropic Messages switch path, including provider namespace selection, inference.local /v1/messages traffic, and the new mock Anthropic provider helper.

Optional E2E

• inference-routing-e2e (low): Useful adjacent coverage for inference.local credential isolation and route error classification, though the PR primarily changes inference set/switch behavior.
• bedrock-runtime-compatible-anthropic-e2e (medium): Good confidence check for the compatible Anthropic endpoint family and Bedrock Runtime adapter behavior, especially because the PR preserves OpenAI-compatible routing for same-provider Bedrock-style switches.
• hermes-e2e (medium): Basic Hermes install/onboard/live inference smoke can catch broad regressions in Hermes config handling, but the dedicated switch jobs are more targeted.

New E2E recommendations

• None.

Dispatch hint

• Workflow: .github/workflows/nightly-e2e.yaml
• jobs input: hermes-inference-switch-e2e,openclaw-inference-switch-e2e,hermes-anthropic-inference-switch-e2e,openclaw-anthropic-inference-switch-e2e

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: medium

Required scenario E2E

• ubuntu-repo-cloud-openclaw-double-provider-switch: The PR changes inference route API resolution and nemoclaw inference set behavior. This routed scenario is the dispatchable scenario coverage for OpenClaw provider-switch lifecycle behavior.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw-double-provider-switch
• ubuntu-repo-cloud-openclaw-double-same-provider: The PR changes same-provider route API preservation logic used by nemoclaw inference set; this scenario is the smallest routed scenario that exercises the same-provider switch lifecycle.
  • Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw-double-same-provider

Optional scenario E2E

• None.

Relevant changed files

• src/lib/actions/inference-route-api.ts
• src/lib/actions/inference-set.ts

🛠️ Needs attention

• Large inference-switch changes still grow existing hotspots (src/lib/actions/inference-set.test.ts:1): The PR extracted route-family resolution into `inference-route-api.ts`, which resolves much of the prior monolith concern, but the changed hotspot files still grow materially. `inference-set.test.ts` grows by 294 lines and `inference-set.ts` still grows by 23 lines in a central path that coordinates OpenShell route state, registry updates, onboard session state, and in-sandbox config/hash synchronization.
  • Recommendation: Move repeated compatible-Anthropic fixtures/helpers out of `inference-set.test.ts`, keep route-mode behavior in the extracted module, and offset the remaining `inference-set.ts` growth where possible before merge.
  • Evidence: Deterministic drift scan reports `src/lib/actions/inference-set.test.ts` 606 -> 900 lines (+294) and `src/lib/actions/inference-set.ts` 448 -> 471 lines (+23).

🔎 Worth checking

• Source-of-truth review needed: src/lib/actions/inference-route-api.ts compatible-Anthropic route-family resolution: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: `resolveRuntimeInferenceApi` checks session/config sources and defaults `compatible-anthropic-endpoint` to `anthropic-messages`; `readOpenClawRouteApi` documents primary-ref-before-provider-block behavior.
• Source-of-truth review needed: src/lib/actions/inference-route-api.ts tolerant route readers: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: `readProviderApi`, `readOpenClawPrimaryProviderKey`, `readHermesRouteApi`, `sessionRouteApi`, and `hermesApiMode` silently return `null` for missing or unsupported state.
• New Anthropic switch jobs expand the target-ref secret boundary (.github/workflows/nightly-e2e.yaml:1019): The new Anthropic switch jobs no longer pass `GITHUB_TOKEN`, which resolves the prior token-specific finding, but they do pass `NVIDIA_API_KEY` into a reusable workflow that checks out and runs `target_ref` code. The jobs use a mock compatible-Anthropic provider, so the remaining need for a real NVIDIA key is the install/onboard setup, not the new mocked switch route itself.
  • Recommendation: Avoid `nvidia_api_key: true` for these mock-route jobs if the setup can be made hermetic, or document why the initial install/onboard still requires a real NVIDIA key and keep the jobs out of PR-triggered untrusted contexts.
  • Evidence: `hermes-anthropic-inference-switch-e2e` sets `nvidia_api_key: true` at line 1019, and `openclaw-anthropic-inference-switch-e2e` sets `nvidia_api_key: true` at line 1343. `.github/workflows/e2e-script.yaml` checks out `inputs.ref` and runs the target-ref script.
• Mock Anthropic provider binds all interfaces (test/e2e/lib/anthropic-switch-provider.sh:131): The E2E mock server binds `0.0.0.0` so the sandbox can reach it through `host.openshell.internal`. That is likely intentional for this topology, and the mock does not expose real credentials, but it unnecessarily exposes an unauthenticated test HTTP server beyond loopback on self-hosted/shared runners.
  • Recommendation: Bind to the minimum host interface needed for sandbox reachability, or document why `0.0.0.0` is required and ensure the runner firewall/network isolation is the intended control.
  • Evidence: `ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever()` starts the mock provider, while the workflow supplies `NEMOCLAW_SWITCH_MOCK_ANTHROPIC=1` for the new jobs.
• Route-family fallback still needs an explicit source-of-truth removal condition (src/lib/actions/inference-route-api.ts:61): The extracted resolver now handles the stale-provider-block bug by preferring OpenClaw `agents.defaults.model.primary`, and it writes `preferredInferenceApi` back to matching sessions. However, compatible-Anthropic route family is still inferred from session, then config, then defaulted to Anthropic Messages. The code explains the OpenClaw config boundary, but it does not state when provider-block/default fallbacks can be removed or why route family cannot be made authoritative in registry/provider metadata in this PR.
  • Recommendation: Document the invalid legacy states and removal condition near `resolveRuntimeInferenceApi`, and consider persisting route family in registry/provider metadata so future switches do not need provider-block/default inference for active route family.
  • Evidence: `readOpenClawRouteApi` falls back from primary ref to provider blocks; `resolveRuntimeInferenceApi` defaults `compatible-anthropic-endpoint` to `anthropic-messages` when session/config do not resolve a route.
• Related [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230 fresh-onboard reproduction is only partially covered (test/e2e/test-hermes-inference-switch.sh:328): Issue [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230 describes fresh Hermes onboarding with an Anthropic-compatible provider followed by in-sandbox Hermes chat failing with HTTP 403. This PR adds runtime `inference set` route synchronization and switch E2Es after a sandbox already exists, including `/v1/messages` checks through `inference.local`. It does not add or identify fresh onboarding coverage for the literal reproduction steps.
  • Recommendation: If this PR is intended to cover [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230, add or identify a Hermes onboarding test that selects a compatible-Anthropic provider, writes `api_mode: anthropic_messages`, and verifies Hermes agent traffic through `https://inference.local/v1/messages\`. Otherwise, document that [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230's fresh-onboard policy/config fix was completed by prior work and this PR covers only runtime switching.
  • Evidence: [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230 Steps to Reproduce start with `nemoclaw onboard --agent hermes`; this diff changes `nemohermes inference set` / `nemoclaw inference set` behavior and adds switch E2E calls to `https://inference.local/v1/messages\`.
• Credential fallback fix lacks a negative test (test/e2e/lib/anthropic-switch-provider.sh:174): The prior NVIDIA-key reuse risk appears fixed: non-mock compatible-Anthropic switches now require `COMPATIBLE_ANTHROPIC_API_KEY` and no longer copy `NVIDIA_API_KEY` into the compatible provider credential. A targeted negative test would lock this security boundary down.
  • Recommendation: Add a behavior test that non-mock compatible-Anthropic provider registration fails for an arbitrary `NEMOCLAW_SWITCH_ENDPOINT_URL` when `COMPATIBLE_ANTHROPIC_API_KEY` is absent, and assert that `NVIDIA_API_KEY` is not used as a fallback.
  • Evidence: `ensure_compatible_anthropic_switch_provider` fails when `COMPATIBLE_ANTHROPIC_API_KEY` is empty, but the diff does not add a negative test for that non-mock path.

🌱 Nice ideas

• None.

• **Runtime validation** — Fresh Hermes onboarding with compatible-Anthropic provider writes `model.api_mode: anthropic_messages` and Hermes agent traffic reaches `https://inference.local/v1/messages\`.. This PR changes sandbox runtime route synchronization, OpenShell provider registration, workflow-dispatched E2E jobs, route-family source precedence, config hashes, and live `inference.local` Anthropic Messages paths. Unit tests are strong for the pure resolver, but stale runtime state and workflow/secret boundaries need behavior-specific validation.
• **Runtime validation** — Same-provider compatible-Anthropic Hermes switch with no matching session and no `model.api_mode` preserves the old OpenAI-compatible Bedrock adapter route when that is the configured route.. This PR changes sandbox runtime route synchronization, OpenShell provider registration, workflow-dispatched E2E jobs, route-family source precedence, config hashes, and live `inference.local` Anthropic Messages paths. Unit tests are strong for the pure resolver, but stale runtime state and workflow/secret boundaries need behavior-specific validation.
• **Runtime validation** — Non-mock compatible-Anthropic provider registration fails without `COMPATIBLE_ANTHROPIC_API_KEY` for arbitrary `NEMOCLAW_SWITCH_ENDPOINT_URL` and does not copy `NVIDIA_API_KEY`.. This PR changes sandbox runtime route synchronization, OpenShell provider registration, workflow-dispatched E2E jobs, route-family source precedence, config hashes, and live `inference.local` Anthropic Messages paths. Unit tests are strong for the pure resolver, but stale runtime state and workflow/secret boundaries need behavior-specific validation.
• **Runtime validation** — New Anthropic switch workflow jobs run without `GITHUB_TOKEN` and either avoid `NVIDIA_API_KEY` for mock-only routing or document the install/onboard requirement.. This PR changes sandbox runtime route synchronization, OpenShell provider registration, workflow-dispatched E2E jobs, route-family source precedence, config hashes, and live `inference.local` Anthropic Messages paths. Unit tests are strong for the pure resolver, but stale runtime state and workflow/secret boundaries need behavior-specific validation.
• **Runtime validation** — Compatible-Anthropic OpenClaw runtime switch with both stale `anthropic` and active `inference` provider blocks and no matching session keeps routing from `agents.defaults.model.primary=inference/...`.. This PR changes sandbox runtime route synchronization, OpenShell provider registration, workflow-dispatched E2E jobs, route-family source precedence, config hashes, and live `inference.local` Anthropic Messages paths. Unit tests are strong for the pure resolver, but stale runtime state and workflow/secret boundaries need behavior-specific validation.
• **Credential fallback fix lacks a negative test** — Add a behavior test that non-mock compatible-Anthropic provider registration fails for an arbitrary `NEMOCLAW_SWITCH_ENDPOINT_URL` when `COMPATIBLE_ANTHROPIC_API_KEY` is absent, and assert that `NVIDIA_API_KEY` is not used as a fallback.
• **Acceptance clause:** [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230 title: "[macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 \"connection not allowed by policy\" — managed_inference policy missing /v1/messages path" — add test evidence or identify existing coverage. Current checkout has `POST /v1/messages` in `agents/hermes/policy-additions.yaml`; this PR adds runtime-switch validation through `https://inference.local/v1/messages\`, but the policy change itself is not in this diff.
• **Acceptance clause:** [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230 Description: "When onboarding NemoHermes with an Anthropic-compatible provider (e.g. `aws/anthropic/bedrock-claude-opus-4-6` via NVIDIA Inference API), the inference smoke test during onboard passes (runs outside sandbox), but chatting inside the sandbox fails with HTTP 403 \"connection not allowed by policy\"." — add test evidence or identify existing coverage. The PR validates runtime switching after install/onboard. It does not add the fresh Hermes onboarding selection flow from the issue description.

Current findings:

• Source-of-truth review needed: src/lib/actions/inference-route-api.ts compatible-Anthropic route-family resolution: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: `resolveRuntimeInferenceApi` checks session/config sources and defaults `compatible-anthropic-endpoint` to `anthropic-messages`; `readOpenClawRouteApi` documents primary-ref-before-provider-block behavior.
• Source-of-truth review needed: src/lib/actions/inference-route-api.ts tolerant route readers: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: `readProviderApi`, `readOpenClawPrimaryProviderKey`, `readHermesRouteApi`, `sessionRouteApi`, and `hermesApiMode` silently return `null` for missing or unsupported state.
• Large inference-switch changes still grow existing hotspots (src/lib/actions/inference-set.test.ts:1): The PR extracted route-family resolution into `inference-route-api.ts`, which resolves much of the prior monolith concern, but the changed hotspot files still grow materially. `inference-set.test.ts` grows by 294 lines and `inference-set.ts` still grows by 23 lines in a central path that coordinates OpenShell route state, registry updates, onboard session state, and in-sandbox config/hash synchronization.
  • Recommendation: Move repeated compatible-Anthropic fixtures/helpers out of `inference-set.test.ts`, keep route-mode behavior in the extracted module, and offset the remaining `inference-set.ts` growth where possible before merge.
  • Evidence: Deterministic drift scan reports `src/lib/actions/inference-set.test.ts` 606 -> 900 lines (+294) and `src/lib/actions/inference-set.ts` 448 -> 471 lines (+23).
• New Anthropic switch jobs expand the target-ref secret boundary (.github/workflows/nightly-e2e.yaml:1019): The new Anthropic switch jobs no longer pass `GITHUB_TOKEN`, which resolves the prior token-specific finding, but they do pass `NVIDIA_API_KEY` into a reusable workflow that checks out and runs `target_ref` code. The jobs use a mock compatible-Anthropic provider, so the remaining need for a real NVIDIA key is the install/onboard setup, not the new mocked switch route itself.
  • Recommendation: Avoid `nvidia_api_key: true` for these mock-route jobs if the setup can be made hermetic, or document why the initial install/onboard still requires a real NVIDIA key and keep the jobs out of PR-triggered untrusted contexts.
  • Evidence: `hermes-anthropic-inference-switch-e2e` sets `nvidia_api_key: true` at line 1019, and `openclaw-anthropic-inference-switch-e2e` sets `nvidia_api_key: true` at line 1343. `.github/workflows/e2e-script.yaml` checks out `inputs.ref` and runs the target-ref script.
• Mock Anthropic provider binds all interfaces (test/e2e/lib/anthropic-switch-provider.sh:131): The E2E mock server binds `0.0.0.0` so the sandbox can reach it through `host.openshell.internal`. That is likely intentional for this topology, and the mock does not expose real credentials, but it unnecessarily exposes an unauthenticated test HTTP server beyond loopback on self-hosted/shared runners.
  • Recommendation: Bind to the minimum host interface needed for sandbox reachability, or document why `0.0.0.0` is required and ensure the runner firewall/network isolation is the intended control.
  • Evidence: `ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever()` starts the mock provider, while the workflow supplies `NEMOCLAW_SWITCH_MOCK_ANTHROPIC=1` for the new jobs.
• Route-family fallback still needs an explicit source-of-truth removal condition (src/lib/actions/inference-route-api.ts:61): The extracted resolver now handles the stale-provider-block bug by preferring OpenClaw `agents.defaults.model.primary`, and it writes `preferredInferenceApi` back to matching sessions. However, compatible-Anthropic route family is still inferred from session, then config, then defaulted to Anthropic Messages. The code explains the OpenClaw config boundary, but it does not state when provider-block/default fallbacks can be removed or why route family cannot be made authoritative in registry/provider metadata in this PR.
  • Recommendation: Document the invalid legacy states and removal condition near `resolveRuntimeInferenceApi`, and consider persisting route family in registry/provider metadata so future switches do not need provider-block/default inference for active route family.
  • Evidence: `readOpenClawRouteApi` falls back from primary ref to provider blocks; `resolveRuntimeInferenceApi` defaults `compatible-anthropic-endpoint` to `anthropic-messages` when session/config do not resolve a route.
• Related [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230 fresh-onboard reproduction is only partially covered (test/e2e/test-hermes-inference-switch.sh:328): Issue [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230 describes fresh Hermes onboarding with an Anthropic-compatible provider followed by in-sandbox Hermes chat failing with HTTP 403. This PR adds runtime `inference set` route synchronization and switch E2Es after a sandbox already exists, including `/v1/messages` checks through `inference.local`. It does not add or identify fresh onboarding coverage for the literal reproduction steps.
  • Recommendation: If this PR is intended to cover [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230, add or identify a Hermes onboarding test that selects a compatible-Anthropic provider, writes `api_mode: anthropic_messages`, and verifies Hermes agent traffic through `https://inference.local/v1/messages\`. Otherwise, document that [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230's fresh-onboard policy/config fix was completed by prior work and this PR covers only runtime switching.
  • Evidence: [macOS][Policy&Network] Hermes Agent with Anthropic-compatible provider gets HTTP 403 "connection not allowed by policy" — managed_inference policy missing /v1/messages path #4230 Steps to Reproduce start with `nemoclaw onboard --agent hermes`; this diff changes `nemohermes inference set` / `nemoclaw inference set` behavior and adds switch E2E calls to `https://inference.local/v1/messages\`.
• Credential fallback fix lacks a negative test (test/e2e/lib/anthropic-switch-provider.sh:174): The prior NVIDIA-key reuse risk appears fixed: non-mock compatible-Anthropic switches now require `COMPATIBLE_ANTHROPIC_API_KEY` and no longer copy `NVIDIA_API_KEY` into the compatible provider credential. A targeted negative test would lock this security boundary down.
  • Recommendation: Add a behavior test that non-mock compatible-Anthropic provider registration fails for an arbitrary `NEMOCLAW_SWITCH_ENDPOINT_URL` when `COMPATIBLE_ANTHROPIC_API_KEY` is absent, and assert that `NVIDIA_API_KEY` is not used as a fallback.
  • Evidence: `ensure_compatible_anthropic_switch_provider` fails when `COMPATIBLE_ANTHROPIC_API_KEY` is empty, but the diff does not add a negative test for that non-mock path.

Walkthrough

Adds runtime inference API types and resolver, maps resolved API to Hermes/OpenClaw sandbox patches and session metadata, extends unit tests, adds E2E Anthropic mock/provider helpers and script changes, updates transient retry detection, and wires two nightly CI jobs for Anthropic inference-switch coverage.

Changes

Anthropic Inference API Switching Feature

Sequence Diagram(s)

sequenceDiagram
  participant runInferenceSet
  participant resolveRuntimeInferenceApi
  participant patchHermesInferenceConfig
  participant patchOpenClawInferenceConfig
  participant updateMatchingOnboardSession
  runInferenceSet->>resolveRuntimeInferenceApi: compute preferredInferenceApi(agentName, entry, session, provider)
  resolveRuntimeInferenceApi-->>runInferenceSet: InferenceApi | null
  alt agent is Hermes
    runInferenceSet->>patchHermesInferenceConfig: apply preferredInferenceApi
  else agent is OpenClaw
    runInferenceSet->>patchOpenClawInferenceConfig: apply preferredInferenceApi
  end
  runInferenceSet->>updateMatchingOnboardSession: persist patched.route.inferenceApi

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• NVIDIA/NemoClaw#4402: Related Hermes mapping of inference API to anthropic_messages and managed-inference policy for /v1/messages.
• NVIDIA/NemoClaw#4718: Overlaps on Hermes E2E config validation changes for inference switching.

Suggested labels

Platform: Ubuntu, E2E

Suggested reviewers

• cv

Poem

🐰 I hopped through configs, routes, and tests,
Mocked a tiny Anthropic server in nests,
Hermes and OpenClaw learned which API to plea,
Sessions remember where responses should be,
Nightly jobs now dance and report with glee.

❌ Failed checks (1 warning)

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}