E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• macos-e2e (medium): Required because this PR adds a macOS-native installer app and macOS-specific assess/build behavior. The existing macOS E2E workflow validates the CLI build on Apple Silicon macOS and runs the full OpenClaw E2E when Docker is available.
• cloud-onboard-e2e (high): Required because native-installer mac install delegates to the standard onboarding path and maps provider, model, sandbox name, mode, policy, messaging, and ports into onboard flags/env. This validates the core installer/onboarding path remains functional for stock OpenClaw.
• hermes-e2e (high): Required because Mac Installer Preview explicitly supports stock Hermes, includes Hermes-specific provider/auth behavior, and returns Hermes launch API details. Existing Hermes E2E validates install, health probe, and live inference for the Hermes sandbox path.
• diagnostics-e2e (high): Required because this PR adds nemoclaw diagnostics export and native-app diagnostics context/export behavior. Existing diagnostics E2E validates debug/export tarball behavior and credential sanitization in diagnostics artifacts.
• inference-routing-e2e (medium): Required because native installer provider/model/endpoint selections are translated into onboarding inference routes and the app surfaces provider errors. Existing inference routing E2E validates credential isolation and error classification for inference routes.
• credential-sanitization-e2e (high): Required because the new native installer path handles temporary provider secrets, rejects secrets in JSON config, and exports diagnostics. Existing credential sanitization E2E is the closest runtime guard for real key leakage into logs/files/artifacts.

Optional E2E

• network-policy-e2e (high): Useful adjacent coverage because Mac Installer Preview introduces trust tiers and security preset mapping into onboarding policy env vars. This is optional because the PR does not modify the core network policy engine directly.
• sandbox-operations-e2e (high): Useful confidence for launch/registry/sandbox lifecycle behavior touched indirectly by native-installer mac launch, but not merge-blocking because existing native launch logic is mostly registry/URL assembly and unit-tested.
• onboard-negative-paths-e2e (high): Useful for validating onboarding failure/recovery surfaces adjacent to the new native installer config validation and preflight handling, but existing tests do not directly exercise the new native config JSON path.

New E2E recommendations

• native mac installer end-to-end flow (high): No existing E2E appears to exercise nemoclaw native-installer mac describe/assess/install/launch as a user-visible flow. Add a macOS scenario that runs describe/assess, creates a config for OpenClaw and Hermes, invokes install with --json-progress, verifies progress phases, and validates launch JSON. It should gracefully skip Docker-dependent install assertions on GitHub-hosted macOS when Docker is unavailable.
  • Suggested test: Add a Mac Installer Preview scenario/job covering native-installer mac describe, assess, install --json-progress, and launch --json for OpenClaw and Hermes.
• native mac installer artifact build (high): The new DMG/app build workflow is not an E2E test and is manual-only. Add CI coverage that runs npm run build:native-mac-installer -- --dry-run on Linux and a real unsigned app bundle build on macOS to catch payload staging, Swift package, Info.plist, and bundled CLI regressions before release dispatch.
  • Suggested test: Add a native-mac-installer-build-smoke job that validates dry-run payload staging and macOS Swift app bundle creation.
• native installer credential redaction (high): Existing credential E2Es cover standard onboarding/diagnostics, but not the Swift app command log or NEMOCLAW_MAC_INSTALLER_DIAGNOSTICS_DIR artifacts. Add coverage that injects a fake temporary key through the native installer path and asserts it is absent from config JSON, command logs, diagnostics context, and exported tarball.
  • Suggested test: Add native-mac-installer-credential-redaction-e2e for temporary secret injection and diagnostics export sanitization.

Dispatch hint

• Workflow: nightly-e2e.yaml
• jobs input: cloud-onboard-e2e,hermes-e2e,diagnostics-e2e,inference-routing-e2e,credential-sanitization-e2e

PR Review Advisor

Base: origin/main
Head: HEAD
Analyzed SHA: 380f1477afc706328d5cf44b545c61f64455626c
Recommendation: blocked
Confidence: high

Blocked: required CI failed, mergeState is BLOCKED/draft, E2E Advisor-required jobs are missing for this head SHA, and the Swift app appears to write token-bearing CLI output to raw temp diagnostics logs.

Gate status

• CI: fail — Required status context checks failed for head SHA 380f147; cli-parity also failed. Required contexts commit-lint, dco-check, check-hash, and changes passed.
• Mergeability: fail — GraphQL mergeStateStatus=BLOCKED; PR is draft and reviewDecision=REVIEW_REQUIRED.
• Review threads: warning — GraphQL reviewThreads.nodes is empty, but CodeRabbit skipped review because draft was detected; no resolved/unresolved thread state beyond that was available.
• Risky code tested: fail — Risky areas include credentials/inference/network, installer/bootstrap shell, onboarding/host glue, and workflow enforcement. Unit tests were added, but E2E Advisor required cloud-onboard-e2e, hermes-e2e, diagnostics-e2e, inference-routing-e2e, and credential-sanitization-e2e in addition to macos-e2e; only macos-e2e is shown as passed.

🔴 Blockers

• Raw CLI output can persist provider/API tokens in temp command logs (apps/native-installers/macos/NemoClawMacInstaller/Sources/NemoClawMacInstaller/NemoClawMacInstallerApp.swift:656): The native app appends every CLI command stdout/stderr to nemoclaw-mac-installer-app.commands.log without redaction. The new launch path intentionally returns tokenized OpenClaw URLs and Hermes Authorization: Bearer ... details in JSON, and install/onboard failures may include provider credential diagnostics. The app then copies this raw log into diagnostics context via app-command-log.txt; exported tarballs redact later, but the raw temp files remain on disk and can contain secrets.
  • Recommendation: Redact before writing any app command log, omit token-bearing fields from logs entirely, and add tests that inject fake OpenClaw/Hermes/provider tokens and assert they are absent from the raw command log, diagnostics context, and exported tarball.
  • Evidence: runCli reads stdout/stderr then calls appendCommandLog(... output: output, error: error); refreshDiagnosticsContext writes the raw command log into app-command-log.txt; launch --json can include OpenClaw token URLs and Hermes auth headers.
• Required CI failed for the reviewed head SHA: The required checks context failed for head SHA 380f147. cli-parity also failed. This is a hard gate before merge consideration.
  • Recommendation: Fix the failing checks and re-run required CI on the same or updated head SHA before further merge evaluation.
  • Evidence: statusCheckRollup: checks COMPLETED/FAILURE, cli-parity COMPLETED/FAILURE; gate context reports required status context(s) failed: checks.
• Required E2E coverage is missing for installer, credentials, diagnostics, and inference paths: The E2E Advisor required macos-e2e, cloud-onboard-e2e, hermes-e2e, diagnostics-e2e, inference-routing-e2e, and credential-sanitization-e2e. Only macos-e2e is shown as passed for this head SHA. The PR touches a native installer, onboarding handoff, provider credentials, diagnostics export, and launch token handling; unit tests cannot prove these runtime boundaries.
  • Recommendation: Run or add the required E2E coverage for cloud onboarding, Hermes, diagnostics export, inference routing, and credential sanitization for this head SHA. Add the Advisor-suggested native Mac installer flow/build/credential-redaction scenarios before treating this as release-ready.
  • Evidence: E2E Advisor comment lists required E2E jobs; statusCheckRollup shows macos-e2e success but does not show cloud-onboard-e2e, hermes-e2e, diagnostics-e2e, inference-routing-e2e, or credential-sanitization-e2e as passed.
• Current monolith hotspot grew by 20 lines (src/lib/diagnostics/debug.ts:1): Trusted monolith analysis reports src/lib/diagnostics/debug.ts grew from 535 to 555 lines, crossing the configured hotspot growth blocker. This file already has active overlap in recent work and should not grow further without extraction or offsetting reduction.
  • Recommendation: Extract the native-installer diagnostics-context collection into a focused helper module, or offset the growth by reducing this hotspot before merge.
  • Evidence: monolithDeltas: src/lib/diagnostics/debug.ts, baseLines=535, headLines=555, delta=20, severity=blocker.

🟡 Warnings

• Release workflow uses floating third-party action tags (.github/workflows/native-mac-installer-preview.yaml:25): The new release workflow pins actions/checkout by SHA but uses actions/setup-node@v6 and actions/upload-artifact@v4 as floating major tags. This workflow handles release artifacts and Apple signing/notarization secrets, so the trusted-code boundary should avoid mutable action references.
  • Recommendation: Pin all GitHub Actions in this workflow to full commit SHAs and document the source/version update process.
  • Evidence: Workflow diff shows uses: actions/setup-node@v6 and uses: actions/upload-artifact@v4; workflow also passes Apple signing/notarization secrets to the build step.
• Bundled OpenShell and Node runtime inputs are copied without integrity verification (scripts/native-installers/macos/build-preview.sh:82): The preview builder copies optional NEMOCLAW_OPENSHELL_BIN and NEMOCLAW_MAC_INSTALLER_NODE_TARBALL directly into the app payload. It also stages broad payload content including scripts and node_modules. For an installer artifact, this creates a supply-chain trust gap unless the inputs are pinned and verified.
  • Recommendation: Require SHA256/signature verification for bundled OpenShell and Node runtime artifacts, record their versions in the payload manifest/SBOM, and fail release builds when required pinned inputs are absent or unverifiable.
  • Evidence: cp "$NEMOCLAW_OPENSHELL_BIN" "$PAYLOAD_DIR/tools/openshell" and cp "$NEMOCLAW_MAC_INSTALLER_NODE_TARBALL" "$PAYLOAD_DIR/tools/node-runtime.tar.gz" occur without checksum/signature checks.
• Installer config validation delegates several user-controlled values without local allowlists (src/lib/native-installer/macos/config.ts:144): The adapter validates agent, provider, secrets, custom Dockerfile, and port ranges, but accepts sandboxName, endpoint URL, security tier/presets, messaging channel names, and model as trimmed strings/arrays before passing them into onboard environment/flags. Downstream onboard validation may catch many cases, but this app-facing API is a new public contract and should have explicit negative tests for pathologic values and contract drift.
  • Recommendation: Add adapter-level validation or explicit contract tests for sandbox name format/reserved names, endpoint URL scheme/host handling, supported policy tiers/presets, supported messaging channels per agent, and malicious model strings.
  • Evidence: validateNativeInstallerConfig reads sandboxName, endpoint, model, security.tier, security.presets, and messaging and returns them in the config; tests currently cover provider rejection, secret-key rejection, custom Dockerfile rejection, Docker assessment, delegation, and launch happy paths.
• Documentation changes overlap many active PRs (docs/reference/commands.mdx:78): The PR modifies docs/reference/commands.mdx, which has many open PR overlaps, including documentation and CLI behavior updates. This raises rebase/conflict and stale-doc risk.
  • Recommendation: Rebase close to merge and re-check the command reference against active docs/CLI PRs before final review.
  • Evidence: Trusted overlap scan lists open PRs 3349, 3351, 3652, 3762, 3766, 3794, 3796, 3825, 3866, 3925, and 4001 touching docs/reference/commands.mdx.

🔵 Suggestions

• None.

Acceptance coverage

• unknown — No linked issues were detected for PR feat(mac): add native Mac installer preview #4006.: github.linkedIssues is empty, so there are no linked issue acceptance clauses to map literally to diff/test evidence.
• partial — Required E2E: macos-e2e, cloud-onboard-e2e, hermes-e2e, diagnostics-e2e, inference-routing-e2e, credential-sanitization-e2e: E2E Advisor comment is present. macos-e2e passed for the head SHA, but cloud-onboard-e2e, hermes-e2e, diagnostics-e2e, inference-routing-e2e, and credential-sanitization-e2e are not shown as passed.
• missing — native mac installer end-to-end flow: Advisor says no existing E2E appears to exercise nemoclaw native-installer mac describe/assess/install/launch as a user-visible flow. Unit tests cover pieces but no native installer end-to-end scenario is shown.
• missing — native mac installer artifact build: Advisor notes the DMG/app build workflow is manual-only and recommends native-mac-installer-build-smoke coverage. No required build-smoke job is shown as passed.
• missing — native installer credential redaction: Advisor recommends coverage that injects a fake temporary key and asserts absence from config JSON, command logs, diagnostics context, and exported tarball. Existing tests only check source text contains diagnostics env/path and config rejects secret-key fields.

Security review

• fail — 1. Secrets and Credentials: Confirmed credential-leak risk: the Swift app logs raw stdout/stderr from token-bearing CLI calls to temp files before redaction. Config JSON rejects secret-looking keys and temporary secrets are passed through environment rather than config, which is positive, but raw app command logs can still persist tokens.
• warning — 2. Input Validation and Data Sanitization: Config parsing rejects unsupported agents/providers, secret-like keys, custom Dockerfile fields, and out-of-range ports. However endpoint URL, sandboxName, model, messaging, and policy fields are not fully allowlisted at the adapter boundary; downstream onboard may validate, but the new app-facing contract needs explicit negative coverage.
• warning — 3. Authentication and Authorization: No new server endpoints are added, but launch JSON intentionally exposes OpenClaw/Hermes bearer tokens to the app. That is expected for launch UX, but it becomes unsafe when combined with raw command logging. No privilege escalation path was found in the reviewed CLI adapter code.
• warning — 4. Dependencies and Third-Party Libraries: No new npm/Swift third-party dependencies are added beyond using existing js-yaml. The release workflow uses floating action tags for setup-node/upload-artifact, and the installer builder can bundle OpenShell/Node artifacts without checksum/signature verification.
• fail — 5. Error Handling and Logging: The native app stores raw command stdout/stderr and surfaces raw errors. This can leak provider credentials, OpenClaw tokenized URLs, or Hermes Authorization headers into local temp logs and diagnostics context.
• pass — 6. Cryptography and Data Protection: No custom cryptography or weakened algorithms are introduced. macOS entitlements keep JIT and library-validation relaxation disabled. Notarization/signing are supported when credentials are provided.
• warning — 7. Configuration and Security Headers: No HTTP security headers are introduced. Workflow permissions are contents:read, which is good, but workflow action pinning is incomplete. The install plan references :latest base image refs, although applyByDefault is false, so they should not affect default security posture.
• warning — 8. Security Testing: Unit tests cover config secret-key rejection, custom Dockerfile rejection, provider catalog validation, assessment Docker distinctions, env mapping, and launch happy paths. Missing: raw log/token redaction tests, malicious endpoint/sandbox/policy/messaging tests, and required credential-sanitization/diagnostics E2E.
• fail — 9. Holistic Security Posture: The change creates a new installer and credential-handling path. It has good design intent—delegating Docker/onboard behavior to existing paths and rejecting secrets in config—but merge should be blocked until token logging is fixed, required runtime E2E passes, CI passes, and installer supply-chain verification is tightened.

Test / E2E status

• Test depth: e2e_required — This PR changes a native macOS installer app, release packaging workflow, onboarding handoff, provider credential injection, diagnostics export, and launch-token handling. Unit tests cover some adapter contracts, but cannot prove runtime installer behavior, credential redaction in real artifacts, or onboarding/launch integration.
• E2E Advisor: missing
• Required E2E jobs: macos-e2e, cloud-onboard-e2e, hermes-e2e, diagnostics-e2e, inference-routing-e2e, credential-sanitization-e2e
• Missing for analyzed SHA: cloud-onboard-e2e, hermes-e2e, diagnostics-e2e, inference-routing-e2e, credential-sanitization-e2e

✅ What looks good

• The native installer adapter intentionally delegates Docker, image resolution, remediation, and onboarding behavior to existing nemoclaw onboard paths instead of reimplementing high-risk sandbox lifecycle code.
• Config validation rejects secret-looking JSON keys and custom Dockerfile fields, limiting the preview to stock OpenClaw/Hermes paths.
• The workflow uses read-only contents permission and pins actions/checkout to a full SHA.
• macOS entitlements do not enable JIT or disable library validation.
• Tests cover provider catalog validation, Docker missing vs unreachable assessment behavior, onboard argv/env translation, base image opt-in behavior, and OpenClaw/Hermes launch JSON shape.

Review completeness

• The supplied git diff was truncated, though key changed files for installer, workflow, diagnostics, config, install, launch, and tests were reviewed with read-only file tools.
• No commands, scripts, package-manager installs, or tests were executed during this advisory review.
• No linked issue acceptance clauses were available; acceptance mapping is limited to PR/E2E Advisor comments and deterministic metadata.
• Review thread state is limited to provided GraphQL data and issue comments; CodeRabbit skipped because the PR is draft.
• Human maintainer review required: yes