Skip to content

fix(e2e): stabilize nightly recovery coverage#5401

Merged
cv merged 3 commits into
mainfrom
codex/fix-nightly-e2e-followups
Jun 13, 2026
Merged

fix(e2e): stabilize nightly recovery coverage#5401
cv merged 3 commits into
mainfrom
codex/fix-nightly-e2e-followups

Conversation

@cv

@cv cv commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stabilizes the nightly E2E follow-up failures by keeping the Kimi scenario on the public NVIDIA Kimi endpoint, recognizing the generated routed Kimi model ref, and repairing narrow OpenClaw scope-approval failures that report nonzero after local state changes. It also aligns the crash-loop recovery assertion with the current gateway guard markers and keeps the test-size budget ratcheted after moving coverage into a focused policy test.

Related Issue

Related to #2478, #4462, #2620, #3046.

Changes

  • Keep the Kimi E2E on public NVIDIA Endpoints via nvidia-prod with moonshotai/kimi-k2.6; retain the local mock only behind NEMOCLAW_KIMI_USE_MOCK=1 and sanitize Kimi failure logs.
  • Recognize the generated inference/moonshotai/kimi-k2.6 model ref in the Kimi compatibility plugin.
  • Add constrained OpenClaw approval recovery for failed allowlisted scope upgrades that leave original or replacement pending state, without granting operator.admin.
  • Wire the recovery helper into sandbox auto-pair approval and the startup guard wrapper.
  • Align the crash-loop E2E guard assertion with current gateway safety marker behavior.
  • Add/update targeted tests and ratchet the test/nemoclaw-start.test.ts size budget downward.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
  • Targeted tests pass for changed behavior
  • Full npm test passes (broad runtime changes only)
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Docs review found no user-facing docs changes were warranted; docs/ and generated .agents/skills/ remained clean.


Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • New Features

    • Added recovery support for failed OpenClaw device approval flows, including gateway-connect compatibility scenarios.
    • Improved Kimi inference compatibility handling for managed Kimi model references and stream/tool rewrite alignment.
  • Bug Fixes

    • Auto-pair approval now conditionally retries via the recovery policy on specific approval failures, updating pending/paired scope state.
  • Security & CI

    • Nightly Kimi inference E2E now supports live-vs-mock execution with public NVIDIA key validation and redacts NVIDIA keys in sanitized logs.
  • Tests / Chores

    • Expanded E2E and policy recovery tests; updated gateway-guard assertions and adjusted a test file size budget.

@cv cv self-assigned this Jun 13, 2026
@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bdfccaa4-488b-4cbe-b69f-ad19ae9dc435

📥 Commits

Reviewing files that changed from the base of the PR and between 0444194 and 8ca0ccd.

📒 Files selected for processing (3)
  • .github/workflows/nightly-e2e.yaml
  • test/e2e-script-workflow.test.ts
  • test/e2e/test-kimi-inference-compat.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/e2e/test-kimi-inference-compat.sh

📝 Walkthrough

Walkthrough

This PR adds failed device-approval recovery in policy and watcher flows, expands Kimi inference compatibility handling for routed/live runs and nightly secrets, and simplifies one crash-loop recovery E2E guard assertion.

Changes

OpenClaw approval recovery

Layer / File(s) Summary
Policy recovery helper
scripts/lib/openclaw_device_approval_policy.py
Adds JSON/state helpers and recover_failed_scope_approval(...) to repair pending approvals by updating paired scopes with implied relationships, clearing pending entries, and returning a compatibility marker.
Watcher and shell integration
src/lib/actions/sandbox/auto-pair-approval.ts, scripts/nemoclaw-start.sh
Loads the optional recovery hook into the auto-pair flow, invokes it after failed approvals with request id and error context, and updates shell-side recovery selection and original pending key tracking with emitted compatibility metadata.
Recovery validation and test maintenance
src/lib/actions/sandbox/auto-pair-approval.test.ts, test/openclaw-device-approval-policy.test.ts, test/nemoclaw-start.test.ts, ci/test-file-size-budget.json
Adds recovery assertions and integration-style recovered-approval test covering scope merging and allowlist enforcement, adds Python subprocess test harness, fixes one misplaced test boundary, and updates size budget by one line.

Kimi inference compatibility coverage

Layer / File(s) Summary
Managed model detection
nemoclaw-blueprint/openclaw-plugins/kimi-inference-compat/index.js, test/kimi-inference-compat-plugin.test.ts
The plugin now recognizes both raw and inference/... routed Kimi model identifiers across multiple context fields, and the new unit test covers wrapped stream rewriting for routed model references.
Live versus mock E2E flow
test/e2e/test-kimi-inference-compat.sh
Adds mode helpers (use_kimi_mock(), ensure_public_nvidia_api_key(), check_upstream_observed_agent_traffic()), validates nvapi-* key patterns for live runs, branches onboarding and PASS messages by mode, and checks upstream observed routing only in live mode.
Nightly secret injection and sanitization
.github/workflows/nightly-e2e.yaml, test/e2e-script-workflow.test.ts
The nightly Kimi job exports NVIDIA_API_KEY with untrusted-ref guard, adds it to a guarded secret template, and redacts both API key and token values from failure logs; test validates workflow env setup and bash fixture messaging.

Crash-loop recovery E2E assertion

Layer / File(s) Summary
Proxy-env based guard verification
test/e2e/test-issue-2478-crash-loop-recovery.sh
Rewrites gateway_guards_active() to verify guard exports from /tmp/nemoclaw-proxy-env.sh plus a live gateway PID instead of checking gateway log markers and boundaries; removes the log_boundary parameter from the function signature.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related issues

Possibly related PRs

  • NVIDIA/NemoClaw#5210 — Both PRs change nemoclaw-start.sh approval-failure recovery that reconciles pending.json and paired.json with scope merging and admin-scope exclusion.
  • NVIDIA/NemoClaw#5348 — Both PRs touch the crash-loop recovery guard-chain test contract and gateway_guards_active validation signature.
  • NVIDIA/NemoClaw#5342 — Both PRs update the same crash-loop recovery test's guard verification logic and proxy-env validation flow.

Suggested labels

nightly-e2e, integration: openclaw, bug-fix, area: sandbox

Suggested reviewers

  • sandl99

Poem

🐇 I patched a burrow path gone wrong,
where pending paws lingered far too long.
Kimi now knows the routed trail,
and nightly masks each secret tail.
The gateway guards stand watch once more,
while rabbit feet thump past the door.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 12.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix(e2e): stabilize nightly recovery coverage' directly and specifically addresses the main objective of stabilizing failing nightly E2E tests and recovery scenarios.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-nightly-e2e-followups

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-code-quality

github-code-quality Bot commented Jun 13, 2026

Copy link
Copy Markdown

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the branch is 96%. Coverage data for the branch is not yet available.

Show a code coverage summary of the most covered files.
File 8ca0ccd +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the branch is 44%. Coverage data for the branch is not yet available.

Show a code coverage summary of the most covered files.
File 8ca0ccd +/-
src/lib/state/o...oard-session.ts 90%
src/lib/inference/local.ts 77%
src/lib/sandbox/config.ts 72%
src/lib/inference/nim.ts 72%
src/lib/onboard/preflight.ts 64%
src/lib/state/sandbox.ts 55%
src/lib/onboard...er-gpu-patch.ts 50%
src/lib/actions...licy-channel.ts 49%
src/lib/policy/index.ts 48%
src/lib/onboard.ts 17%

Updated June 13, 2026 23:33 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 5 worth checking, 0 nice ideas
Since last review: 1 prior item resolved, 4 still apply, 0 new items found

Review findings

🛠️ Needs attention

  • None.

🔎 Worth checking

  • Source-of-truth review needed: OpenClaw approval recovery: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: scripts/lib/openclaw_device_approval_policy.py adds recover_failed_scope_approval(), while scripts/nemoclaw-start.sh still embeds a separate state-repair implementation.
  • Source-of-truth review needed: Crash-loop guard assertion: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: gateway_guards_active() now treats proxy-env guard exports plus gateway_pid() as success and no longer checks fresh preload markers.
  • Security-sensitive approval recovery remains duplicated outside the shared helper (scripts/nemoclaw-start.sh:2464): The PR adds and tests a shared recover_failed_scope_approval() helper, and auto-pair now uses it, but the guarded openclaw devices approve wrapper still embeds a separate Python implementation that mutates the same pending.json and paired.json authorization state. This path can grant operator.read/operator.write to already paired devices, so future drift between the implementations could create an inconsistent recovery decision or policy bypass.
    • Recommendation: Prefer calling scripts/lib/openclaw_device_approval_policy.py from the startup wrapper. If that is not practical, add parity/golden coverage proving the wrapper and shared helper make identical decisions for original-pending, replacement-pending with and without requestId output, ambiguous replacements, already-applied-after-nonzero, disallowed/admin scopes, missing operator.pairing, and authorization-denied output.
    • Evidence: scripts/lib/openclaw_device_approval_policy.py implements recover_failed_scope_approval(), while scripts/nemoclaw-start.sh still contains embedded branches for openclaw-approve-recovered-replacement and openclaw-approve-recovered-original.
  • Crash-loop assertion no longer proves the recovered gateway loaded the guard chain (test/e2e/test-issue-2478-crash-loop-recovery.sh:222): gateway_guards_active() now treats /tmp/nemoclaw-proxy-env.sh plus a live gateway PID as proof that the guard chain is active. That proves the launch configuration exists and a gateway process is alive, but a stale or correct proxy-env file with an unguarded recovered gateway would still pass. For [DGX Spark] Gateway crash loop on startup: @homebridge/ciao networkInterfaces() returns EPERM in OpenShell sandbox #2478-style crash recovery, the important behavior is that the recovered process actually loaded the safety-net and ciao preloads.
    • Recommendation: Keep the proxy-env and inference.local checks, but add a runtime proof tied to the recovered gateway process, such as a process-emitted guard marker, preload-created marker, or another deterministic effect that can only occur after that process loads the safety-net and ciao guard chain.
    • Evidence: The updated gateway_guards_active() polls proxy_env_contents() for guard names and checks gateway_pid(), while the previous fresh gateway.log preload marker checks were removed.
  • Kimi failure artifact redaction remains best-effort for a live NVIDIA credential (.github/workflows/nightly-e2e.yaml:886): The Kimi job now runs against the public NVIDIA endpoint with a real nvapi-shaped credential and uploads failure logs after a sanitizer step. The sanitizer redacts exact environment values and common token regexes, which is useful, but it may miss transformed, encoded, truncated, split, or differently formatted credentials emitted by tooling before artifacts are uploaded.
    • Recommendation: Keep the artifact paths narrow and add a static/unit check for the Kimi sanitizer that covers exact NVIDIA_API_KEY values, nvapi-shaped tokens, GitHub-shaped tokens, and any documented transformed or encoded credential forms that the Kimi/onboard tooling can emit.
    • Evidence: The sanitizer replaces $NVIDIA_API_KEY, $GITHUB_TOKEN, nvapi-* patterns, and gh[pousr]_* patterns before uploading three Kimi logs, but the PR does not add sanitizer-specific coverage for transformed credential forms.

🌱 Nice ideas

  • None.
Consider writing more tests for
  • **Runtime validation** — nemoclaw-start embedded approval wrapper and openclaw_device_approval_policy recover original-pending, replacement-pending, ambiguous replacement, already-applied-after-nonzero, disallowed-scope, missing-operator.pairing, and authorization-denied states identically. The PR changes sandbox authorization repair, startup shell wrappers, workflow secret handling, live inference E2E behavior, and crash-recovery assertions. Unit tests are useful, but runtime/source-of-truth validation remains important for these high-risk surfaces.
  • **Runtime validation** — crash-loop recovery assertion fails when proxy-env contains guard exports but the recovered gateway process did not load the safety-net and ciao preload effects. The PR changes sandbox authorization repair, startup shell wrappers, workflow secret handling, live inference E2E behavior, and crash-recovery assertions. Unit tests are useful, but runtime/source-of-truth validation remains important for these high-risk surfaces.
  • **Runtime validation** — Kimi failure sanitizer redacts exact NVIDIA_API_KEY, nvapi-shaped tokens, GitHub-shaped tokens, and documented transformed or encoded credential forms before artifact upload. The PR changes sandbox authorization repair, startup shell wrappers, workflow secret handling, live inference E2E behavior, and crash-recovery assertions. Unit tests are useful, but runtime/source-of-truth validation remains important for these high-risk surfaces.
  • **Runtime validation** — workflow_dispatch with non-empty target_ref withholds Kimi NVIDIA_API_KEY in both the run and sanitize steps. The PR changes sandbox authorization repair, startup shell wrappers, workflow secret handling, live inference E2E behavior, and crash-recovery assertions. Unit tests are useful, but runtime/source-of-truth validation remains important for these high-risk surfaces.
  • **Acceptance clause:** Stabilizes the nightly E2E follow-up failures by keeping the Kimi scenario on the public NVIDIA Kimi endpoint, recognizing the generated routed Kimi model ref, and repairing narrow OpenClaw scope-approval failures that report nonzero after local state changes. — add test evidence or identify existing coverage. .github/workflows/nightly-e2e.yaml and test/e2e/test-kimi-inference-compat.sh default Kimi to live NVIDIA with moonshotai/kimi-k2.6; the plugin recognizes inference/moonshotai/kimi-k2.6 with a unit test; approval recovery is added and tested. Remaining gap: approval recovery still has duplicated implementations without parity coverage.
  • **Acceptance clause:** It also aligns the crash-loop recovery assertion with the current gateway guard markers and keeps the test-size budget ratcheted after moving coverage into a focused policy test. — add test evidence or identify existing coverage. ci/test-file-size-budget.json ratchets test/nemoclaw-start.test.ts from 5231 to 5230 and test/openclaw-device-approval-policy.test.ts adds focused coverage. The crash-loop assertion is updated, but now proves proxy-env configuration plus a live process rather than runtime guard preload activation.
  • **Acceptance clause:** Keep the Kimi E2E on public NVIDIA Endpoints via `nvidia-prod` with `moonshotai/kimi-k2.6`; retain the local mock only behind `NEMOCLAW_KIMI_USE_MOCK=1` and sanitize Kimi failure logs. — add test evidence or identify existing coverage. The workflow and shell test default to live mode, require an nvapi-* NVIDIA_API_KEY, and check the OpenShell route for nvidia-prod and the Kimi model; the mock path is gated by NEMOCLAW_KIMI_USE_MOCK=1. Sanitization is present before artifact upload, but remains best-effort and lacks dedicated transformed-token coverage.
  • **Acceptance clause:** Add constrained OpenClaw approval recovery for failed allowlisted scope upgrades that leave original or replacement pending state, without granting `operator.admin`. — add test evidence or identify existing coverage. scripts/lib/openclaw_device_approval_policy.py constrains recovered scopes to operator.pairing/read/write and tests assert operator.admin is not persisted. Existing and new tests cover several original and replacement paths, but the duplicated startup wrapper lacks full parity coverage against the shared helper.
Since last review details

Current findings:

  • Source-of-truth review needed: OpenClaw approval recovery: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: scripts/lib/openclaw_device_approval_policy.py adds recover_failed_scope_approval(), while scripts/nemoclaw-start.sh still embeds a separate state-repair implementation.
  • Source-of-truth review needed: Crash-loop guard assertion: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: gateway_guards_active() now treats proxy-env guard exports plus gateway_pid() as success and no longer checks fresh preload markers.
  • Security-sensitive approval recovery remains duplicated outside the shared helper (scripts/nemoclaw-start.sh:2464): The PR adds and tests a shared recover_failed_scope_approval() helper, and auto-pair now uses it, but the guarded openclaw devices approve wrapper still embeds a separate Python implementation that mutates the same pending.json and paired.json authorization state. This path can grant operator.read/operator.write to already paired devices, so future drift between the implementations could create an inconsistent recovery decision or policy bypass.
    • Recommendation: Prefer calling scripts/lib/openclaw_device_approval_policy.py from the startup wrapper. If that is not practical, add parity/golden coverage proving the wrapper and shared helper make identical decisions for original-pending, replacement-pending with and without requestId output, ambiguous replacements, already-applied-after-nonzero, disallowed/admin scopes, missing operator.pairing, and authorization-denied output.
    • Evidence: scripts/lib/openclaw_device_approval_policy.py implements recover_failed_scope_approval(), while scripts/nemoclaw-start.sh still contains embedded branches for openclaw-approve-recovered-replacement and openclaw-approve-recovered-original.
  • Crash-loop assertion no longer proves the recovered gateway loaded the guard chain (test/e2e/test-issue-2478-crash-loop-recovery.sh:222): gateway_guards_active() now treats /tmp/nemoclaw-proxy-env.sh plus a live gateway PID as proof that the guard chain is active. That proves the launch configuration exists and a gateway process is alive, but a stale or correct proxy-env file with an unguarded recovered gateway would still pass. For [DGX Spark] Gateway crash loop on startup: @homebridge/ciao networkInterfaces() returns EPERM in OpenShell sandbox #2478-style crash recovery, the important behavior is that the recovered process actually loaded the safety-net and ciao preloads.
    • Recommendation: Keep the proxy-env and inference.local checks, but add a runtime proof tied to the recovered gateway process, such as a process-emitted guard marker, preload-created marker, or another deterministic effect that can only occur after that process loads the safety-net and ciao guard chain.
    • Evidence: The updated gateway_guards_active() polls proxy_env_contents() for guard names and checks gateway_pid(), while the previous fresh gateway.log preload marker checks were removed.
  • Kimi failure artifact redaction remains best-effort for a live NVIDIA credential (.github/workflows/nightly-e2e.yaml:886): The Kimi job now runs against the public NVIDIA endpoint with a real nvapi-shaped credential and uploads failure logs after a sanitizer step. The sanitizer redacts exact environment values and common token regexes, which is useful, but it may miss transformed, encoded, truncated, split, or differently formatted credentials emitted by tooling before artifacts are uploaded.
    • Recommendation: Keep the artifact paths narrow and add a static/unit check for the Kimi sanitizer that covers exact NVIDIA_API_KEY values, nvapi-shaped tokens, GitHub-shaped tokens, and any documented transformed or encoded credential forms that the Kimi/onboard tooling can emit.
    • Evidence: The sanitizer replaces $NVIDIA_API_KEY, $GITHUB_TOKEN, nvapi-* patterns, and gh[pousr]_* patterns before uploading three Kimi logs, but the PR does not add sanitizer-specific coverage for transformed credential forms.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: kimi-inference-compat-e2e, issue-4462-scope-upgrade-approval-e2e, issue-4462-gateway-pinned-approval-characterization-e2e, device-auth-health-e2e
Optional E2E: sandbox-survival-e2e, openclaw-slack-pairing-e2e, openclaw-discord-pairing-e2e, issue-2478-crash-loop-recovery-e2e

Dispatch hint: kimi-inference-compat-e2e,issue-4462-scope-upgrade-approval-e2e,issue-4462-gateway-pinned-approval-characterization-e2e,device-auth-health-e2e

Auto-dispatched E2E: kimi-inference-compat-e2e via nightly-e2e.yaml at 8ca0ccdc125f27ea3b6f45474b3d90b482fb560dnightly run

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • kimi-inference-compat-e2e (medium): Directly covers the changed Kimi plugin, the changed Kimi E2E script, the live NVIDIA Endpoints route, mock fallback assumptions, safe exec splitting, and trajectory verification.
  • issue-4462-scope-upgrade-approval-e2e (medium): Directly covers real CLI scope-upgrade approval behavior, which is the primary runtime path changed in openclaw_device_approval_policy.py, auto-pair-approval.ts, and the in-sandbox openclaw wrapper.
  • issue-4462-gateway-pinned-approval-characterization-e2e (medium): Exercises the legacy gateway-pinned approval failure/recovery shape that this PR explicitly extends with local state recovery for original or replacement pending approvals.
  • device-auth-health-e2e (medium): Validates the real sandbox device-auth health surface after changes to approval policy, auto-pair recovery, scope grants, and token scope mutation.

Optional E2E

  • sandbox-survival-e2e (medium): Useful broad confidence check because scripts/nemoclaw-start.sh is sandbox runtime startup code; verifies onboard, gateway restart, sandbox survival, workspace, and inference still work after the approval-wrapper changes.
  • openclaw-slack-pairing-e2e (medium): Adjacent pairing coverage for OpenClaw state roots and approval behavior; not as direct as the OpenClaw CLI scope-upgrade approval deadlocks and forces openclaw agent into embedded fallback #4462 jobs but useful if maintainers want extra confidence in cross-root pairing flows.
  • openclaw-discord-pairing-e2e (medium): Adjacent pairing coverage for another messaging provider and OpenClaw state-root approval path; optional because the PR does not directly change Discord-specific code.
  • issue-2478-crash-loop-recovery-e2e (medium): Optional because the PR modifies this E2E script's guard-chain assertion and touches the shared sandbox start script, but the runtime changes are not primarily in gateway crash-loop recovery.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: .github/workflows/nightly-e2e.yaml
  • jobs input: kimi-inference-compat-e2e,issue-4462-scope-upgrade-approval-e2e,issue-4462-gateway-pinned-approval-characterization-e2e,device-auth-health-e2e

@github-actions

github-actions Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: ubuntu-repo-cloud-openclaw
Optional Vitest E2E scenarios: ubuntu-repo-docker-post-reboot-recovery

Dispatch required Vitest E2E scenarios:

  • gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: medium

Required Vitest E2E scenarios

  • ubuntu-repo-cloud-openclaw: Core sandbox startup, OpenClaw device approval policy, and connect-time auto-pair approval code changed. The live-supported Ubuntu cloud OpenClaw scenario is the smallest Vitest scenario path that onboards a real OpenClaw sandbox and exercises the smoke, inference, and credential surfaces affected by these changes.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Optional Vitest E2E scenarios

  • ubuntu-repo-docker-post-reboot-recovery: Adjacent live-supported lifecycle coverage for sandbox/container recovery invariants after changes in sandbox startup and gateway/device approval handling. Useful if reviewers want additional confidence beyond the primary OpenClaw onboarding path.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-docker-post-reboot-recovery

Relevant changed files

  • nemoclaw-blueprint/openclaw-plugins/kimi-inference-compat/index.js
  • scripts/lib/openclaw_device_approval_policy.py
  • scripts/nemoclaw-start.sh
  • src/lib/actions/sandbox/auto-pair-approval.ts

@github-actions

Copy link
Copy Markdown
Contributor

Selective E2E Results — ❌ Some jobs failed

Run: 27481775942
Target ref: f86f9def524dc116a6386ad7d882a8d6cecbafc9
Workflow ref: main
Requested jobs: kimi-inference-compat-e2e
Summary: 0 passed, 1 failed, 0 cancelled, 0 skipped

Job Result
kimi-inference-compat-e2e ❌ failure

Failed jobs: kimi-inference-compat-e2e. Check run artifacts for logs.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
scripts/nemoclaw-start.sh (1)

1882-2029: Run the sandbox boot E2Es for these entrypoint changes.

These recovery branches only execute inside the sandbox entrypoint/watchers, so unit coverage will not exercise Landlock, non-root, and process-lifecycle behavior. Please run sandbox-survival-e2e, sandbox-operations-e2e, cloud-e2e, and openclaw-slack-pairing-e2e before merge.

As per coding guidelines, scripts/nemoclaw-start.sh: "This file is a sandbox entrypoint script. Changes affect every sandbox boot and are invisible to unit tests."

Also applies to: 2424-2486

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/nemoclaw-start.sh` around lines 1882 - 2029, The new recovery logic
in scripts/nemoclaw-start.sh should be validated with the sandbox boot
end-to-end suites, since this entrypoint/watcher behavior is not covered by unit
tests. Please run sandbox-survival-e2e, sandbox-operations-e2e, cloud-e2e, and
openclaw-slack-pairing-e2e against the changes around run() and the
pending-approval recovery branch to verify Landlock, non-root, and
process-lifecycle behavior before merge.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/lib/openclaw_device_approval_policy.py`:
- Around line 211-213: The current code deletes the original pending entry using
request_id, but the pending dict is keyed by arbitrary labels (like 'original',
'replacement') not by request_id, leaving stale entries behind. In
scripts/lib/openclaw_device_approval_policy.py at lines 211-213, replace the
pending.pop(request_id, None) call to instead remove the original_key when it
exists, then conditionally remove recovery_key only if it differs from
original_key. In scripts/nemoclaw-start.sh at lines 2481-2482, mirror the same
key-based deletion logic in the inline shell recovery path so the interactive
openclaw devices approve command clears both stale entries consistently.

In `@test/e2e/test-issue-2478-crash-loop-recovery.sh`:
- Around line 227-253: Remove the stale third argument from every
gateway_guards_active call and delete the now-unused gateway_log_marker setup.
The function now only accepts pid and timeout, so update the call sites in
test/e2e/test-issue-2478-crash-loop-recovery.sh that still pass guard_log_start,
negative_guard_log_start, and restore_guard_log_start, and keep the
gateway_guards_active helper aligned with its 2-parameter signature.

In `@test/e2e/test-kimi-inference-compat.sh`:
- Around line 112-123: The ensure_public_nvidia_key() function currently
validates only NVIDIA_INFERENCE_API_KEY against the nvapi-* pattern, but it
should validate that at least one of the two alias variables
(NVIDIA_INFERENCE_API_KEY or NVIDIA_API_KEY) matches the pattern. After the
synchronization logic copies values between the two variables, update the final
validation check to return success if either NVIDIA_INFERENCE_API_KEY or
NVIDIA_API_KEY matches the nvapi-* pattern, ensuring both keys are treated as
valid alternatives for the validation.

---

Nitpick comments:
In `@scripts/nemoclaw-start.sh`:
- Around line 1882-2029: The new recovery logic in scripts/nemoclaw-start.sh
should be validated with the sandbox boot end-to-end suites, since this
entrypoint/watcher behavior is not covered by unit tests. Please run
sandbox-survival-e2e, sandbox-operations-e2e, cloud-e2e, and
openclaw-slack-pairing-e2e against the changes around run() and the
pending-approval recovery branch to verify Landlock, non-root, and
process-lifecycle behavior before merge.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 05b49698-6094-4adc-8ec8-214535bfd4de

📥 Commits

Reviewing files that changed from the base of the PR and between cc1fa5c and f86f9de.

📒 Files selected for processing (12)
  • .github/workflows/nightly-e2e.yaml
  • ci/test-file-size-budget.json
  • nemoclaw-blueprint/openclaw-plugins/kimi-inference-compat/index.js
  • scripts/lib/openclaw_device_approval_policy.py
  • scripts/nemoclaw-start.sh
  • src/lib/actions/sandbox/auto-pair-approval.test.ts
  • src/lib/actions/sandbox/auto-pair-approval.ts
  • test/e2e/test-issue-2478-crash-loop-recovery.sh
  • test/e2e/test-kimi-inference-compat.sh
  • test/kimi-inference-compat-plugin.test.ts
  • test/nemoclaw-start.test.ts
  • test/openclaw-device-approval-policy.test.ts
💤 Files with no reviewable changes (1)
  • test/nemoclaw-start.test.ts

Comment on lines +211 to +213
pending.pop(request_id, None)
if recovery_key:
pending.pop(recovery_key, None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Delete the original pending entry by its map key, not by requestId.

Both recovery implementations mutate pending.json as a dict keyed by arbitrary labels (original, replacement, ...), but the success path removes the original request with pending.pop(request_id, None). When OpenClaw leaves both the original pending entry and a replacement entry behind, the replacement branch reports recovery success while the original entry stays pending.

  • scripts/lib/openclaw_device_approval_policy.py#L211-L213: remove original_key when it exists, then remove recovery_key only if it is different from the original key.
  • scripts/nemoclaw-start.sh#L2481-L2482: mirror the same key-based deletion in the inline shell recovery path so interactive openclaw devices approve clears both stale entries too.
📍 Affects 2 files
  • scripts/lib/openclaw_device_approval_policy.py#L211-L213 (this comment)
  • scripts/nemoclaw-start.sh#L2481-L2482
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/lib/openclaw_device_approval_policy.py` around lines 211 - 213, The
current code deletes the original pending entry using request_id, but the
pending dict is keyed by arbitrary labels (like 'original', 'replacement') not
by request_id, leaving stale entries behind. In
scripts/lib/openclaw_device_approval_policy.py at lines 211-213, replace the
pending.pop(request_id, None) call to instead remove the original_key when it
exists, then conditionally remove recovery_key only if it differs from
original_key. In scripts/nemoclaw-start.sh at lines 2481-2482, mirror the same
key-based deletion logic in the inline shell recovery path so the interactive
openclaw devices approve command clears both stale entries consistently.

Comment on lines 227 to 253
gateway_guards_active() {
local pid="$1"
local timeout="${2:-30}"
local log_boundary="${3:-0}"
local elapsed=0

if [ -z "$pid" ]; then
return 1
fi

local env_contents
env_contents="$(proxy_env_contents)"
if ! echo "$env_contents" | grep -q 'nemoclaw-sandbox-safety-net'; then
echo " [guards] proxy-env.sh missing safety-net export"
return 1
fi
if ! echo "$env_contents" | grep -q 'nemoclaw-ciao-network-guard'; then
echo " [guards] proxy-env.sh missing ciao-network-guard export"
return 1
fi

while [ "$elapsed" -lt "$timeout" ]; do
if gateway_log_after_boundary "$log_boundary" | grep -Eq '\[sandbox-safety-net\] loaded \((openclaw-gateway|launcher)\)' \
&& gateway_log_after_boundary "$log_boundary" | grep -Eq '\[guard\] ciao-network-guard loaded \((openclaw-gateway|launcher)\)'; then
# Confirm gateway is still alive after guard activations.
if [ -n "$(gateway_pid)" ]; then
return 0
fi
echo " [guards] guard fired but gateway no longer running"
return 1
fi
# Backward-compatible proof for older images: this line is emitted by
# the ciao preload only when ciao calls os.networkInterfaces().
if gateway_log_after_boundary "$log_boundary" | grep -Fq '[guard] os.networkInterfaces() failed:'; then
local env_contents
env_contents="$(proxy_env_contents)"
if echo "$env_contents" | grep -q 'nemoclaw-sandbox-safety-net' \
&& echo "$env_contents" | grep -q 'nemoclaw-ciao-network-guard'; then
if [ -n "$(gateway_pid)" ]; then
return 0
fi
echo " [guards] guard fired but gateway no longer running"
echo " [guards] proxy-env.sh has guard exports but gateway no longer running"
return 1
fi
sleep 3
elapsed=$((elapsed + 3))
done

echo " [guards] no fresh gateway-process guard activation signatures in gateway.log within ${timeout}s"
echo " [guards] proxy-env.sh missing safety-net or ciao guard exports within ${timeout}s"
return 1
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Find all gateway_guards_active call sites and gateway_log_marker assignments

# Find all calls to gateway_guards_active
echo "=== gateway_guards_active call sites ==="
rg -n 'gateway_guards_active\s+' test/e2e/test-issue-2478-crash-loop-recovery.sh

# Find all gateway_log_marker assignments that capture into *_guard_log_start variables
echo ""
echo "=== gateway_log_marker assignments to *guard_log_start variables ==="
rg -n '(guard_log_start|negative_guard_log_start|restore_guard_log_start)=.*gateway_log_marker' test/e2e/test-issue-2478-crash-loop-recovery.sh

Repository: NVIDIA/NemoClaw

Length of output: 729


🏁 Script executed:

# Read the function definition to confirm its signature
sed -n '227,253p' test/e2e/test-issue-2478-crash-loop-recovery.sh

Repository: NVIDIA/NemoClaw

Length of output: 771


🏁 Script executed:

# Check if the marker variables are used anywhere beyond their assignment and the call sites
echo "=== Checking for other uses of guard_log_start ==="
rg -n 'guard_log_start' test/e2e/test-issue-2478-crash-loop-recovery.sh

echo ""
echo "=== Checking for other uses of negative_guard_log_start ==="
rg -n 'negative_guard_log_start' test/e2e/test-issue-2478-crash-loop-recovery.sh

echo ""
echo "=== Checking for other uses of restore_guard_log_start ==="
rg -n 'restore_guard_log_start' test/e2e/test-issue-2478-crash-loop-recovery.sh

Repository: NVIDIA/NemoClaw

Length of output: 1116


Function signature change incomplete — four call sites still pass the removed log_boundary parameter.

The function signature was updated to accept only 2 parameters (pid and timeout), but the following call sites still pass three arguments:

  • Line 556: gateway_guards_active "$new_pid" 30 "$guard_log_start"
  • Line 639: gateway_guards_active "$NEGATIVE_PID" 30 "$negative_guard_log_start"
  • Line 665: gateway_guards_active "$NEGATIVE_PID" 5 "$negative_guard_log_start"
  • Line 685: gateway_guards_active "$SOAK_START_PID" 30 "$restore_guard_log_start"

While bash silently ignores extra arguments, the variables guard_log_start (line 528), negative_guard_log_start (line 600), and restore_guard_log_start (line 674) are computed via gateway_log_marker() but never used—they only serve as dead 3rd arguments.

Remove the third argument from all four call sites and delete the unused gateway_log_marker() invocations.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/test-issue-2478-crash-loop-recovery.sh` around lines 227 - 253,
Remove the stale third argument from every gateway_guards_active call and delete
the now-unused gateway_log_marker setup. The function now only accepts pid and
timeout, so update the call sites in
test/e2e/test-issue-2478-crash-loop-recovery.sh that still pass guard_log_start,
negative_guard_log_start, and restore_guard_log_start, and keep the
gateway_guards_active helper aligned with its 2-parameter signature.

Comment thread test/e2e/test-kimi-inference-compat.sh Outdated
Comment on lines +112 to +123
ensure_public_nvidia_key() {
if [ -z "${NVIDIA_INFERENCE_API_KEY:-}" ] && [ -n "${NVIDIA_API_KEY:-}" ]; then
export NVIDIA_INFERENCE_API_KEY="$NVIDIA_API_KEY"
fi
if [ -z "${NVIDIA_API_KEY:-}" ] && [ -n "${NVIDIA_INFERENCE_API_KEY:-}" ]; then
export NVIDIA_API_KEY="$NVIDIA_INFERENCE_API_KEY"
fi
if [ -n "${NVIDIA_INFERENCE_API_KEY:-}" ] && [[ "${NVIDIA_INFERENCE_API_KEY}" == nvapi-* ]]; then
return 0
fi
return 1
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Validate both NVIDIA key aliases against the same rule before returning success.

ensure_public_nvidia_key() currently succeeds/fails solely based on NVIDIA_INFERENCE_API_KEY after one-way fill. If both vars are set and only NVIDIA_API_KEY is valid, the function can fail despite the “either key” contract.

Suggested fix
 ensure_public_nvidia_key() {
-  if [ -z "${NVIDIA_INFERENCE_API_KEY:-}" ] && [ -n "${NVIDIA_API_KEY:-}" ]; then
-    export NVIDIA_INFERENCE_API_KEY="$NVIDIA_API_KEY"
-  fi
-  if [ -z "${NVIDIA_API_KEY:-}" ] && [ -n "${NVIDIA_INFERENCE_API_KEY:-}" ]; then
-    export NVIDIA_API_KEY="$NVIDIA_INFERENCE_API_KEY"
-  fi
-  if [ -n "${NVIDIA_INFERENCE_API_KEY:-}" ] && [[ "${NVIDIA_INFERENCE_API_KEY}" == nvapi-* ]]; then
-    return 0
-  fi
-  return 1
+  local candidate=""
+  if [ -n "${NVIDIA_API_KEY:-}" ] && [[ "${NVIDIA_API_KEY}" == nvapi-* ]]; then
+    candidate="${NVIDIA_API_KEY}"
+  elif [ -n "${NVIDIA_INFERENCE_API_KEY:-}" ] && [[ "${NVIDIA_INFERENCE_API_KEY}" == nvapi-* ]]; then
+    candidate="${NVIDIA_INFERENCE_API_KEY}"
+  else
+    return 1
+  fi
+  export NVIDIA_API_KEY="${candidate}"
+  export NVIDIA_INFERENCE_API_KEY="${candidate}"
+  return 0
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
ensure_public_nvidia_key() {
if [ -z "${NVIDIA_INFERENCE_API_KEY:-}" ] && [ -n "${NVIDIA_API_KEY:-}" ]; then
export NVIDIA_INFERENCE_API_KEY="$NVIDIA_API_KEY"
fi
if [ -z "${NVIDIA_API_KEY:-}" ] && [ -n "${NVIDIA_INFERENCE_API_KEY:-}" ]; then
export NVIDIA_API_KEY="$NVIDIA_INFERENCE_API_KEY"
fi
if [ -n "${NVIDIA_INFERENCE_API_KEY:-}" ] && [[ "${NVIDIA_INFERENCE_API_KEY}" == nvapi-* ]]; then
return 0
fi
return 1
}
ensure_public_nvidia_key() {
local candidate=""
if [ -n "${NVIDIA_API_KEY:-}" ] && [[ "${NVIDIA_API_KEY}" == nvapi-* ]]; then
candidate="${NVIDIA_API_KEY}"
elif [ -n "${NVIDIA_INFERENCE_API_KEY:-}" ] && [[ "${NVIDIA_INFERENCE_API_KEY}" == nvapi-* ]]; then
candidate="${NVIDIA_INFERENCE_API_KEY}"
else
return 1
fi
export NVIDIA_API_KEY="${candidate}"
export NVIDIA_INFERENCE_API_KEY="${candidate}"
return 0
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/test-kimi-inference-compat.sh` around lines 112 - 123, The
ensure_public_nvidia_key() function currently validates only
NVIDIA_INFERENCE_API_KEY against the nvapi-* pattern, but it should validate
that at least one of the two alias variables (NVIDIA_INFERENCE_API_KEY or
NVIDIA_API_KEY) matches the pattern. After the synchronization logic copies
values between the two variables, update the final validation check to return
success if either NVIDIA_INFERENCE_API_KEY or NVIDIA_API_KEY matches the nvapi-*
pattern, ensuring both keys are treated as valid alternatives for the
validation.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

Selective E2E Results — ❌ Some jobs failed

Run: 27482229875
Target ref: 0444194cf24fa21f27bd594100885c24866c1747
Workflow ref: main
Requested jobs: kimi-inference-compat-e2e
Summary: 0 passed, 1 failed, 0 cancelled, 0 skipped

Job Result
kimi-inference-compat-e2e ❌ failure

Failed jobs: kimi-inference-compat-e2e. Check run artifacts for logs.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

Selective E2E Results — ❌ Some jobs failed

Run: 27482361445
Target ref: 8ca0ccdc125f27ea3b6f45474b3d90b482fb560d
Workflow ref: main
Requested jobs: kimi-inference-compat-e2e
Summary: 0 passed, 1 failed, 0 cancelled, 0 skipped

Job Result
kimi-inference-compat-e2e ❌ failure

Failed jobs: kimi-inference-compat-e2e. Check run artifacts for logs.

@cv cv merged commit b0b1362 into main Jun 13, 2026
45 checks passed
@cv cv deleted the codex/fix-nightly-e2e-followups branch June 13, 2026 23:39
cv added a commit that referenced this pull request Jun 14, 2026
## Summary
Declares `supportsStore: false` in the OpenClaw Kimi K2.6 model-specific
setup manifest so managed NVIDIA/Kimi routes get the same compatibility
shape as the previous custom mock path. This fixes the live
`kimi-inference-compat-e2e` K2 config assertion that failed after #5401
moved the scenario to public NVIDIA Endpoints.

## Related Issue
Related to #5401.

## Changes
- Add `supportsStore: false` to
`nemoclaw-blueprint/model-specific-setup/openclaw/kimi-k2.6-managed-inference.json`.
- Update `test/generate-openclaw-config.test.ts` so registry-only Kimi
compat includes the store flag.

## Type of Change
- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Verification
- [ ] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [x] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Tests added or updated for new or changed behavior
- [x] No secrets, API keys, or credentials committed
- [ ] Docs updated for user-facing behavior changes
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

Targeted verification:
- `npx biome check --write
nemoclaw-blueprint/model-specific-setup/openclaw/kimi-k2.6-managed-inference.json
test/generate-openclaw-config.test.ts`
- `npx vitest run --project cli test/generate-openclaw-config.test.ts
test/validate-config-schemas.test.ts --testNamePattern
"Kimi|model-specific|registry compat"`

Docs review: no user-facing docs changes needed; this only completes an
existing model-specific OpenClaw compatibility manifest.

Note: local broad pre-commit/pre-push hooks currently fail in unrelated
runtime recovery preload tests because temp preload files are seen as
group-writable (`mode=664`), matching the existing local hook failure
observed on #5401 follow-up work. Targeted changed tests passed.

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
  * Updated model-specific configuration to disable store support

* **Tests**
  * Updated test case formatting

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant