Skip to content

fix(gateway): honor remote status probe timeout#89859

Merged
steipete merged 1 commit into
openclaw:mainfrom
mushuiyu886:feat/issue-65355-remote-probe-budget
Jun 22, 2026
Merged

fix(gateway): honor remote status probe timeout#89859
steipete merged 1 commit into
openclaw:mainfrom
mushuiyu886:feat/issue-65355-remote-probe-budget

Conversation

@mushuiyu886

@mushuiyu886 mushuiyu886 commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #65355.

openclaw gateway probe --timeout 30000 was still capping active non-loopback probes, including gateway.remote.url / configRemote, at 1500ms. This keeps the short budgets for inactive remote probes, inactive local loopback probes, and SSH tunnel probes, but lets active configured or explicit remote probes use the caller's timeout.

Changes

  • src/commands/gateway-status/helpers.ts: let active status probe targets use the full caller budget before applying inactive/SSH short caps.
  • src/commands/gateway-status/helpers.test.ts: lock the active remote, inactive remote, and SSH tunnel budget cases.
  • src/commands/gateway-status.test.ts: prove the gateway probe command passes a 15000ms caller timeout through to the active configured remote probe.

Current main check

Current origin/main still needs this fix: daily_fix.sh check-upstream-fixed reported upstream_fixed=false. After the required sync-main, the PR remains focused on the same three gateway-status files.

Why it matters / User impact

  • Why it matters / User impact: Remote gateway users can set a larger timeout for slow but reachable configured gateways without gateway probe reporting a false negative after the old 1500ms background cap.
  • What did NOT change: No unrelated gateway server behavior changed; channel delivery, provider/model routing, auth scopes, sessions, retries, config schema, migrations, defaults, inactive remote caps, inactive local caps, and SSH tunnel caps are unchanged.
  • Architecture / source-of-truth check: The source-of-truth resolver is resolveProbeBudgetMs() in src/commands/gateway-status/helpers.ts; all configured, explicit, local, and SSH probe targets pass through this canonical budget boundary before probeGateway() receives timeoutMs, and no protocol/schema/default migration is involved.
  • Contract surface: gateway.remote.url, CLI --url, and --timeout are existing public surfaces; this PR does not add config fields, change schema validation, change defaults, or migrate stored config.
  • Compatibility / defaults: Inactive configured remotes still cap at 1500ms, inactive local loopback still caps at 800ms, and SSH tunnels still cap at 2000ms. Only active configured/explicit targets honor the caller's existing timeout.
  • Related open PR scan: This is the existing linked PR for [Bug]: gateway probe false-negatives on remote configRemote because non-loopback targets are hard-capped to 1500ms #65355; check-upstream-fixed after fetching origin/main reported that main has not covered the fix, so this branch remains the implementation candidate rather than a duplicate replacement.
  • Out of scope: No changes to probeGateway() protocol handling, WebSocket auth/read scopes, service startup, transport/channel delivery, provider routing, or external network retries.

Real behavior proof

  • Behavior or issue addressed: Active configured and explicit remote gateway probe targets now honor the caller timeout instead of being capped at 1500ms, while inactive remote/local and SSH tunnel probe caps remain short.
  • Real environment tested: Linux source checkout at current PR head after sync-main, Node v22.22.0, local OpenClaw dev gateway started from this worktree, and a local TCP proxy bound to a non-loopback host address with a 2.2s initial delay before forwarding to the real gateway.
  • Exact steps or command run after this patch:
node scripts/test-projects.mjs src/commands/gateway-status.test.ts src/commands/gateway-status/helpers.test.ts

OPENCLAW_REPO="$TASK_WORKTREE" \
OPENCLAW_GATEWAY_PORT=19059 \
OPENCLAW_GATEWAY_TOKEN="<redacted-token>" \
OPENCLAW_READY_TIMEOUT_SECONDS=300 \
OPENCLAW_LOG="$EVIDENCE_DIR/live-gateway-start.log" \
OPENCLAW_DEV_PID_FILE="$EVIDENCE_DIR/live-gateway.pid" \
"$START_OPENCLAW"

python3 -u slow_tcp_proxy.py <non-loopback-local-ip> 19060 127.0.0.1 19059 2.2

node openclaw.mjs gateway probe --timeout 5000 --json --token <redacted-token>
# isolated config: gateway.mode=remote, gateway.remote.url=ws://<non-loopback-local-ip>:19060
  • Evidence after fix:
Focused tests after sync:
[test] starting test/vitest/vitest.unit-fast.config.ts
Test Files  1 passed (1)
Tests       17 passed (17)

[test] starting test/vitest/vitest.commands.config.ts
Test Files  1 passed (1)
Tests       25 passed (25)

[test] passed 2 Vitest shards in 52.13s
{
  "command": "node openclaw.mjs gateway probe --timeout 5000 --json --token <redacted-token> (isolated config gateway.mode=remote gateway.remote.url=ws://<non-loopback-local-ip>:19060)",
  "proxyInitialDelayMs": 2200,
  "exitCode": 0,
  "ok": true,
  "targetSummary": [
    {
      "id": "configRemote",
      "kind": "configRemote",
      "active": true,
      "url": "ws://<non-loopback-local-ip>:19060",
      "connect": {
        "ok": true,
        "rpcOk": false,
        "scopeLimited": true,
        "latencyMs": 2250,
        "error": "missing scope: operator.read"
      }
    },
    {
      "id": "localLoopback",
      "kind": "localLoopback",
      "active": false,
      "url": "ws://127.0.0.1:19059",
      "connect": {
        "ok": true,
        "rpcOk": false,
        "scopeLimited": true,
        "latencyMs": 55,
        "error": "missing scope: operator.read"
      }
    }
  ]
}
{
  "command": "node openclaw.mjs gateway probe --timeout 5000 --json --url ws://<non-loopback-local-ip>:19060 --token <redacted-token>",
  "proxyInitialDelayMs": 2200,
  "exitCode": 0,
  "ok": true,
  "targetSummary": [
    {
      "id": "explicit",
      "kind": "explicit",
      "active": true,
      "url": "ws://<non-loopback-local-ip>:19060",
      "connect": {
        "ok": true,
        "rpcOk": false,
        "scopeLimited": true,
        "latencyMs": 2290,
        "error": "missing scope: operator.read"
      }
    }
  ]
}
  • Observed result after fix: The active configRemote probe connected through the non-loopback delayed proxy with latencyMs=2250 under a 5000ms caller timeout, so the live path crossed the old 1500ms cap and still succeeded. The explicit remote path also connected at latencyMs=2290. Inactive local loopback remained a short background probe.
  • What was not tested: The temporary dev token allowed connection but did not provide operator.read, so the proof verifies live WebSocket reachability/timeout budget rather than full read-status RPC output. No third-party channel, Mantis, Desktop session, or external slow network was used.
  • Fix classification: Root cause fix.

Review findings addressed

  • RF-001: Addressed by adding redacted live slow-remote gateway probe output above. The proof uses a real OpenClaw gateway from this worktree behind a 2.2s delayed proxy on a non-loopback URL.
  • RF-002: Addressed. The live configRemote target is active, non-loopback, and connected with latencyMs=2250 under --timeout 5000; that crosses the old 1500ms cap and exercises the production gateway probe path after the patch.
  • RF-003: No separate repair lane is needed; this PR remains the implementation candidate, and no additional automated code repair was requested.

Regression Test Plan

  • Maintainer-ready confidence: High. The diff is XS/S-sized, limited to one budget resolver plus two focused regression test files, and current-head verification includes both Vitest coverage and a live slow configRemote probe.
  • Coverage level: focused unit and command-level Vitest regression tests plus live local gateway/proxy proof.
  • Target test file: src/commands/gateway-status/helpers.test.ts and src/commands/gateway-status.test.ts.
  • Scenario locked in: active configured and explicit remote targets receive the caller timeout; inactive remote/local and SSH tunnel probes remain capped.
  • Why this is the smallest reliable guardrail: the regression lives at the budget resolver that caused the cap and at the command caller that passes the budget into gateway probes.
  • Patch quality notes: The timeout change is not a broad retry increase or fallback. It reorders the canonical budget resolver so active user-selected targets receive the caller's existing timeout, while the previous short caps remain intact for background/inactive probes.

Merge risk

  • Risk labels considered: merge-risk: availability.
  • Risk explanation: Gateway probe/status availability is affected because this code decides how long CLI diagnostics wait for each target. A wrong change could either keep false negatives for slow active remotes or make background probes wait too long.
  • Why acceptable: The branch preserves all inactive and SSH caps, changes only active configured/explicit target budgeting, and verifies the live slow remote path with a 2.2s delayed non-loopback configRemote probe under --timeout 5000.

Root Cause

  • Root cause: The root cause was the ordering inside the source-of-truth resolver resolveProbeBudgetMs(): it checked for non-loopback targets before considering whether the target was the active user-selected remote endpoint, causing active configured remote probes to be forced through the legacy 1500ms background-probe cap.
  • Why this is root-cause fix: The fix changes the source-of-truth budget resolver so active targets receive the caller budget before inactive/background caps apply; tests cover both the resolver and command-level propagation into gateway probe calls.
  • Missing guardrail: Existing tests covered active local loopback using the caller timeout, but encoded active non-loopback targets as capped instead of distinguishing active remote probes from inactive background probes.

@openclaw-barnacle openclaw-barnacle Bot added commands Command implementations size: XS proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 3, 2026
@clawsweeper

clawsweeper Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 22, 2026, 3:44 AM ET / 07:44 UTC.

Summary
The PR changes resolveProbeBudgetMs() so active configured and explicit remote gateway probe targets use the caller timeout while inactive remote/local and SSH tunnel probes keep short caps, with focused helper and command tests.

PR surface: Source 0, Tests +19. Total +19 across 3 files.

Reproducibility: yes. from source. Current main routes active non-loopback gateway-status targets through the 1500ms cap before they can receive the caller budget, and the linked issue plus PR proof show slow reachable remotes crossing that cap.

Review metrics: 1 noteworthy metric.

  • Probe Budget Behavior: 2 active target classes expanded; 3 short-cap cases preserved. Maintainers should notice that active configRemote and explicit remote now use the caller budget while inactive remote, inactive loopback, and SSH tunnel probes remain bounded.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #65355
Summary: This PR is the current implementation candidate for the canonical active remote gateway probe budget bug; the earlier similar PR is closed unmerged and the local-loopback timeout issue is adjacent history.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P2] Active configured and explicit remote probes can now wait up to the caller-supplied timeout, so existing diagnostic scripts that implicitly relied on the old fast false-negative cap may run longer.
  • [P1] Gateway probe/status availability depends on this resolver: an incorrect follow-up could either keep false negatives for slow active remotes or make inactive/background probes wait too long.
  • [P1] Exact-head CI was still unstable at review time because some jobs were queued, so maintainers should wait for the normal merge gate even though no code finding remains.

Maintainer options:

  1. Accept Active-Remote Timeout Semantics (recommended)
    Land the resolver reorder and intentionally allow active configured and explicit remote probes to use the caller timeout while preserving inactive and SSH caps.
  2. Pause For Broader Operator Proof
    Ask for additional maintainer-side proof around long-timeout scripts or mixed SSH/direct setups if the diagnostic latency tradeoff is still uncertain.

Next step before merge

  • [P2] No automated code repair is needed; maintainers should wait for the exact-head merge gate and decide whether to accept the active-remote timeout tradeoff in this existing PR.

Security
Cleared: The diff only changes timeout-budget ordering and focused tests; it does not touch dependencies, workflows, secrets, package metadata, or new code execution surfaces.

Review details

Best possible solution:

Land the focused resolver change if maintainers accept the active-remote diagnostic latency tradeoff, then let it close #65355.

Do we have a high-confidence way to reproduce the issue?

Yes, from source. Current main routes active non-loopback gateway-status targets through the 1500ms cap before they can receive the caller budget, and the linked issue plus PR proof show slow reachable remotes crossing that cap.

Is this the best way to solve the issue?

Yes. resolveProbeBudgetMs() is the narrow owner boundary consumed by runGatewayStatusProbePass(), and the PR changes active remote behavior while preserving inactive remote/local and SSH short caps.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f9ebb8d91bea.

Label changes

Label justifications:

  • P2: This is a normal-priority gateway diagnostics bug fix with limited command-surface blast radius.
  • merge-risk: 🚨 compatibility: The PR intentionally changes effective timeout behavior for existing active configured and explicit remote gateway probe targets.
  • merge-risk: 🚨 availability: Gateway probe/status diagnostics can now wait longer for active remote endpoints when callers provide a larger timeout budget.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes after-fix live output from a real OpenClaw gateway behind a delayed non-loopback TCP proxy, with configRemote and explicit probes crossing the old 1500ms cap under --timeout 5000.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix live output from a real OpenClaw gateway behind a delayed non-loopback TCP proxy, with configRemote and explicit probes crossing the old 1500ms cap under --timeout 5000.
Evidence reviewed

PR surface:

Source 0, Tests +19. Total +19 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 1 6 6 0
Tests 2 21 2 +19
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 27 8 +19

What I checked:

  • Repository policy read: Root AGENTS.md was read fully; its whole-path PR review, source/test/proof, and compatibility-risk guidance applies to this core command/gateway diagnostic change. No scoped AGENTS.md owns src/commands/gateway-status*, and the only maintainer note present is Telegram-specific and not applicable. (AGENTS.md:1, f9ebb8d91bea)
  • Current main still has the cap: On current main, resolveProbeBudgetMs() checks !isLoopbackProbeTarget(target) before active-target handling, so active configRemote and explicit non-loopback targets are capped at Math.min(1500, overallMs). (src/commands/gateway-status/helpers.ts:146, f9ebb8d91bea)
  • Command path uses the resolver directly: runGatewayStatusProbePass() passes resolveProbeBudgetMs(params.overallTimeoutMs, target) directly to probeGateway({ timeoutMs }), so the helper is the active owner boundary for the reported timeout behavior. (src/commands/gateway-status/probe-run.ts:132, f9ebb8d91bea)
  • CLI timeout reaches gateway status: The gateway probe CLI exposes --timeout as the overall probe budget and routes into gatewayStatusCommand(), which builds targets and calls the probe pass. (src/cli/gateway-cli/register.ts:756, f9ebb8d91bea)
  • Current tests encode the old behavior: Current helper tests still expect active configRemote and active explicit remote targets with a 15000ms caller budget to resolve to 1500ms, matching the linked issue's source-level repro. (src/commands/gateway-status/helpers.test.ts:439, f9ebb8d91bea)
  • PR diff is focused: The PR diff only reorders the shared budget resolver and updates two focused test files; active targets now return the full budget before inactive local/remote caps apply, while sshTunnel remains capped first. (src/commands/gateway-status/helpers.ts:150, 6a3c8dc67679)

Likely related people:

  • MonkeyLeeT: Authored the merged loopback timeout change that introduced resolveProbeBudgetMs() and the non-local 1500ms cap this PR narrows. (role: introduced adjacent budget behavior; confidence: high; commits: 5bb5d7dab4b2; files: src/commands/gateway-status/helpers.ts, src/commands/gateway-status/helpers.test.ts, src/commands/gateway-status.test.ts)
  • BryanTegomoh: Authored the recent merged gateway health/probe port handling PR that substantially touched the same gateway-status helper, command, and tests on current main. (role: recent area contributor; confidence: medium; commits: 8e4213b1c418; files: src/commands/gateway-status/helpers.ts, src/commands/gateway-status/helpers.test.ts, src/commands/gateway-status.test.ts)
  • vincentkoc: Merged the recent current-main gateway health/probe port PR that carried forward the resolver and adjacent tests in this checkout's current history. (role: recent merger; confidence: medium; commits: 8e4213b1c418; files: src/commands/gateway-status/helpers.ts, src/commands/gateway-status/helpers.test.ts, src/commands/gateway-status.test.ts)
  • steipete: Posted land-ready maintainer verification for this PR, was assigned, and force-pushed prepared PR heads after the contributor's original branch. (role: recent reviewer/preparer; confidence: medium; commits: 6a3c8dc67679; files: src/commands/gateway-status/helpers.ts, src/commands/gateway-status/helpers.test.ts, src/commands/gateway-status.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jun 3, 2026
@mushuiyu886 mushuiyu886 force-pushed the feat/issue-65355-remote-probe-budget branch from 4308dcf to 5dbcdde Compare June 18, 2026 05:23
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 18, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jun 19, 2026
@steipete steipete self-assigned this Jun 22, 2026
@steipete

steipete commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Land-ready maintainer verification:

  • Reviewed the changed budget resolver with its caller, sibling target types, and existing gateway-status coverage. This is the best fix: active explicit/configured remote targets honor the caller budget; inactive alternatives and SSH discovery retain short caps.
  • Fresh autoreview: clean, no actionable findings.
  • Focused tests after resolving the current-main overlap: node scripts/run-vitest.mjs src/commands/gateway-status/helpers.test.ts src/commands/gateway-status.test.ts — 47 passed.
  • Live Linux E2E: Azure Crabbox run run_d47f28c049b2 placed the configured remote gateway behind a delayed non-loopback TCP proxy. gateway probe --json --timeout 5000 connected after 2254 ms, proving the active remote target is no longer cut off at 1500 ms.
  • Current prepared head: 056707dbc76344a3d579eb7ffb21a0bf392fd283. Fresh post-rebase autoreview is clean; exact-head hosted CI passed in run 27942608836; scripts/pr prepare-run 89859 completed.

Known gaps: none for this narrow timeout-routing change.

@steipete steipete force-pushed the feat/issue-65355-remote-probe-budget branch 7 times, most recently from de6c2f8 to e43072a Compare June 22, 2026 08:54
@steipete steipete force-pushed the feat/issue-65355-remote-probe-budget branch from e43072a to 056707d Compare June 22, 2026 09:21
@steipete steipete merged commit 9ce4c92 into openclaw:main Jun 22, 2026
198 of 199 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

Thanks @mushuiyu886!

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 23, 2026
Merged via squash.

Prepared head SHA: 056707d
Co-authored-by: mushuiyu886 <266724580+mushuiyu886@users.noreply.github.com>
Co-authored-by: steipete <58493+steipete@users.noreply.github.com>
Reviewed-by: @steipete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commands Command implementations merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: gateway probe false-negatives on remote configRemote because non-loopback targets are hard-capped to 1500ms

2 participants