Skip to content

Fix heartbeat pending final delivery cleanup#87583

Closed
lilesjtu wants to merge 1 commit into
openclaw:mainfrom
lilesjtu:codex/fix-heartbeat-pending-replay
Closed

Fix heartbeat pending final delivery cleanup#87583
lilesjtu wants to merge 1 commit into
openclaw:mainfrom
lilesjtu:codex/fix-heartbeat-pending-replay

Conversation

@lilesjtu

@lilesjtu lilesjtu commented May 28, 2026

Copy link
Copy Markdown
Contributor

Problem

Heartbeat runs can persist non-ack final output in pendingFinalDelivery* so the final user-visible message can be retried. Normal reply dispatch clears that pending state after successful delivery, but runHeartbeatOnce sends heartbeat output through sendDurableMessageBatch directly and bypasses the normal cleanup path.

That leaves a stale pendingFinalDeliveryText in the session store after the heartbeat message was already delivered. On later heartbeat wakes, the pending text can be replayed or re-classified before fresh work runs, which is why an old heartbeat report can keep resurfacing instead of disappearing after a successful delivery.

Relationship to #83187

#83187 covers the heartbeat send-success cleanup path and already has strong proof for that narrower case.

This PR additionally covers the duplicate-suppression path: if the same heartbeat payload was already delivered recently (lastHeartbeatText / lastHeartbeatSentAt) but matching stale pendingFinalDelivery* fields remain, the duplicate branch currently returns before the send-success store write and therefore needs its own cleanup. If maintainers prefer #83187 as the canonical landing branch, the duplicate-suppression cleanup and regression coverage from this PR can be ported there.

Fix

  • Add heartbeat-runner cleanup for delivered pending final delivery state.
  • Clear pendingFinalDelivery* only when the stored pending text exactly matches the heartbeat text that was delivered.
  • Run the cleanup after a successful heartbeat send.
  • Also run it when duplicate suppression skips the same payload, because lastHeartbeatText/lastHeartbeatSentAt show that exact heartbeat payload was already sent recently.

Safety

The cleanup is intentionally narrow: it requires pendingFinalDelivery === true and a trimmed exact text match. It does not clear unrelated pending delivery state, and it does not run on failed sends, no-target skips, or alerts-disabled skips.

Verification

  • node scripts/run-vitest.mjs run --config test/vitest/vitest.infra.config.ts src/infra/heartbeat-runner.returns-default-unset.test.ts — 41 tests passed.
  • pnpm exec oxfmt --check --threads=1 src/infra/heartbeat-runner.ts src/infra/heartbeat-runner.returns-default-unset.test.ts — clean.

Real behavior proof

Environment: disposable local OpenClaw source-runtime run on branch codex/fix-heartbeat-pending-replay, head 9497a7a032cf. The run used a temporary config/workspace/session store and the actual runHeartbeatOnce + sendDurableMessageBatch heartbeat path with a Telegram outbound adapter registered in the plugin registry. The model reply was stubbed to deterministic text so the proof isolates heartbeat delivery/session-state behavior. No production OpenClaw data or real chat credentials were used.

Command shape:

node --import tsx --input-type=module <<'EOF'
# imports runHeartbeatOnce from src/infra/heartbeat-runner.ts
# registers Telegram outbound adapter
# seeds temp sessions.json with matching pendingFinalDelivery* fields
# runs send-success and duplicate-suppression heartbeat scenarios
# prints before/after pending field states
EOF

Observed runtime logs:

[heartbeat] cleared delivered pending final delivery
[heartbeat] cleared delivered pending final delivery

Send-success scenario:

{
  "scenario": "send-success",
  "branch": "codex/fix-heartbeat-pending-replay",
  "head": "9497a7a032cf",
  "result": { "status": "ran", "durationMs": 387 },
  "sendCallCount": 1,
  "sends": [
    {
      "to": "proof-chat",
      "text": "PR87583 send-success heartbeat payload",
      "accountId": null
    }
  ],
  "before": {
    "pendingFinalDelivery": true,
    "pendingFinalDeliveryText": "PR87583 send-success heartbeat payload",
    "pendingFinalDeliveryCreatedAt": 1779957645636,
    "pendingFinalDeliveryLastAttemptAt": 1779957705636,
    "pendingFinalDeliveryAttemptCount": 2,
    "pendingFinalDeliveryLastError": "redacted prior delivery failure",
    "pendingFinalDeliveryContext": { "channel": "telegram", "to": "proof-chat" },
    "pendingFinalDeliveryIntentId": "intent-send-success",
    "lastHeartbeatText": "<absent>",
    "lastHeartbeatSentAt": "<absent>"
  },
  "after": {
    "pendingFinalDelivery": "<absent>",
    "pendingFinalDeliveryText": "<absent>",
    "pendingFinalDeliveryCreatedAt": "<absent>",
    "pendingFinalDeliveryLastAttemptAt": "<absent>",
    "pendingFinalDeliveryAttemptCount": "<absent>",
    "pendingFinalDeliveryLastError": "<absent>",
    "pendingFinalDeliveryContext": "<absent>",
    "pendingFinalDeliveryIntentId": "<absent>",
    "lastHeartbeatText": "PR87583 send-success heartbeat payload",
    "lastHeartbeatSentAt": 1779957765636
  },
  "clearedAllPendingFields": true,
  "deliveryEvidence": "sendDurableMessageBatch reached Telegram outbound adapter"
}

Duplicate-suppression scenario:

{
  "scenario": "duplicate-skip",
  "branch": "codex/fix-heartbeat-pending-replay",
  "head": "9497a7a032cf",
  "result": { "status": "ran", "durationMs": 51 },
  "sendCallCount": 0,
  "before": {
    "pendingFinalDelivery": true,
    "pendingFinalDeliveryText": "PR87583 duplicate-skip heartbeat payload",
    "pendingFinalDeliveryCreatedAt": 1779957646044,
    "pendingFinalDeliveryLastAttemptAt": 1779957706044,
    "pendingFinalDeliveryAttemptCount": 2,
    "pendingFinalDeliveryLastError": "redacted prior delivery failure",
    "pendingFinalDeliveryContext": { "channel": "telegram", "to": "proof-chat" },
    "pendingFinalDeliveryIntentId": "intent-duplicate-skip",
    "lastHeartbeatText": "PR87583 duplicate-skip heartbeat payload",
    "lastHeartbeatSentAt": 1779957706044
  },
  "after": {
    "pendingFinalDelivery": "<absent>",
    "pendingFinalDeliveryText": "<absent>",
    "pendingFinalDeliveryCreatedAt": "<absent>",
    "pendingFinalDeliveryLastAttemptAt": "<absent>",
    "pendingFinalDeliveryAttemptCount": "<absent>",
    "pendingFinalDeliveryLastError": "<absent>",
    "pendingFinalDeliveryContext": "<absent>",
    "pendingFinalDeliveryIntentId": "<absent>",
    "lastHeartbeatText": "PR87583 duplicate-skip heartbeat payload",
    "lastHeartbeatSentAt": 1779957706044
  },
  "clearedAllPendingFields": true,
  "deliveryEvidence": "duplicate suppression skipped outbound send because lastHeartbeatText matched within 24h"
}

@openclaw-barnacle openclaw-barnacle Bot added size: M triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. triage: refactor-only Candidate: refactor/cleanup-only PR without maintainer context. labels May 28, 2026
@clawsweeper

clawsweeper Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Keep open, but human-only before merge. The patch targets a real heartbeat pending-final cleanup gap on current main, yet it lacks contributor real-behavior proof and overlaps the proof-positive canonical cleanup path at #83187, with an extra duplicate-suppression cleanup decision maintainers should reconcile.

Canonical path: Close this PR as superseded by #83187.

So I’m closing this here and keeping the remaining discussion on #83187.

Review details

Best possible solution:

Close this PR as superseded by #83187.

Do we have a high-confidence way to reproduce the issue?

Yes at source level, but not with a live run in this read-only review. Current main shows heartbeat send success records lastHeartbeatText without clearing pending final-delivery state, while normal dispatch has a cleanup helper after successful final delivery.

Is this the best way to solve the issue?

Unclear as the final merge path. The send-success cleanup is well aligned with the existing dispatch cleanup, but this PR also adds duplicate-suppression cleanup and overlaps a proof-positive sibling PR, so maintainers should pick or combine one canonical path.

Security review:

Security review cleared: The diff touches heartbeat runtime and tests only; no concrete security or supply-chain regression was found.

AGENTS.md: found and applied where relevant.

What I checked:

  • linked superseding PR: fix(heartbeat): clear pendingFinalDelivery* on send success #83187 (fix(heartbeat): clear pendingFinalDelivery* on send success) is still open as the canonical replacement.
  • cluster evidence: the durable review links that PR in the work cluster or recommended risk path.
  • no human follow-up: live comments and timeline hydrated by apply contain no non-automation activity after the ClawSweeper review.

Likely related people:

  • kesslerio: Authored the recent fix(heartbeat): stop pending final replay commit that directly changed heartbeat pending-final behavior in heartbeat-runner.ts and get-reply.ts. (role: recent adjacent contributor; confidence: high; commits: d00e764e6655; files: src/infra/heartbeat-runner.ts, src/auto-reply/reply/get-reply.ts)
  • MertBasar0: Authored the durable main-session pending-final delivery feature commit that introduced the central pendingFinalDelivery behavior and heartbeat-related cleanup semantics. (role: feature introducer; confidence: medium; commits: c240e718e91e; files: src/auto-reply/reply/agent-runner.ts, src/auto-reply/reply/dispatch-from-config.ts, src/infra/heartbeat-runner.ts)
  • steipete: Recent commits touched session-store writer behavior and large agent/session runtime refactors around the same persisted session-state surface. (role: recent area contributor; confidence: medium; commits: 15b1e99df310, bb46b79d3c14; files: src/config/sessions/store.ts, src/infra/heartbeat-runner.ts, src/auto-reply/reply/get-reply.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against bd02977e294d.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels May 28, 2026
@clawsweeper

clawsweeper Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg: 🎁 locked until real behavior proof passes.

Details
  • No creature or rarity is rolled until proof passes.
  • Eggs are collectible flavor only; they do not affect labels, ratings, merge decisions, or automation.

@clawsweeper clawsweeper Bot added P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels May 28, 2026
@clawsweeper

clawsweeper Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. triage: refactor-only Candidate: refactor/cleanup-only PR without maintainer context.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant