Skip to content

Commit f5e7557

Browse files
committed
fix(heartbeat): defer during cron and nested lane pressure
1 parent 422d139 commit f5e7557

22 files changed

Lines changed: 422 additions & 46 deletions

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Docs: https://docs.openclaw.ai
2121
- Channels/Discord: split long CJK replies at punctuation and code-point-safe fallback boundaries so Discord chunking stays readable without corrupting astral characters. Fixes #38597; repairs #71384. Thanks @p3nchan.
2222
- Browser/gateway: ignore Playwright dialog-close races from `Page.handleJavaScriptDialog` so browser automation no longer crashes the Gateway when a dialog disappears before Playwright accepts it. (#40067) Thanks @randyjtw.
2323
- Cron/Gateway: defer missed isolated agent-turn catch-up out of the channel startup window, so overdue cron work cannot starve Discord or Telegram while providers connect after a restart. Thanks @vincentkoc.
24+
- Heartbeat/cron: defer heartbeat turns while cron work is active or queued, add opt-in `heartbeat.skipWhenBusy` for subagent/nested lane pressure, and retry busy skips without advancing the schedule so local Ollama hosts do not run heartbeat and cron prompts concurrently. Fixes #50773. Thanks @scottgl9.
2425
- Plugins/runtime-deps: prune stale `openclaw-unknown-*` bundled runtime dependency roots during Gateway startup while keeping recent or locked roots, so old staging debris cannot keep growing across restarts. Thanks @vincentkoc.
2526
- Ollama: compose caller abort signals with guarded-fetch timeouts for native `/api/chat` streams, so `/stop` and early cancellation still interrupt local Ollama requests that also carry provider timeout budgets. Refs #74133. Thanks @obviyus.
2627
- Doctor/TTS: migrate legacy `messages.tts.enabled`, agent TTS, channel TTS, and voice-call plugin TTS toggles to `auto` mode during `openclaw doctor --fix`, matching the documented TTS config contract. Thanks @vincentkoc.
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
96edba9fd67fa057f6b6c43d54a168db25d0e27ddd4e91a7e2918c8657f0f212 config-baseline.json
2-
510ed7af2e3731c8a307dbc10181328f82764a4e8dd9e9dddc6118db6f882ff7 config-baseline.core.json
1+
592d25e08647ced4fae0c4fdbff95e50d1749c42d39070f6b6bc6a3e0475d4f0 config-baseline.json
2+
9cd2c40b4a45976b74458f9ada8ecc31c532ee81f10145a9828bbff31777c03e config-baseline.core.json
33
9f5fad66a49fa618d64a963470aa69fed9fe4b4639cc4321f9ec04bfb2f8aa50 config-baseline.channel.json
44
0dd6583fafae6c9134e46c4cf9bddee9822d6436436dcb1a6dcba6d012962e51 config-baseline.plugin.json

docs/automation/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ See [Hooks](/automation/hooks).
9393

9494
### Heartbeat
9595

96-
Heartbeat is a periodic main-session turn (default every 30 minutes). It batches multiple checks (inbox, calendar, notifications) in one agent turn with full session context. Heartbeat turns do not create task records and do not extend daily/idle session reset freshness. Use `HEARTBEAT.md` for a small checklist, or a `tasks:` block when you want due-only periodic checks inside heartbeat itself. Empty heartbeat files skip as `empty-heartbeat-file`; due-only task mode skips as `no-tasks-due`.
96+
Heartbeat is a periodic main-session turn (default every 30 minutes). It batches multiple checks (inbox, calendar, notifications) in one agent turn with full session context. Heartbeat turns do not create task records and do not extend daily/idle session reset freshness. Use `HEARTBEAT.md` for a small checklist, or a `tasks:` block when you want due-only periodic checks inside heartbeat itself. Empty heartbeat files skip as `empty-heartbeat-file`; due-only task mode skips as `no-tasks-due`. Heartbeats defer while cron work is active or queued, and `heartbeat.skipWhenBusy` can also defer them while subagent or nested lanes are busy.
9797

9898
See [Heartbeat](/gateway/heartbeat).
9999

docs/gateway/config-agents.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -516,6 +516,7 @@ Periodic heartbeat runs.
516516
includeSystemPromptSection: true, // default: true; false omits the Heartbeat section from the system prompt
517517
lightContext: false, // default: false; true keeps only HEARTBEAT.md from workspace bootstrap files
518518
isolatedSession: false, // default: false; true runs each heartbeat in a fresh session (no conversation history)
519+
skipWhenBusy: false, // default: false; true also waits for subagent/nested lanes
519520
session: "main",
520521
to: "+15555550123",
521522
directPolicy: "allow", // allow (default) | block
@@ -537,6 +538,7 @@ Periodic heartbeat runs.
537538
- `directPolicy`: direct/DM delivery policy. `allow` (default) permits direct-target delivery. `block` suppresses direct-target delivery and emits `reason=dm-blocked`.
538539
- `lightContext`: when true, heartbeat runs use lightweight bootstrap context and keep only `HEARTBEAT.md` from workspace bootstrap files.
539540
- `isolatedSession`: when true, each heartbeat runs in a fresh session with no prior conversation history. Same isolation pattern as cron `sessionTarget: "isolated"`. Reduces per-heartbeat token cost from ~100K to ~2-5K tokens.
541+
- `skipWhenBusy`: when true, heartbeat runs defer on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeats, even without this flag.
540542
- Per-agent: set `agents.list[].heartbeat`. When any agent defines `heartbeat`, **only those agents** run heartbeats.
541543
- Heartbeats run full agent turns — shorter intervals burn more tokens.
542544

docs/gateway/heartbeat.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ Example config:
5050
directPolicy: "allow", // default: allow direct/DM targets; set "block" to suppress
5151
lightContext: true, // optional: only inject HEARTBEAT.md from bootstrap files
5252
isolatedSession: true, // optional: fresh session each run (no conversation history)
53+
skipWhenBusy: true, // optional: also defer when subagent or nested lanes are busy
5354
// activeHours: { start: "08:00", end: "24:00" },
5455
// includeReasoning: true, // optional: send separate `Reasoning:` message too
5556
},
@@ -65,6 +66,7 @@ Example config:
6566
- The heartbeat prompt is sent **verbatim** as the user message. The system prompt includes a "Heartbeat" section only when heartbeats are enabled for the default agent, and the run is flagged internally.
6667
- When heartbeats are disabled with `0m`, normal runs also omit `HEARTBEAT.md` from bootstrap context so the model does not see heartbeat-only instructions.
6768
- Active hours (`heartbeat.activeHours`) are checked in the configured timezone. Outside the window, heartbeats are skipped until the next tick inside the window.
69+
- Heartbeats automatically defer while cron work is active or queued. Set `heartbeat.skipWhenBusy: true` to defer on extra busy lanes (subagent or nested command work) as well; this is useful for local Ollama and other constrained single-runtime hosts.
6870

6971
## What the heartbeat prompt is for
7072

@@ -98,6 +100,7 @@ Outside heartbeats, stray `HEARTBEAT_OK` at the start/end of a message is stripp
98100
includeReasoning: false, // default: false (deliver separate Reasoning: message when available)
99101
lightContext: false, // default: false; true keeps only HEARTBEAT.md from workspace bootstrap files
100102
isolatedSession: false, // default: false; true runs each heartbeat in a fresh session (no conversation history)
103+
skipWhenBusy: false, // default: false; true also waits for subagent/nested lanes
101104
target: "last", // default: none | options: last | none | <channel id> (core or plugin, e.g. "bluebubbles")
102105
to: "+15551234567", // optional channel-specific override
103106
accountId: "ops-bot", // optional multi-account channel id
@@ -230,6 +233,9 @@ Use `accountId` to target a specific account on multi-account channels like Tele
230233
<ParamField path="isolatedSession" type="boolean" default="false">
231234
When true, each heartbeat runs in a fresh session with no prior conversation history. Uses the same isolation pattern as cron `sessionTarget: "isolated"`. Dramatically reduces per-heartbeat token cost. Combine with `lightContext: true` for maximum savings. Delivery routing still uses the main session context.
232235
</ParamField>
236+
<ParamField path="skipWhenBusy" type="boolean" default="false">
237+
When true, heartbeat runs defer on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeats, even without this flag, so local-model hosts do not run cron and heartbeat prompts at the same time.
238+
</ParamField>
233239
<ParamField path="session" type="string">
234240
Optional session key for heartbeat runs.
235241

@@ -287,7 +293,8 @@ Use `accountId` to target a specific account on multi-account channels like Tele
287293
- `session` only affects the run context; delivery is controlled by `target` and `to`.
288294
- To deliver to a specific channel/recipient, set `target` + `to`. With `target: "last"`, delivery uses the last external channel for that session.
289295
- Heartbeat deliveries allow direct/DM targets by default. Set `directPolicy: "block"` to suppress direct-target sends while still running the heartbeat turn.
290-
- If the main queue is busy, the heartbeat is skipped and retried later.
296+
- If the main queue, target session lane, cron lane, or an active cron job is busy, the heartbeat is skipped and retried later.
297+
- If `skipWhenBusy: true`, subagent and nested lanes also defer heartbeat runs.
291298
- If `target` resolves to no external destination, the run still happens but no outbound message is sent.
292299

293300
</Accordion>

docs/gateway/troubleshooting.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -437,7 +437,7 @@ Look for:
437437

438438
- Cron enabled and next wake present.
439439
- Job run history status (`ok`, `skipped`, `error`).
440-
- Heartbeat skip reasons (`quiet-hours`, `requests-in-flight`, `alerts-disabled`, `empty-heartbeat-file`, `no-tasks-due`).
440+
- Heartbeat skip reasons (`quiet-hours`, `requests-in-flight`, `cron-in-progress`, `lanes-busy`, `alerts-disabled`, `empty-heartbeat-file`, `no-tasks-due`).
441441

442442
<AccordionGroup>
443443
<Accordion title="Common signatures">

src/config/schema.base.generated.ts

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5464,6 +5464,12 @@ export const GENERATED_BASE_CONFIG_SCHEMA: BaseConfigSchemaResponse = {
54645464
isolatedSession: {
54655465
type: "boolean",
54665466
},
5467+
skipWhenBusy: {
5468+
type: "boolean",
5469+
title: "Heartbeat Skip When Busy",
5470+
description:
5471+
"When true, defer heartbeat turns on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeat turns.",
5472+
},
54675473
},
54685474
additionalProperties: false,
54695475
},
@@ -7198,6 +7204,12 @@ export const GENERATED_BASE_CONFIG_SCHEMA: BaseConfigSchemaResponse = {
71987204
isolatedSession: {
71997205
type: "boolean",
72007206
},
7207+
skipWhenBusy: {
7208+
type: "boolean",
7209+
title: "Heartbeat Skip When Busy",
7210+
description:
7211+
"Per-agent override that defers heartbeat turns on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeat turns.",
7212+
},
72017213
},
72027214
additionalProperties: false,
72037215
},
@@ -27261,6 +27273,16 @@ export const GENERATED_BASE_CONFIG_SCHEMA: BaseConfigSchemaResponse = {
2726127273
label: "Heartbeat Timeout (Seconds)",
2726227274
tags: ["performance", "automation"],
2726327275
},
27276+
"agents.defaults.heartbeat.skipWhenBusy": {
27277+
label: "Heartbeat Skip When Busy",
27278+
help: "When true, defer heartbeat turns on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeat turns.",
27279+
tags: ["automation"],
27280+
},
27281+
"agents.list.*.heartbeat.skipWhenBusy": {
27282+
label: "Heartbeat Skip When Busy",
27283+
help: "Per-agent override that defers heartbeat turns on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeat turns.",
27284+
tags: ["automation"],
27285+
},
2726427286
"agents.defaults.sandbox.browser.network": {
2726527287
label: "Sandbox Browser Network",
2726627288
help: "Docker network for sandbox browser containers (default: openclaw-sandbox-browser). Avoid bridge if you need stricter isolation.",
@@ -28419,6 +28441,11 @@ export const GENERATED_BASE_CONFIG_SCHEMA: BaseConfigSchemaResponse = {
2841928441
help: "Per-agent maximum time in seconds allowed for a heartbeat agent turn before it is aborted. Leave unset to inherit the merged heartbeat/default agent timeout.",
2842028442
tags: ["performance", "automation"],
2842128443
},
28444+
"agents.list[].heartbeat.skipWhenBusy": {
28445+
label: "Agent Heartbeat Skip When Busy",
28446+
help: "Per-agent override that defers heartbeat turns on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeat turns.",
28447+
tags: ["automation"],
28448+
},
2842228449
"agents.list[].sandbox.browser.network": {
2842328450
label: "Agent Sandbox Browser Network",
2842428451
help: "Per-agent override for sandbox browser Docker network.",

src/config/schema.help.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,10 @@ export const FIELD_HELP: Record<string, string> = {
256256
"Maximum time in seconds allowed for a heartbeat agent turn before it is aborted. Leave unset to use agents.defaults.timeoutSeconds.",
257257
"agents.list[].heartbeat.timeoutSeconds":
258258
"Per-agent maximum time in seconds allowed for a heartbeat agent turn before it is aborted. Leave unset to inherit the merged heartbeat/default agent timeout.",
259+
"agents.defaults.heartbeat.skipWhenBusy":
260+
"When true, defer heartbeat turns on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeat turns.",
261+
"agents.list[].heartbeat.skipWhenBusy":
262+
"Per-agent override that defers heartbeat turns on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeat turns.",
259263
browser:
260264
"Browser runtime controls for local or remote CDP attachment, profile routing, and screenshot/snapshot behavior. Keep defaults unless your automation workflow requires custom browser transport settings.",
261265
"browser.enabled":
@@ -1684,6 +1688,8 @@ export const FIELD_HELP: Record<string, string> = {
16841688
'Controls whether heartbeat delivery may target direct/DM chats: "allow" (default) permits DM delivery and "block" suppresses direct-target sends.',
16851689
"agents.list.*.heartbeat.directPolicy":
16861690
'Per-agent override for heartbeat direct/DM delivery policy; use "block" for agents that should only send heartbeat alerts to non-DM destinations.',
1691+
"agents.list.*.heartbeat.skipWhenBusy":
1692+
"Per-agent override that defers heartbeat turns on extra busy lanes: subagent or nested command work. Cron lanes always defer heartbeat turns.",
16871693
"channels.mattermost.configWrites":
16881694
"Allow Mattermost to write config in response to channel events/commands (default: true).",
16891695
"channels.modelByChannel":

src/config/schema.labels.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -634,6 +634,8 @@ export const FIELD_LABELS: Record<string, string> = {
634634
"agents.list.*.heartbeat.suppressToolErrorWarnings": "Heartbeat Suppress Tool Error Warnings",
635635
"agents.defaults.heartbeat.timeoutSeconds": "Heartbeat Timeout (Seconds)",
636636
"agents.list.*.heartbeat.timeoutSeconds": "Heartbeat Timeout (Seconds)",
637+
"agents.defaults.heartbeat.skipWhenBusy": "Heartbeat Skip When Busy",
638+
"agents.list.*.heartbeat.skipWhenBusy": "Heartbeat Skip When Busy",
637639
"agents.defaults.sandbox.browser.network": "Sandbox Browser Network",
638640
"agents.defaults.sandbox.browser.cdpSourceRange": "Sandbox Browser CDP Source Port Range",
639641
"agents.defaults.sandbox.docker.dangerouslyAllowContainerNamespaceJoin":
@@ -883,6 +885,7 @@ export const FIELD_LABELS: Record<string, string> = {
883885
"agents.list[].heartbeat.suppressToolErrorWarnings":
884886
"Agent Heartbeat Suppress Tool Error Warnings",
885887
"agents.list[].heartbeat.timeoutSeconds": "Agent Heartbeat Timeout (Seconds)",
888+
"agents.list[].heartbeat.skipWhenBusy": "Agent Heartbeat Skip When Busy",
886889
"agents.list[].sandbox.browser.network": "Agent Sandbox Browser Network",
887890
"agents.list[].sandbox.browser.cdpSourceRange": "Agent Sandbox Browser CDP Source Port Range",
888891
"agents.list[].sandbox.docker.dangerouslyAllowContainerNamespaceJoin":

src/config/types.agent-defaults.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -378,6 +378,11 @@ export type AgentDefaultsConfig = {
378378
* per-heartbeat token cost by avoiding the full session transcript.
379379
*/
380380
isolatedSession?: boolean;
381+
/**
382+
* If true, defer heartbeat runs while subagent or nested command lanes are busy.
383+
* Cron lanes are always treated as busy for heartbeat deferral.
384+
*/
385+
skipWhenBusy?: boolean;
381386
/**
382387
* When enabled, deliver the model's reasoning payload for heartbeat runs (when available)
383388
* as a separate message prefixed with `Reasoning:` (same as `/reasoning on`).

0 commit comments

Comments
 (0)