-
-
Notifications
You must be signed in to change notification settings - Fork 79.1k
Plugin discovery loads all dist/extensions/ manifests at boot regardless of tools.allow (~500 MB structural heap) #70533
Copy link
Copy link
Closed
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.ClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.ClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Give feedbackNo fields configured for issues without a type.
Summary
Plugin discovery reads every manifest under
dist/extensions/viareadFileSyncand runs each through zod validation at boot — even whentools.allowrestricts the enabled plugin set to a small subset. For an install with 104 bundled extensions and a 6-plugin allowlist, this contributes ~500 MB of resident heap that scales with bundle size, not user config.Ask: apply the
tools.allowfilter at discovery time (before reading/validating the manifest), not only at enable time.Environment
/usr/local/opt/node)npm i -g openclawdist/extensions/tools.allowrestricts to icecream-ito + a handful of providers)Evidence
Captured a V8 sampling heap profile (
node --heap-prof --heap-prof-interval=524288) against a warm, live gateway with workload. Aggregated top contributors by source file:node_modules/openclaw/[zod]node:fs readFileSyncnode_modules/openclaw/[source-map]node:vm Scriptnode_modules/openclaw/[jiti]node_modules/openclaw/[json5]node_modules/openclawcoreicecream-ito-pluginCombined
zod + readFileSync + jiti + json5 + source-map + vm= ~75% of sampled heap (~450 MB) — all module-load overhead proportional to the number of bundled extensions, not the allowlist.The user's plugin itself is 1.7%. The workload-side state (sessions, conversation history, search indices) is <5% combined and reclaimable by GC. The 500 MB floor is structural and persists across idle.
Current behavior
Discovery walks
dist/extensions/*, reads each manifest withreadFileSync, transpiles viajitiwhere needed, and validates with zod — before thetools.allowgate is consulted. With 104 bundled extensions, this happens 104 times at every boot.The allowlist only prevents instantiation/registration later, not the discovery-time work that built the validated schema graph. Zod schemas, once compiled, are retained for the process lifetime.
Expected behavior
When
tools.allow(or equivalent) is non-empty:readdir).readFileSync+ jiti + zod validate the manifests that survive the filter.For the 6-of-104 case, this would save ~94% of the discovery cost (~470 MB).
Impact
--max-old-space-sizecap below ~1.5 GB OOMs at boot because ~1 GB is discovery overhead.Reproduction
tools.allowwith a small subset (e.g., 6 plugins).ps -o rss=on the gateway PID → resident ~1.2–1.3 GB.node --heap-prof --heap-prof-interval=524288against the same config and aggregate by file-level location —readFileSync,zod,jiti,json5dominate.Heap profile (1.2 MB, loadable in Chrome DevTools → Memory → Load profile) and the aggregator script are available on request — not attaching inline due to GitHub issue size limits.
Workaround
None. The 104 extensions ship inside the openclaw npm package — users can't selectively remove them without breaking the install. Only upstream can filter at discovery.
Suggested fix sketch
In the plugin discovery path, approximately:
The
loadAndValidateManifeststep is wherereadFileSync+ zod compile happens today. Gating that behind the allowlist check is the intended change.Happy to test a fix against Ito's live config and report before/after RSS numbers.