Skip to content

Performance regression in 2026.5.12 mirrors #70186 — jiti Babel traversal now dominates CLI startup (was: normalizeAliases) #82881

@s-salamatov

Description

@s-salamatov

Performance regression in 2026.5.12 mirrors #70186 — jiti Babel traversal now dominates CLI startup (was: normalizeAliases)

TL;DR

After upgrading to 2026.5.12, every openclaw <subcommand> invocation takes 30–40 s of wall time on a warm Node compile cache. CPU profile shows 48% self time in jiti/dist/babel.cjs (runtime TS→JS transpilation of openclaw's own bundled dist/*.js modules), 10% in V8 GC, and ~12% in lstat/stat/existsSync/open syscalls.

This looks like the same class of regression fixed in #70186 (closed, jiti normalizeAliases O(n²)), but the hot function has shifted: normalizeAliases is no longer dominant — the cost is now in Babel AST traversal (visitQueue, visit, traverseNode, traverseFast, getOrCreateCachedPaths). The earlier fix capped one specific code path; in 2026.5.12 jiti is being re-entered via a different one.

This also reproduces the symptom described in #70533 (open, "Plugin discovery loads all manifests at boot regardless of tools.allow"): each plugin.json is openat'd ~405 times per CLI invocation, even when the plugin is disabled — confirmed empirically below.

Environment

  • openclaw: 2026.5.12 (commit f066dd2)
  • Node.js: v22.22.0
  • Platform: Linux x86_64
  • Installed globally via npm i -g openclaw~/.npm-global/lib/node_modules/openclaw
  • NODE_COMPILE_CACHE is auto-enabled by openclaw.mjs (module.enableCompileCache()); cache directory holds 2 820 compiled files — confirmed populated, does not help
  • 70 plugins loaded, 24 disabled, 0 errors (openclaw plugins doctor clean)
  • 3 bundle extensions installed (context7, security-guidance, superpowers)

Reproduction

time openclaw models           # ~37 s
time openclaw models           # ~37 s again (no warm-up benefit)
time openclaw gateway status   #  ~2.2 s
time openclaw plugins update --all   # ~5 s

Wall-time vs CPU-time confirms it is CPU-bound and multithreaded:

$ time openclaw models > /dev/null
real    0m37.156s
user    0m44.970s
sys     0m4.797s

The long-running WebSocket gateway itself (127.0.0.1:18789) is fine. Only the short-lived CLI invocations are slow — every one re-pays the full bootstrap.

Relationship to existing issues

# Status Relation
#70186 closed Same class of regression (jiti dominates CLI startup), originally hot function normalizeAliases. Fix landed in 2026.4.21. In 2026.5.12 the hot function has shifted to Babel AST traversal (visitQueue/visit/traverseNode), so the original WeakMap patch no longer covers it. Practically a regression of the same user-visible symptom via a different code path.
#70533 open Independently confirmed here on 2026.5.12: a plugin.json continues to be openat'd ~405 times per invocation even when the plugin is disabled, so enable/disable is not a workaround. See "Disabling extensions does not help" below.
#59281 open Same root tooling (jiti slow for TS source loading). Windows-specific; this report is Linux but the underlying jiti pattern looks similar.
#77347 open Plugin loader LRU keyed by workspaceDir. Not applicable here (single workspace), but symptomatically related: cache layer exists but does not protect cold CLI calls.

Evidence

1. CPU profile (warm cache, 44.7 s wall, 140 242 samples)

Captured via NODE_OPTIONS="--cpu-prof --cpu-prof-dir=/tmp/oc-prof --cpu-prof-interval=200" openclaw models.

Top files by self time:

self % self (ms) File
48.20 % 21 561.8 node_modules/jiti/dist/babel.cjs
10.37 % 4 637.4 (garbage collector)
7.22 % 3 231.3 lstat (syscall)
2.74 % 1 226.4 node_modules/jiti/dist/jiti.cjs
1.87 % 835.8 stat (syscall)
1.70 % 761.8 existsSync
1.24 % 552.6 open (syscall)
1.06 % 472.0 node:internal/modules/package_json_reader
1.03 % 461.4 dist/json-files-CahFuwKs.js
0.60 % 267.4 dist/installed-plugin-index-store-DetkjvO9.js
0.53 % 235.1 dist/discovery-BEbYTYvv.js
0.42 % 189.9 dist/manifest-registry-Dgt5v-vG.js
0.26 % 117.7 dist/plugin-cache-primitives-M9JN_JCw.js ← cache layer barely runs

Top functions by self time are all Babel AST traversal:

self % self (ms) Function File
3.88 % 1 733 visitQueue jiti/babel.cjs
3.29 % 1 472 visit jiti/babel.cjs
2.95 % 1 320 getOrCreateCachedPaths jiti/babel.cjs
2.17 % 971 traverseNode jiti/babel.cjs
2.01 % 901 visitMultiple jiti/babel.cjs
1.96 % 876 traverseFast jiti/babel.cjs
1.10 % 493 visit jiti/babel.cjs
0.89 % 399 pushContext jiti/babel.cjs
0.87 % 388 popContext jiti/babel.cjs

Note: normalizeAliases (the function fixed in #70186) is no longer at the top — the fix held. But Babel AST traversal now dominates, and is happening on every CLI invocation against openclaw's own bundled modules.

Top files by inclusive time (file appears anywhere in stack):

incl % File
81.09 % dist/list.status-command-Buoxqzi2.js (the models command itself)
67.01 % dist/provider-runtime-DXB7r8u2.js
62.23 % dist/plugin-module-loader-cache-MuKAXPrS.js
61.99 % dist/plugin-load-profile-BSCTMdA8.js
61.09 % node_modules/jiti/dist/jiti.cjs
50.98 % dist/loader-DkTFEskE.js
48.99 % node_modules/jiti/lib/jiti.cjs
48.93 % node_modules/jiti/dist/babel.cjs
48.81 % dist/providers.runtime-Bj4QRzbJ.js
48.47 % dist/plugin-cache-primitives-M9JN_JCw.js
44.09 % (plugin runtime) @openclaw/codex/dist/index.js

plugin-cache-primitives-*.js is on the stack for 48% of all sample ticks but only 0.26% self time — the cache layer is traversed but never short-circuits.

2. Hot stack: jiti is transpiling openclaw's own bundled dist/* modules

One representative sample (43.6 ms self time at leaf):

ajv compileSchema           node_modules/ajv/dist/compile/index.js:32
ajv _compileSchemaEnv       node_modules/ajv/dist/core.js:468
ajv compile                 node_modules/ajv/dist/core.js:157
(anon)                      dist/protocol-QC2mQFMN.js
eval_evalModule             node_modules/jiti/dist/jiti.cjs
jitiRequire                 node_modules/jiti/dist/jiti.cjs
(anon)                      dist/client-BGs41kAq.js
eval_evalModule             node_modules/jiti/dist/jiti.cjs
jitiRequire                 node_modules/jiti/dist/jiti.cjs
(anon)                      dist/call-DO7ujqcl.js

This stack shows openclaw's already-bundled own dist files (protocol-*.js, client-*.js, call-*.js) being require()'d through jiti, which Babel-parses each one on every CLI invocation. The Node compile cache cannot help because jiti has its own translation pipeline.

3. Strace — file access patterns

$ strace -f -e trace=openat openclaw models 2>/tmp/trace.txt
$ wc -l /tmp/trace.txt
89550   /tmp/trace.txt

89 550 openat calls per single CLI invocation. Hot files (count of openat per invocation):

407x  ~/.openclaw/extensions/security-guidance/.claude-plugin/plugin.json
407x  ~/.openclaw/extensions/context7/.claude-plugin/plugin.json
405x  ~/.openclaw/extensions/superpowers/.codex-plugin/plugin.json     <-- DISABLED plugin
352x  ~/.openclaw/plugins/installs.json
345x  ~/.openclaw/extensions/superpowers/package.json
228x  ~/.openclaw/npm/package.json

3 820 unique JS modules are loaded from node_modules; on average each JS file is openat'd ~23 times per invocation.

4. Disabling extensions does not help (confirms #70533)

All 3 bundle extensions enabled All 3 disabled Δ
openclaw models 34.2 s 37.0 s / 36.7 s +2.5 s (slower)
openclaw gateway status 2.2 s 2.3 s +0.1 s
openat total 89 550 89 542 −8 (noise)
plugin.json reads per extension 407 405 −2

Disable status does not gate manifest scanning. Each plugin.json is still opened ~405 times even when the plugin is disabled. This independently confirms #70533 on 2026.5.12, and rules out plugins disable as a workaround.

5. What was tried and did not help

All measured in sequence on the same host:

  • openclaw plugins registry --refresh — improved gateway status 4.5 s → 2.2 s, but models unchanged
  • openclaw plugins update --all — no impact on models
  • openclaw doctor --fix --non-interactive --yes — no impact on subsequent CLI timing (does restart gateway)
  • Disabling all 3 bundle extensions — see table above
  • Explicit NODE_COMPILE_CACHE=... — launcher already enables it; respawns to use packaged cache dir; cache contains 2 820 files
  • Removing 50 MB session-transcript jsonl files (agents/general/sessions/*.jsonl)
  • Removing memory.bak-pre-memory-arch/, stale openclaw.json.{bak,bak.N,clobbered.*,pre-*,last-good}
  • Removing stale ~/.openclaw/gateway-restart-intent.json
  • Removing stale /tmp/openclaw-1000/openclaw-claude-skills-* (24 directories)
  • Gateway restart (manual and via doctor)

Hypothesized contributing causes

  1. jiti is being used to require() openclaw's own already-bundled dist/* modules. Those are emitted as plain JS in the bundle; runtime transpilation shouldn't be needed. Routing them through jiti+Babel re-parses ~3 800 files per CLI invocation. The previous [Bug]: 2026.4.21 CLI startup regressed 8–12× — jiti.normalizeAliases O(n²) called fresh per plugin (no cross-call cache) #70186 fix capped normalizeAliases; the cost has moved into Babel traversal.
  2. The plugin-cache-primitives cache layer is reached but does not short-circuit (48% inclusive vs 0.26% self). Either invalidated on every cold process start, or only memoizes intra-process and is never persisted.
  3. Plugin-manifest scanning is repeated by independent subsystems (Plugin discovery loads all dist/extensions/ manifests at boot regardless of tools.allow (~500 MB structural heap) #70533 root cause): a single plugin.json is opened ~400 times per invocation, regardless of enabled status.

Suggested directions

  • Audit which require() sites in dist/* are routed through jiti. For openclaw's own bundled output, prefer native require/import so the Node compile cache covers them.
  • Persist plugin discovery + manifest scan results to a single mtime-keyed cache file under ~/.openclaw/, valid across CLI invocations, so commands like openclaw models / gateway status / doctor don't re-pay the full ~37 s.
  • Memoize the manifest registry per process so independent subsystems share one read of each plugin.json.
  • Apply enabled/disabled and tools.allow filters at discovery time, before manifest read+validate (per Plugin discovery loads all dist/extensions/ manifests at boot regardless of tools.allow (~500 MB structural heap) #70533).

Profile artifact

The full 162 MB V8 .cpuprofile, 14 MB strace, and step-by-step reproduction script are available — I can attach via gist or upload on request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions