Summary

• Problem: ContextEngine.assemble receives only tokenBudget (the full model context window) and has no signal for how many of those tokens are already consumed by messages + systemPrompt + prompt. Engines that prepend a systemPromptAddition have no way to size it against actual remaining headroom.
• Why it matters: On long sessions an engine's injection can push the prompt past the model's context limit. The runtime's preemptive-compaction (which has full visibility) runs before assemble, so it does not catch overflow caused by the engine's injection — the next LLM call simply fails with a context-limit error.
• What changed: Added an optional currentTokenCount?: number field to ContextEngine.assemble params, plumbed it through assembleHarnessContextEngine, and wired computation at every production caller: the PI runner main assemble (runEmbeddedAttempt), the PI runner tool-loop hook (installContextEngineLoopHook via the getRuntimeContext callback), and the Codex harness (extensions/codex/src/app-server/run-attempt.ts). estimatePrePromptTokens is re-exported from openclaw/plugin-sdk/agent-harness-runtime so extension harnesses use the same estimator the PI runner uses — defining the semantics once. Docs updated.
• What did NOT change (scope boundary): Behavior of engines that don't read the new field (it's optional). Semantics or value of tokenBudget. preemptive-compaction logic. No public type was renamed or removed. No new dependency was added.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes #
• Related #
• This PR fixes a bug or regression

No upstream OpenClaw issue. Motivation comes from a downstream context-engine plugin (@byterover/byterover) that hit a real overflow on long sessions and could not implement a precise guard without this field.

Real behavior proof (required for external PRs)

• Behavior or issue addressed: Context-engine plugins cannot precisely guard their systemPromptAddition against the model's remaining headroom because assemble receives only tokenBudget, not currentTokenCount. After this change, engines can compute remaining = tokenBudget - currentTokenCount - <reserve> and skip injection when the curated content would not fit.
• Real environment tested: macOS (Darwin 25.2.0), local pnpm workspace, pnpm exec vitest run on the touched test files. Not yet tested against a live model on a long session — verification so far is at the unit/integration-test layer. A follow-up real-session repro on a Gemini 1M-context channel where pre-this-PR overflow was observed is planned before merge.
• Exact steps or command run after this patch:

cd ~/dpmemories/openclawdp/openclaw
pnpm install
pnpm exec vitest run \
  src/agents/harness/context-engine-lifecycle.test.ts \
  src/agents/pi-embedded-runner/tool-result-context-guard.test.ts \
  src/agents/pi-embedded-runner/run/attempt.spawn-workspace.context-injection.test.ts \
  src/agents/pi-embedded-runner/run/attempt.spawn-workspace.context-engine.test.ts \
  extensions/codex/src/app-server/run-attempt.context-engine.test.ts

• Evidence after fix:

RUN  v4.1.6 /Users/datpham/dpmemories/openclawdp/openclaw

Test Files  9 passed (9)
     Tests  194 passed (194)
  Duration  19.96s

• Observed result after fix: New field is plumbed end-to-end across every production assemble path.
  • PI runner main assemble: runEmbeddedAttempt computes preassemblyCurrentTokenCount via estimatePrePromptTokens from { activeSession.messages, systemPromptText, params.prompt } and passes it through assembleAttemptContextEngine.
  • PI runner tool-loop hook: installContextEngineLoopHook reads currentTokenCount from the existing getRuntimeContext() callback. The production install site at attempt.ts:1996 now computes the estimate from the loop messages and systemPromptText (with empty prompt — the continuation is driven by tool results already in messages), so engines see a real value on this path too.
  • Codex harness: extensions/codex/src/app-server/run-attempt.ts:624 computes the same estimate from { historyMessages, developerInstructions, params.prompt } and passes it through assembleHarnessContextEngine.
• What was not tested: A live overflow scenario with a real LLM provider. The downstream plugin's consumption of this field (which is where the real-world payoff happens) is tracked in a separate plugin PR.
• Before evidence (optional but encouraged): N/A — this PR adds new capability rather than fixing a behavior already present in OpenClaw.

Root Cause (if applicable)

N/A — this is a capability addition, not a bug fix in OpenClaw. The existing behavior (passing only tokenBudget) is by design.

• Root cause: N/A
• Missing detection / guardrail: N/A
• Contributing context (if known): Plugin authors building context engines that inject systemPromptAddition discovered they had no way to size injection safely. The ContextEngineRuntimeContext type already defined currentTokenCount for other lifecycle hooks (afterTurn, maintain, compact) — assemble was the only one that didn't receive it.

Regression Test Plan (if applicable)

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file:
  • src/agents/harness/context-engine-lifecycle.test.ts — wrapper pass-through (present + absent)
  • src/agents/pi-embedded-runner/tool-result-context-guard.test.ts — tool-loop hook propagation (from runtimeContext, missing-from-context, no-callback)
  • src/agents/pi-embedded-runner/run/attempt.spawn-workspace.context-injection.test.ts — attempt-level alias plumbs the field
  • src/agents/pi-embedded-runner/run/attempt.spawn-workspace.context-engine.test.ts — captures loop-hook params and asserts getRuntimeContext() returns a positive currentTokenCount in production
  • extensions/codex/src/app-server/run-attempt.context-engine.test.ts — Codex harness path now asserts a positive currentTokenCount reaches the engine
• Scenario the test should lock in: currentTokenCount reaches engine.assemble from every production harness path; field is omitted (not undefined-valued) when no estimate is available, so engines can detect older runtimes.
• Why this is the smallest reliable guardrail: Each wiring layer (wrapper, PI main computation site, PI loop-hook computation site, Codex computation site, attempt-level re-export) has its own test. Refactoring any one of them without preserving pass-through will fail an existing test.
• Existing test that already covers this (if any): preemptive-compaction.test.ts already covers the underlying estimatePrePromptTokens helper that produces the value.
• If no new test is added, why not: N/A (9 new tests / assertions added across 5 test files).

User-visible / Behavior Changes

None for OpenClaw end users. Plugin authors implementing ContextEngine get a new optional currentTokenCount?: number param on assemble. Plugins that do not read it see unchanged behavior. Extension authors gain a new re-export estimatePrePromptTokens on openclaw/plugin-sdk/agent-harness-runtime for use when wiring custom harnesses.

Diagram (if applicable)

Before:
runEmbeddedAttempt          (PI harness — main + loop-hook)
  └─ contextEngine.assemble({ tokenBudget, messages, prompt, ... })

extensions/codex run-attempt (Codex harness)
  └─ contextEngine.assemble({ tokenBudget, messages, prompt, ... })

  (engines have no view of pre-assembly consumed tokens)

After:
runEmbeddedAttempt          (PI harness)
  ├─ main assemble
  │   ├─ estimatePrePromptTokens(messages + systemPrompt + prompt) → N
  │   └─ contextEngine.assemble({ tokenBudget, currentTokenCount: N, ... })
  └─ tool-loop hook (getRuntimeContext callback at install site)
      ├─ estimatePrePromptTokens(loopMessages + systemPrompt + "") → N
      └─ contextEngine.assemble({ tokenBudget, currentTokenCount: N, ... })

extensions/codex run-attempt (Codex harness)
  ├─ estimatePrePromptTokens(historyMessages + developerInstructions + prompt) → N
  └─ contextEngine.assemble({ tokenBudget, currentTokenCount: N, ... })

  (engines can compute remaining = tokenBudget - currentTokenCount - reserve)

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No
• Command/tool execution surface changed? No
• Data access scope changed? No
• If any Yes, explain risk + mitigation: N/A.

Repro + Verification

Environment

• OS: macOS Darwin 25.2.0
• Runtime/container: pnpm 11.x workspace, Node 22, vitest 4.1.6
• Model/provider: N/A (tests use mocked ContextEngine)
• Integration/channel (if any): N/A
• Relevant config (redacted): default tsconfig.json + workspace pnpm-workspace.yaml

Steps

1. git checkout feat/context-engine-current-token-count
2. pnpm install
3. pnpm exec vitest run src/agents/harness/context-engine-lifecycle.test.ts src/agents/pi-embedded-runner/tool-result-context-guard.test.ts src/agents/pi-embedded-runner/run/attempt.spawn-workspace.context-injection.test.ts src/agents/pi-embedded-runner/run/attempt.spawn-workspace.context-engine.test.ts extensions/codex/src/app-server/run-attempt.context-engine.test.ts

Expected

• All 194 tests across the 9 affected test files pass.
• git diff --stat origin/main..HEAD shows only the touched source, test, and docs files.

Actual

• 194/194 tests pass (see Evidence above).

Evidence

• Failing test/log before + passing after — not applicable; this is a feature addition, no prior failing test existed.
• Trace/log snippets — vitest summary above.
• Screenshot/recording
• Perf numbers (if relevant) — estimatePrePromptTokens is now invoked at each main assemble call site (PI main, Codex) and once per tool-loop iteration (PI loop hook). The helper already runs at the preemptive-compaction precheck on the same turn (PI main path), so the marginal cost there is one extra encode of messages + systemPrompt + prompt. Loop-hook iterations add one encode per iteration. Not measured separately.

Human Verification (required)

• Verified scenarios:
  • assembleHarnessContextEngine passes currentTokenCount to the engine when supplied.
  • assembleHarnessContextEngine omits the field when caller does not supply it (so engines can detect older runtimes via params.currentTokenCount === undefined).
  • installContextEngineLoopHook reads currentTokenCount from getRuntimeContext() when the callback returns it; omits otherwise; omits when no callback is wired.
  • The production loop-hook install site at attempt.ts:1996 computes currentTokenCount and feeds it into the getRuntimeContext runtime context (asserted by capturing the callback in attempt.spawn-workspace.context-engine.test.ts).
  • The attempt-level assembleAttemptContextEngine re-export still plumbs the field.
  • The Codex harness call at extensions/codex/src/app-server/run-attempt.ts:624 computes a positive currentTokenCount and passes it through (asserted in run-attempt.context-engine.test.ts).
• Edge cases checked:
  • Missing getRuntimeContext callback in the loop hook → field omitted.
  • getRuntimeContext returns an object without currentTokenCount → field omitted.
  • Older callers that don't supply currentTokenCount → unchanged behavior.
• What you did NOT verify:
  • End-to-end run against a real LLM provider where pre-this-PR overflow would have occurred. This is the most important real-world signal and will be performed before merge.
  • Performance impact of the additional estimatePrePromptTokens calls under stress (large transcripts, fast tool loops).

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? Yes
• Config/env changes? No
• Migration needed? No
• If yes, exact upgrade steps: N/A. The new field is optional. Existing context engines compile and run unchanged. Engines that want to opt in simply read params.currentTokenCount and gate behavior on whether it is present.

Risks and Mitigations

• Risk: The additional estimatePrePromptTokens call at each main assemble call site adds one extra token-estimation pass per turn, plus one per tool-loop iteration on the PI loop-hook path.
  • Mitigation: Uses the same helper the runtime already invokes at the preemptive-compaction precheck on the same turn (PI main path), so the marginal cost there is small and amortized. Loop-hook iterations add one encode per iteration; if this proves measurably costly under sustained tool loops, the result can be cached or computed lazily.
• Risk: Engines that misread the new field (e.g. treat absent as zero) could over-restrict their injection on older runtimes that don't pass the field.
  • Mitigation: Field is explicitly omitted rather than set to undefined or 0 when not supplied (see the ...(typeof x === "number" ? { x } : {}) spread idiom in the wrappers). Engines that follow the documented contract (params.currentTokenCount === undefined ⇒ no headroom info available) behave correctly.
• Risk: Re-exporting estimatePrePromptTokens from openclaw/plugin-sdk/agent-harness-runtime widens the public extension API by one function and commits us to maintaining its signature.
  • Mitigation: The function is small, pure, and stable — it already exists at its source location and is used internally. Re-exporting just makes the same shape available to extension harnesses. If the internal signature ever needs to change, the barrel re-export can be wrapped at that point.