Skip to content

Codex context-engine projection lacks exact pre-turn token accounting #80765

@100yenadmin

Description

@100yenadmin

Summary (edit to make not stale)

Follow-up to #80760 and #80761.

#80761 makes the native Codex context-engine projection budget-aware and maps the rendered-context cap through the existing openclaw.json compaction reserve fields:

  • agents.defaults.compaction.reserveTokens
  • agents.defaults.compaction.reserveTokensFloor

That fixes the main send-side bug where LCM/context-engine could assemble a large frontier, but native Codex still received only a fixed 24k-character projected context block.

There is still a smaller accounting gap: before turn/start, the Codex app-server projection currently sizes the rendered context using contextTokenBudget, reserve config, and OpenClaw's normal approximate 4 chars/token text heuristic. Exact provider/runtime token usage is only available after the turn completes.

Why this matters

The new projection cap is correctly proportional, but it is still estimate-based. That can leave a slight mismatch between:

  • what OpenClaw expects to send before the turn,
  • what Codex actually tokenizes for the rendered turn/start input,
  • what LCM/status can later report from observed post-turn usage.

This is usually small for ordinary Latin text, but it can drift for high-token-density content, CJK/multilingual content, unusual formatting, large quoted context blocks, or future Codex tokenizer/model changes.

Current behavior

The active Codex context-engine path now effectively does:

  1. context engine assembles messages for the current runtime budget
  2. OpenClaw projects those messages into a quoted Codex prompt string
  3. projection uses contextTokenBudget - reserveTokens and a 4 chars/token estimate to choose a char cap
  4. Codex receives turn/start
  5. exact-ish usage/observed prompt tokens are only known after the turn

So OpenClaw can protect against gross overflow, but cannot pre-turn prove the exact rendered prompt token count.

Expected direction

Add a native Codex pre-turn accounting path or telemetry seam so projection/status can report exact or authoritative token accounting where available.

Possible implementation shapes:

  • expose a Codex app-server/tokenizer API for the exact turn/start input token count before send
  • add a provider/model tokenizer utility for the rendered Codex prompt string and developer instructions
  • return projection stats from the Codex app-server route, including projectedPromptChars, estimated prompt tokens, reserve tokens, cap chars, and exact input tokens when available
  • pass those stats into context-engine afterTurn/status telemetry so LCM can distinguish selected frontier, projected prompt, and provider-observed usage

Acceptance criteria

  • Native Codex context-engine projection can report pre-turn exact token count when the runtime/tokenizer surface supports it, or explicitly mark it as estimated when not available.
  • /status or equivalent diagnostics can distinguish:
    • LCM/frontier tokens selected by the context engine
    • rendered Codex projection chars/tokens before send
    • provider-observed prompt/input tokens after the turn
  • Tests cover the estimate-vs-exact marker and ensure configured reserve fields remain honored.

Notes

This is intentionally separate from #80761. The PR fixes the observed fixed-cap truncation bug. This issue tracks the remaining observability/accounting gap so future debugging does not confuse LCM frontier size with Codex rendered prompt usage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.staleMarked as stale due to inactivity

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions