-
-
Notifications
You must be signed in to change notification settings - Fork 79.1k
Codex context-engine projection lacks exact pre-turn token accounting #80765
Copy link
Copy link
Open
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.staleMarked as stale due to inactivityMarked as stale due to inactivity
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Give feedbackNo fields configured for issues without a type.
Summary (edit to make not stale)
Follow-up to #80760 and #80761.
#80761 makes the native Codex context-engine projection budget-aware and maps the rendered-context cap through the existing
openclaw.jsoncompaction reserve fields:agents.defaults.compaction.reserveTokensagents.defaults.compaction.reserveTokensFloorThat fixes the main send-side bug where LCM/context-engine could assemble a large frontier, but native Codex still received only a fixed 24k-character projected context block.
There is still a smaller accounting gap: before
turn/start, the Codex app-server projection currently sizes the rendered context usingcontextTokenBudget, reserve config, and OpenClaw's normal approximate4 chars/tokentext heuristic. Exact provider/runtime token usage is only available after the turn completes.Why this matters
The new projection cap is correctly proportional, but it is still estimate-based. That can leave a slight mismatch between:
turn/startinput,This is usually small for ordinary Latin text, but it can drift for high-token-density content, CJK/multilingual content, unusual formatting, large quoted context blocks, or future Codex tokenizer/model changes.
Current behavior
The active Codex context-engine path now effectively does:
contextTokenBudget - reserveTokensand a4 chars/tokenestimate to choose a char capturn/startSo OpenClaw can protect against gross overflow, but cannot pre-turn prove the exact rendered prompt token count.
Expected direction
Add a native Codex pre-turn accounting path or telemetry seam so projection/status can report exact or authoritative token accounting where available.
Possible implementation shapes:
turn/startinput token count before sendprojectedPromptChars, estimated prompt tokens, reserve tokens, cap chars, and exact input tokens when availableafterTurn/status telemetry so LCM can distinguish selected frontier, projected prompt, and provider-observed usageAcceptance criteria
/statusor equivalent diagnostics can distinguish:Notes
This is intentionally separate from #80761. The PR fixes the observed fixed-cap truncation bug. This issue tracks the remaining observability/accounting gap so future debugging does not confuse LCM frontier size with Codex rendered prompt usage.