perf(prompt-cache): date-only timestamp + loud gateway-DB roundtrip logging#27675
Merged
Conversation
…ogging
The system prompt's 'Conversation started:' line carried minute precision
(%I:%M %p), making it byte-unstable across every rebuild path. Within a
CLI session the in-memory cache held, but on the gateway path (fresh
AIAgent per turn → restore from session DB), any silent failure in the
read or write path dropped the cache stem and forced a full re-prefill
on every subsequent turn. Local prefix-caching backends (llama.cpp /
vLLM) saw this as KV-cache invalidation; remote prefix-caching providers
saw it as an Anthropic-style cache miss.
Three changes:
1. Date-only timestamp ('Sunday, May 17, 2026' instead of '... 03:42 PM').
System prompt now byte-stable for the full day. The model can still
query exact time via tools when it actually needs it. Credit:
@iamfoz (PR #20451).
2. Loud logging on session DB write failures. The update_system_prompt
call used to log at DEBUG, hiding disk-full / locked-database / schema
drift behind a silent fall-through that forced fresh rebuilds on
every subsequent turn. Now WARN with the session id and exception so
persistent issues show up in agent.log without verbose mode.
3. Three-way stored-state distinction on read. The previous
'session_row.get("system_prompt") or None' collapsed three states
into one (missing row / null column / empty string). Now we tell them
apart and WARN when a continuing session lands on null/empty (which
means the previous turn's write never persisted — every subsequent
turn rebuilds and the prefix cache misses every time).
The restore block is extracted into _restore_or_build_system_prompt()
so the prefix-cache path can be unit-tested in isolation.
E2E proof: fresh AIAgent constructed for turn 2 across a minute-boundary
sleep restores byte-identical bytes from the session DB. NULL stored
prompt fires the new warning. Date-only timestamp survives the rebuild
path. All on real SessionDB, no mocks.
Tests:
- tests/agent/test_system_prompt_restore.py (10 new tests)
- tests/run_agent/test_run_agent.py::TestBuildSystemPrompt::
test_datetime_is_date_only_not_minute_precision
Closes #20451 (date-only), #18547 (prefix stabilization),
#8689 (stabilize timestamp across compression), #15866 (timestamp
caching question), #8687 (compression timestamp), #27339
(claim #3: live timestamp in cached system prompt).
Co-authored-by: Martyn Forryan <9133432+iamfoz@users.noreply.github.com>
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
unresolved-import |
1 |
First entries
tests/agent/test_system_prompt_restore.py:21: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
✅ Fixed issues: none
Unchanged: 4608 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
Collaborator
This was referenced May 18, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
System prompt is now byte-stable for the full day, and silent gateway-side prefix-cache misses now log loudly instead of disappearing into DEBUG.
The
Conversation started:line was minute-precision (%I:%M %p) — byte-unstable across every rebuild path. Within a CLI session the in-memory cache held, but on the gateway path (freshAIAgentper turn, prompt restored from session DB), any silent failure in the read or write path forced a full re-prefill on every subsequent turn. Local prefix-caching backends (llama.cpp, vLLM) saw it as KV-cache invalidation; remote prefix-caching providers saw it as an Anthropic-style cache miss.Changes
Sunday, May 17, 2026instead ofSunday, May 17, 2026 03:42 PM. System prompt byte-stable for the full day. Credit @iamfoz (#20451).update_system_promptfailure waslogger.debug. Nowlogger.warningwith the session id and exception.session_row.get('system_prompt') or Nonecollapsed missing / NULL / empty into one path. Now distinguished and warned on null/empty when a continuing session hits them._restore_or_build_system_prompt()so the prefix-cache path is testable in isolation.Validation
E2E proof (live SessionDB, no mocks): fresh
AIAgentconstructed for turn 2 across a 65-second minute-boundary sleep restored byte-identical bytes from the DB. NULL stored prompt fires the new warning. Date-only timestamp survives the rebuild path.Tests:
tests/agent/test_system_prompt_restore.py— 10 new tests covering happy path, fresh build, all silent-failure recovery paths, and byte-stability invarianttests/run_agent/test_run_agent.py::TestBuildSystemPrompt::test_datetime_is_date_only_not_minute_precision— pins date-only invariantTestSystemPromptStability/TestBuildSystemPrompt/TestInvalidateSystemPrompt— all pass (24/24)Closes
_build_system_promptinvalidate prompt caching for upstream inference servers? #15866 — timestamp invalidates upstream prefix cachingNotes
The Model and Provider lines are kept in the prompt as-is. They're stable within a session (no
/modelswitch) and provide self-identification context. Date-only timestamp closes the only minute-by-minute volatility source.Within-session invariant audit: every call site of
_build_system_promptand every assignment to_cached_system_promptwas inventoried. No path mutates the prompt during a healthy session. The cache holds; this PR makes the prompt itself byte-stable across the cache invalidation boundaries (compression event, fresh-agent gateway turn, session resume).