Skip to content

perf(ttft): move runtime metadata out of the cached system prompt #3353

@kshitijk4poor

Description

@kshitijk4poor

Problem

Live runtime metadata in _build_system_prompt() makes the cached system prompt vary across sessions and hurts prefix-cache reuse.

Finding

Keep runtime metadata in a per-turn note attached to the current user message instead of the cached system prompt.

Evidence

  • Stabilizes the cached system prompt across sessions
  • Aligns Hermes with the prompt-cache-friendly pattern observed in Claude Code
  • Companion live-testing report is included in the TTFT PR via findings.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/agentCore agent loop, run_agent.py, prompt buildertype/perfPerformance improvement or optimization

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions