fix: auto-invalidate stale context length cache when defaults change#1852
Closed
Tranquil-Flow wants to merge 1 commit into
Closed
fix: auto-invalidate stale context length cache when defaults change#1852Tranquil-Flow wants to merge 1 commit into
Tranquil-Flow wants to merge 1 commit into
Conversation
The persistent context length cache (~/.hermes/context_length_cache.yaml) stores discovered model context limits across sessions. Previously, cached values lived forever — if a new hermes-agent version changed the default context length for a model (e.g., upgrading Claude from 200K to 1M for all users), existing cache entries would silently override the new default, leaving users stuck on the old limit. This adds a defaults_hash field to the cache file: a truncated SHA-256 of DEFAULT_CONTEXT_LENGTHS, recomputed on every read. When the hash doesn't match (new hermes-agent version with updated defaults), all cached entries are discarded and models re-probe their actual limits. Legacy cache files without the hash field are treated as stale and invalidated on first read — a one-time migration with no user action needed. Tradeoff: updating ANY model's default invalidates ALL cached entries, including correctly-probed local models. This is acceptable because re-probing is automatic and transparent (one API call per model).
de10701 to
d81a53a
Compare
3 tasks
Contributor
|
Merged via PR #3802. Your cache invalidation implementation was cherry-picked onto current main with authorship preserved. Clean work — thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
~/.hermes/context_length_cache.yaml)DEFAULT_CONTEXT_LENGTHS, stale cached values are automatically discardedProblem
The context length cache stores discovered model limits across sessions with no expiration or versioning. If a hermes-agent update changes a model's default context length (e.g., a provider upgrades from 128K to 256K, or a new model is added with different defaults), existing cache entries silently override the new default. Users would be stuck on the old limit until they manually delete
~/.hermes/context_length_cache.yaml.This affects all models and providers that go through the probing system — not just Anthropic models.
Solution
A
defaults_hashfield (truncated SHA-256 ofDEFAULT_CONTEXT_LENGTHS) is stored alongside cached entries. On every cache read, the stored hash is compared against the current hash:The hash is recomputed from the sorted dict, so it's deterministic and changes only when actual model defaults change.
Files changed
agent/model_metadata.py_compute_defaults_hash(), updated_load_context_cache()andsave_context_length()tests/agent/test_model_metadata.pyTradeoff
Changing ANY model's default invalidates ALL cached entries, including correctly-probed local models (e.g., Ollama at 32K). This is acceptable because re-probing is automatic, transparent (one API call per model), and only happens once per hermes-agent update that modifies defaults.
Related
Test plan
defaults_hash) is invalidated on first session