fix(agent): propagate custom provider context_length to compression feasibility check by Tranquil-Flow · Pull Request #13540 · NousResearch/hermes-agent

Tranquil-Flow · 2026-04-21T13:33:30Z

What does this PR do?

When custom_providers sets a context_length for the main model (e.g. 200K), the compression feasibility check didn't propagate this to the auxiliary model's get_model_context_length() call when the aux model falls back to the main model. This caused a false-positive "compression model too small" warning and unnecessary threshold auto-lowering.

Example from the issue: custom_providers sets context_length=200000, threshold=0.65 (130K). Without the fix, the aux query returns the built-in default 128K, which is below 130K — triggering a spurious warning.

Three-part fix in run_agent.py:

Persist _config_context_length after custom_providers resolution in __init__ (was only set before the providers loop).
Propagate to aux model query when aux_model == self.model and base URLs match (the fallback scenario).
Re-derive threshold from get_model_context_length() with the correct config override, instead of reading the potentially-stale threshold_tokens from the compressor.

Related Issue

Closes #12977

Type of Change

🐛 Bug fix (non-breaking change that fixes an issue)
✨ New feature (non-breaking change that adds functionality)
🔒 Security fix
📝 Documentation update
✅ Tests (adding or improving test coverage)
♻️ Refactor (no behavior change)
🎯 New skill (bundled or hub)

Changes Made

run_agent.py:
- Persist _config_context_length after custom_providers resolution in __init__.
- Propagate _config_context_length to the aux-model get_model_context_length() call when aux model falls back to the main model and base URLs match.
- Re-derive threshold from get_model_context_length() instead of reading possibly-stale threshold_tokens from the compressor.
tests/run_agent/test_compression_feasibility.py:
- 3 new regression tests covering: custom provider propagation when aux matches main, different aux model does NOT get main's override, and end-to-end false-positive elimination.

How to Test

Run the regression suite:
```
pytest tests/run_agent/test_compression_feasibility.py -v
```
19 tests including the 3 new regressions for the exact false-positive scenario from the issue.

Platform tested: macOS 15 (Apple Silicon).

Checklist

Code

I've read the Contributing Guide
My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
I searched for existing PRs to make sure this isn't a duplicate
My PR contains only changes related to this fix/feature (no unrelated commits)
I've run pytest tests/ -q and all tests pass
I've added tests for my changes (required for bug fixes, strongly encouraged for features)
I've tested on my platform: macOS 15 (Darwin 24.6.0)

Documentation & Housekeeping

I've updated relevant documentation (README, docs/, docstrings) — or N/A
I've updated cli-config.yaml.example if I added/changed config keys — or N/A
I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

$ pytest tests/run_agent/test_compression_feasibility.py -v
19 passed (3 new regressions)

alt-glitch · 2026-04-21T23:29:37Z

Related to #8721 and #8786 (both open) which target the same compression feasibility code path.

alt-glitch · 2026-04-21T23:30:17Z

Related to #8721 and #8786 (both open) which target the same compression feasibility code path.

smallerxie · 2026-04-29T15:02:58Z

Related: PR #17460 takes a more general approach to the same problem. Instead of propagating _config_context_length through run_agent.py (which only fixes the compression path), it auto-loads custom_providers inside get_model_context_length() itself — protecting all callers (compression, vision, title, fallback, session hygiene) without manual wiring.

🔧 CI Test Fix

The CI run 26390953273 on this branch failed with 7 test failures in tests/run_agent/test_compression_feasibility.py. The PR code correctly adds a second get_model_context_length call to re-derive the main model threshold, but the tests were not updated:

Root cause: _make_agent did not set compressor.threshold_percent on the MagicMock, so agent.context_compressor.threshold_percent returned MagicMock(), and int(MagicMock()) returns 1, making threshold = max(1, 64k) = 64k — causing all warning-path tests to fail silently.

Fix (commit a1a6dc506 on talwayh1/hermes-agent:fix/custom-provider-compression-ctx):

Added compressor.threshold_percent = threshold_percent to _make_agent
Changed 3 assert_called_once_with → assert_any_call (now 2 calls: aux + main)
Used side_effect=[aux_ctx, main_ctx] in 4 tests instead of single return_value

git fetch https://github.com/talwayh1/hermes-agent.git fix/custom-provider-compression-ctx
git cherry-pick a1a6dc506

All 16 tests + 3 new PR tests now pass.

…lity check call Two CI failures on PR NousResearch#13540: 1. check-attribution: hermes-ci-bot@nousresearch.com not in AUTHOR_MAP. Add the mapping so the CI bot's commits pass the attribution gate. 2. test_init_feasibility_check_uses_aux_context_override_from_config: The test was refactored to use assert_any_call but the explicit agent._check_compression_model_feasibility() call was accidentally dropped, so the mock recorded zero calls. Restore the call before the assertion since the feasibility probe is deferred from init.

talwayh1 · 2026-05-25T10:03:20Z

CI Fixes

Two CI failures have been fixed on talwayh1/hermes-agent:fix/custom-provider-compression-ctx:

1. check-attribution ❌→✅

Added hermes-ci-bot@nousresearch.com to scripts/release.py AUTHOR_MAP.

2. test_init_feasibility_check_uses_aux_context_override_from_config ❌→✅

The deferred agent._check_compression_model_feasibility() call was accidentally dropped during the assert_called_once_with→assert_any_call refactor. Restored it so the mock records the call shape before the assertion.

Cherry-pick:

git fetch https://github.com/talwayh1/hermes-agent.git fix/custom-provider-compression-ctx
git cherry-pick 11ca03064

talwayh1 · 2026-05-25T10:09:24Z

CI Test Fix

The Tests workflow on this branch failed due to a test bug introduced by the lazy feasibility check (#28957).

Fix: PR #32001 — re-adds the explicit _check_compression_model_feasibility() call that was accidentally removed when the test was updated for the dual get_model_context_length call pattern.

Attribution failure (same branch) is handled by PR #31991 (adding hermes-ci-bot to AUTHOR_MAP).

No action needed from you — the test fix will land on main and your PR can rebase.

alt-glitch added type/bug Something isn't working comp/agent Core agent loop, run_agent.py, prompt builder labels Apr 21, 2026

This was referenced Apr 22, 2026

fix(run_agent): read context_length from providers/custom_providers for compression model #13813

Open

fix(agent): honor provider context for aux compression #14119

Open

fix(agent): propagate custom provider context to compression checks #14152

Open

This was referenced Apr 30, 2026

but the main model's compression threshold was #17690

Closed

fix(context): pass custom_providers in aux compression and fallback paths #18877

Closed

fix: use custom provider context for aux compression #20093

Closed

This was referenced May 11, 2026

fix(agent): pass custom providers to aux context checks #23563

Closed

fix(agent): forward custom_providers to context-length probe on fallback activation #25554

Closed

Tranquil-Flow force-pushed the fix/custom-provider-compression-ctx branch 2 times, most recently from 1113227 to 90a0e1a Compare May 25, 2026 08:22

Tranquil-Flow force-pushed the fix/custom-provider-compression-ctx branch from 90a0e1a to 03d4755 Compare May 25, 2026 09:09

talwayh1 mentioned this pull request May 25, 2026

fix(test): re-add explicit feasibility check call in lazy-check integration test #32001

Open

Tranquil-Flow force-pushed the fix/custom-provider-compression-ctx branch from ffde182 to d1ebe73 Compare May 25, 2026 11:06

test: cover auxiliary compression context override

a751b99

Tranquil-Flow force-pushed the fix/custom-provider-compression-ctx branch from c721b42 to a751b99 Compare May 26, 2026 09:08

banditburai mentioned this pull request Jun 1, 2026

fix(custom_providers): resolve per-model context_length for publisher/slug model ids #36852

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(agent): propagate custom provider context_length to compression feasibility check#13540

fix(agent): propagate custom provider context_length to compression feasibility check#13540
Tranquil-Flow wants to merge 1 commit into
NousResearch:mainfrom
Tranquil-Flow:fix/custom-provider-compression-ctx

Tranquil-Flow commented Apr 21, 2026 •

edited

Loading

Uh oh!

alt-glitch commented Apr 21, 2026

Uh oh!

alt-glitch commented Apr 21, 2026

Uh oh!

smallerxie commented Apr 29, 2026

Uh oh!

talwayh1 commented May 25, 2026

Uh oh!

talwayh1 commented May 25, 2026

Uh oh!

talwayh1 commented May 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Tranquil-Flow commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Related Issue

Type of Change

Changes Made

How to Test

Checklist

Code

Documentation & Housekeeping

Screenshots / Logs

Uh oh!

alt-glitch commented Apr 21, 2026

Uh oh!

alt-glitch commented Apr 21, 2026

Uh oh!

smallerxie commented Apr 29, 2026

Uh oh!

talwayh1 commented May 25, 2026

🔧 CI Test Fix

Uh oh!

talwayh1 commented May 25, 2026

CI Fixes

1. check-attribution ❌→✅

2. test_init_feasibility_check_uses_aux_context_override_from_config ❌→✅

Uh oh!

talwayh1 commented May 25, 2026

CI Test Fix

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Tranquil-Flow commented Apr 21, 2026 •

edited

Loading