Skip to content

feat(setup): add context length entry for custom endpoints#2052

Closed
eren-karakus0 wants to merge 2 commits into
NousResearch:mainfrom
eren-karakus0:fix/2007-custom-endpoint-context-length
Closed

feat(setup): add context length entry for custom endpoints#2052
eren-karakus0 wants to merge 2 commits into
NousResearch:mainfrom
eren-karakus0:fix/2007-custom-endpoint-context-length

Conversation

@eren-karakus0

@eren-karakus0 eren-karakus0 commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #2007

Custom endpoint users can now configure context length during setup wizard instead of relying on slow probe-and-stepdown at runtime.

  • Auto-detects from endpoint /models metadata when available
  • Falls back to manual prompt when endpoint doesn't report it (with note that auto-detect at runtime may be slow)
  • Saves value to context_length_cache.yaml with normalized base URL (rstrip("/")) for reliable cache hits at runtime
  • Applies consistent range (1,024–10,000,000) and type (type(x) is int) guards for both auto-detected and manual values

Test plan

  • test_custom_endpoint_auto_detects_context_length — endpoint reports context_length → saved automatically
  • test_custom_endpoint_manual_context_length_prompt — no metadata → user enters "32768" → saved
  • test_custom_endpoint_context_length_blank_skips — user leaves blank → skipped
  • test_custom_endpoint_context_length_invalid_input — user enters "abc" → warning, skipped
  • test_custom_endpoint_trailing_slash_normalized_in_cache_key/v1//v1 in cache key
  • test_custom_endpoint_auto_detect_out_of_range_skips — 99M tokens from endpoint → warning, skipped
  • All 16 targeted tests pass (test_setup + test_setup_model_provider), 0 regressions

CI note

CI failures are pre-existing on main (tests/tools/test_delegate.py_saved_tool_names NameError, 20 tests). Unrelated to this PR.

eren-karakus0 and others added 2 commits March 19, 2026 15:45
…rch#2007)

Custom endpoint users can now configure context length during setup
wizard instead of relying on slow probe-and-stepdown at runtime.
Auto-detects from endpoint metadata when available, otherwise prompts
for manual entry. Values are pre-cached in context_length_cache.yaml
with normalized base URLs for reliable cache hits.

Co-Authored-By: eren-karakus0 <erenkarakus477@gmail.com>
…nt test

The new context length prompt in setup wizard caused StopIteration
in the iterator-based prompt mock. Added empty string for the context
window prompt and mocked fetch_endpoint_model_metadata.

Co-Authored-By: eren-karakus0 <erenkarakus477@gmail.com>
@eren-karakus0 eren-karakus0 force-pushed the fix/2007-custom-endpoint-context-length branch from ea8ffbf to fe89f1b Compare March 19, 2026 12:46
@teknium1

Copy link
Copy Markdown
Contributor

Covered by PR #2158hermes model and hermes setup both now prompt for context length when configuring custom endpoints. Thanks @eren-karakus0!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Manual context length entry on initial setup for custom endpoints

2 participants