feat(setup): add context length entry for custom endpoints#2052
Closed
eren-karakus0 wants to merge 2 commits into
Closed
feat(setup): add context length entry for custom endpoints#2052eren-karakus0 wants to merge 2 commits into
eren-karakus0 wants to merge 2 commits into
Conversation
…rch#2007) Custom endpoint users can now configure context length during setup wizard instead of relying on slow probe-and-stepdown at runtime. Auto-detects from endpoint metadata when available, otherwise prompts for manual entry. Values are pre-cached in context_length_cache.yaml with normalized base URLs for reliable cache hits. Co-Authored-By: eren-karakus0 <erenkarakus477@gmail.com>
…nt test The new context length prompt in setup wizard caused StopIteration in the iterator-based prompt mock. Added empty string for the context window prompt and mocked fetch_endpoint_model_metadata. Co-Authored-By: eren-karakus0 <erenkarakus477@gmail.com>
ea8ffbf to
fe89f1b
Compare
Contributor
|
Covered by PR #2158 — |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #2007
Custom endpoint users can now configure context length during setup wizard instead of relying on slow probe-and-stepdown at runtime.
/modelsmetadata when availablecontext_length_cache.yamlwith normalized base URL (rstrip("/")) for reliable cache hits at runtimetype(x) is int) guards for both auto-detected and manual valuesTest plan
test_custom_endpoint_auto_detects_context_length— endpoint reports context_length → saved automaticallytest_custom_endpoint_manual_context_length_prompt— no metadata → user enters "32768" → savedtest_custom_endpoint_context_length_blank_skips— user leaves blank → skippedtest_custom_endpoint_context_length_invalid_input— user enters "abc" → warning, skippedtest_custom_endpoint_trailing_slash_normalized_in_cache_key—/v1/→/v1in cache keytest_custom_endpoint_auto_detect_out_of_range_skips— 99M tokens from endpoint → warning, skippedCI note
CI failures are pre-existing on
main(tests/tools/test_delegate.py—_saved_tool_namesNameError, 20 tests). Unrelated to this PR.