Skip to content

Commit 2a15a3b

Browse files
wirjovincentkoc
wirjo
and
vincentkoc
authored
fix(amazon-bedrock): add known model context windows to discovery (#65952)
* fix(amazon-bedrock): add known model context windows to discovery Bedrock's ListFoundationModels API does not expose token limits. Discovery was hardcoding contextWindow: 32000 for every model, causing Claude (1M), Nova (300K), and other models to hit premature 'Context limit exceeded' errors and unnecessary session resets. Adds a lookup table of known context windows for Bedrock models: - Anthropic Claude: 200K-1M - Amazon Nova: 128K-1M - Meta Llama: 128K - Mistral: 32K-128K - DeepSeek: 128K - Cohere: 128K - AI21 Jamba: 256K Inference profile prefixes (us., eu., ap., global.) are stripped before lookup, so us.anthropic.claude-opus-4-6-v1 correctly resolves to 1M. Also raises the default fallback from 32K to 128K for unknown models — most modern models have at least 128K context. Single file change, no type system modifications. Complementary to #65030 (provenance flag for warning on unknown models). Fixes #64919 Related: #64250 * add KNOWN_MAX_TOKENS map and expand model coverage - Add KNOWN_MAX_TOKENS lookup table with Bedrock-optimized values that balance response quality against quota burndown (5x rate for Claude 3.7+) - Add missing models to KNOWN_CONTEXT_WINDOWS: Opus 4.7 (1M), Opus 4.1/4.5, Sonnet 4, Claude 3/3.5 Haiku, DeepSeek V3/V3.2, Google Gemma 3 - Refactor prefix-stripping into shared resolveKnownValue() helper - Fix: use !== undefined instead of truthy check for table lookups - Wire resolveKnownMaxTokens into toModelDefinition and resolveInferenceProfiles Quota burndown context: Bedrock reserves input_tokens + max_tokens from TPM at request start. For Claude 3.7+, output burns at 5x. The values in KNOWN_MAX_TOKENS are intentionally conservative (8-16K for Claude) to maximize concurrent throughput while still allowing useful responses. Thinking budget is added separately by the runtime. * remove KNOWN_MAX_TOKENS — maxTokens should be handled upstream Remove the KNOWN_MAX_TOKENS map. Hardcoding maxTokens values in discovery is the wrong layer to solve this — any explicit value still gets reserved against Bedrock's TPM quota at request start. The correct fix is upstream in pi's Bedrock provider: omit maxTokens from inferenceConfig when not explicitly set, letting the model use its internal default. This avoids quota waste entirely. See: earendil-works/pi#3399 and earendil-works/pi#3400 Keep the expanded KNOWN_CONTEXT_WINDOWS (context windows ARE the right thing to set in discovery — they affect compaction thresholds and session management, not API-level quota reservation). * docs: clarify why hardcoded context windows are needed Bedrock's ListFoundationModels and GetFoundationModel APIs return no token limit information — there is no Bedrock API to discover context windows or max output tokens programmatically. Note that this table should become a fallback if AWS adds token metadata in the future. * fix: add au and apac to inference profile prefix regex Add missing geo prefixes discovered by querying inference profiles across multiple regions: - au. (Australia/NZ, used in ap-southeast-2/4/6) - apac. (Asia-Pacific, used for older models in ap-northeast-1) Both resolveKnownContextWindow and resolveBaseModelId now handle all known prefixes: us, eu, ap, apac, au, jp, global. * test: port au. prefix test from #65449 by @alickgithub2, add apac. coverage Port the Australia/NZ inference profile test from PR #65449 (credit: @alickgithub2) and extend it to also cover the apac. prefix discovered in ap-northeast-1. * expand model coverage: Llama 4, MiniMax, NVIDIA, Mistral 3, GLM, Qwen Cross-referenced KNOWN_CONTEXT_WINDOWS against live list-foundation-models API. Added missing models: - Llama 4 Maverick (1M) and Scout (512K) - MiniMax M2/M2.1/M2.5 (1M) - NVIDIA Nemotron Super/Nano variants (128K) - Mistral Large 3 675B (128K) - GLM 4.7/4.7-flash/5 (128K) - Qwen3 Coder/32B/VL (128-256K) Removed deprecated deepseek.v3-v1:0 and claude-opus-4-20250514 (not in active foundation models list). * raise default context window from 128K to 200K 200K matches the floor for all current Claude models (the most popular on Bedrock). Every other active model with a lower actual limit is already in the explicit table. This ensures new Claude models get a correct default without requiring a table update. * test: update discovery test expectations for known context window values * test: fix remaining contextWindow expectation (default 200K) * fix(amazon-bedrock): keep conservative context fallback * docs(changelog): note Bedrock context window fix * fix(amazon-bedrock): normalize known context fallback --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
1 parent 420c96e commit 2a15a3b

3 files changed

Lines changed: 281 additions & 8 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ Docs: https://docs.openclaw.ai
2323

2424
### Fixes
2525

26+
- Providers/Amazon Bedrock: use known context-window metadata for discovered models while keeping the unknown-model fallback conservative, so compaction and overflow handling improve for newer Bedrock models without overstating unlisted model limits. Thanks @wirjo.
2627
- Providers/Amazon Bedrock Mantle: refresh IAM-backed bearer tokens at runtime instead of baking discovery-time tokens into provider config, so long-lived Mantle sessions keep working after the initial token ages out. Thanks @wirjo.
2728
- Codex harness: rotate the shared app-server websocket client when the configured bearer token changes, so auth-token refreshes reconnect with the new `Authorization` header instead of reusing a stale socket. (#70328) Thanks @Lucenx9.
2829
- Telegram/sandbox: keep Telegram bot DMs on per-account sender session keys even when `session.dmScope=main`, so sandbox/tool policy can distinguish Telegram-originated direct chats from the agent main session.

‎extensions/amazon-bedrock/discovery.test.ts‎

Lines changed: 161 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ describe("bedrock discovery", () => {
8787
name: "Claude 3.7 Sonnet",
8888
reasoning: false,
8989
input: ["text", "image"],
90-
contextWindow: 32000,
90+
contextWindow: 200000,
9191
maxTokens: 4096,
9292
});
9393
});
@@ -104,7 +104,11 @@ describe("bedrock discovery", () => {
104104
});
105105

106106
it("uses configured defaults for context and max tokens", async () => {
107-
mockSingleActiveSummary();
107+
mockSingleActiveSummary({
108+
modelId: "example.unknown-text-v1:0",
109+
modelName: "Example Unknown Text",
110+
providerName: "example",
111+
});
108112

109113
const models = await discoverBedrockModels({
110114
region: "us-east-1",
@@ -114,6 +118,68 @@ describe("bedrock discovery", () => {
114118
expect(models[0]).toMatchObject({ contextWindow: 64000, maxTokens: 8192 });
115119
});
116120

121+
it("keeps the conservative fallback for unknown inference profiles", async () => {
122+
sendMock
123+
.mockResolvedValueOnce({
124+
modelSummaries: [],
125+
})
126+
.mockResolvedValueOnce({
127+
inferenceProfileSummaries: [
128+
{
129+
inferenceProfileId: "jp.example.unknown-text-v1:0",
130+
inferenceProfileName: "JP Example Unknown Text",
131+
status: "ACTIVE",
132+
type: "SYSTEM_DEFINED",
133+
models: [
134+
{
135+
modelArn: "arn:aws:bedrock:ap-northeast-1::foundation-model/example.unknown-text-v1:0",
136+
},
137+
],
138+
},
139+
],
140+
});
141+
142+
const models = await discoverBedrockModels({ region: "ap-northeast-1", clientFactory });
143+
144+
expect(models).toHaveLength(1);
145+
expect(models[0]).toMatchObject({
146+
id: "jp.example.unknown-text-v1:0",
147+
contextWindow: 32000,
148+
maxTokens: 4096,
149+
input: ["text"],
150+
});
151+
});
152+
153+
it("normalizes region-prefixed versioned model ids when resolving context windows", async () => {
154+
sendMock
155+
.mockResolvedValueOnce({
156+
modelSummaries: [],
157+
})
158+
.mockResolvedValueOnce({
159+
inferenceProfileSummaries: [
160+
{
161+
inferenceProfileId: "jp.anthropic.claude-sonnet-4-6-v1:0",
162+
inferenceProfileName: "JP Claude Sonnet 4.6",
163+
status: "ACTIVE",
164+
type: "SYSTEM_DEFINED",
165+
models: [
166+
{
167+
modelArn:
168+
"arn:aws:bedrock:ap-northeast-1::foundation-model/anthropic.claude-sonnet-4-6-v1:0",
169+
},
170+
],
171+
},
172+
],
173+
});
174+
175+
const models = await discoverBedrockModels({ region: "ap-northeast-1", clientFactory });
176+
177+
expect(models[0]).toMatchObject({
178+
id: "jp.anthropic.claude-sonnet-4-6-v1:0",
179+
contextWindow: 1_000_000,
180+
});
181+
});
182+
117183
it("caches results when refreshInterval is enabled", async () => {
118184
mockSingleActiveSummary();
119185

@@ -252,7 +318,7 @@ describe("bedrock discovery", () => {
252318
expect(usProfile).toMatchObject({
253319
name: "US Anthropic Claude Sonnet 4.6",
254320
input: ["text", "image"],
255-
contextWindow: 32000,
321+
contextWindow: 1000000,
256322
maxTokens: 4096,
257323
});
258324
expect(euProfile).toMatchObject({ input: ["text", "image"] });
@@ -356,11 +422,43 @@ describe("bedrock discovery", () => {
356422
expect(profile).toMatchObject({
357423
id: "us.my-prod-profile",
358424
input: ["text", "image"],
359-
contextWindow: 32000,
425+
contextWindow: 1000000,
360426
maxTokens: 4096,
361427
});
362428
});
363429

430+
it("uses the resolved base model id for application-profile context fallback", async () => {
431+
sendMock
432+
.mockResolvedValueOnce({
433+
modelSummaries: [],
434+
})
435+
.mockResolvedValueOnce({
436+
inferenceProfileSummaries: [
437+
{
438+
inferenceProfileId: "us.my-prod-profile",
439+
inferenceProfileName: "Prod Claude Profile",
440+
status: "ACTIVE",
441+
type: "APPLICATION",
442+
models: [
443+
{
444+
modelArn:
445+
"arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-opus-4-6-v1:0",
446+
},
447+
],
448+
},
449+
],
450+
});
451+
452+
const models = await discoverBedrockModels({ region: "us-east-1", clientFactory });
453+
454+
expect(models[0]).toMatchObject({
455+
id: "us.my-prod-profile",
456+
contextWindow: 1_000_000,
457+
maxTokens: 4096,
458+
input: ["text"],
459+
});
460+
});
461+
364462
it("merges implicit Bedrock models into explicit provider overrides", () => {
365463
expect(
366464
mergeImplicitBedrockProvider({
@@ -433,4 +531,63 @@ describe("bedrock discovery", () => {
433531
expect(legacyEnabled?.baseUrl).toBe("https://bedrock-runtime.us-west-2.amazonaws.com");
434532
expect(sendMock).toHaveBeenCalledTimes(4);
435533
});
534+
535+
// Ported from #65449 by @alickgithub2 — extended to also cover apac. prefix
536+
it("resolves au. and apac. prefixes for regional inference profiles", async () => {
537+
sendMock
538+
.mockResolvedValueOnce({
539+
modelSummaries: [
540+
{
541+
modelId: "anthropic.claude-sonnet-4-6",
542+
modelName: "Claude Sonnet 4.6",
543+
providerName: "anthropic",
544+
inputModalities: ["TEXT", "IMAGE"],
545+
outputModalities: ["TEXT"],
546+
responseStreamingSupported: true,
547+
modelLifecycle: { status: "ACTIVE" },
548+
},
549+
],
550+
})
551+
.mockResolvedValueOnce({
552+
inferenceProfileSummaries: [
553+
{
554+
inferenceProfileId: "au.anthropic.claude-sonnet-4-6",
555+
inferenceProfileName: "AU Anthropic Claude Sonnet 4.6",
556+
inferenceProfileArn:
557+
"arn:aws:bedrock:ap-southeast-2::inference-profile/au.anthropic.claude-sonnet-4-6",
558+
status: "ACTIVE",
559+
type: "SYSTEM_DEFINED",
560+
models: [], // no ARNs — forces the prefix-regex fallback
561+
},
562+
{
563+
inferenceProfileId: "apac.anthropic.claude-sonnet-4-6",
564+
inferenceProfileName: "APAC Anthropic Claude Sonnet 4.6",
565+
inferenceProfileArn:
566+
"arn:aws:bedrock:ap-northeast-1::inference-profile/apac.anthropic.claude-sonnet-4-6",
567+
status: "ACTIVE",
568+
type: "SYSTEM_DEFINED",
569+
models: [],
570+
},
571+
],
572+
});
573+
574+
const models = await discoverBedrockModels({ region: "ap-southeast-2", clientFactory });
575+
576+
// Foundation model + 2 regional inference profiles
577+
expect(models).toHaveLength(3);
578+
579+
const auProfile = models.find((m) => m.id === "au.anthropic.claude-sonnet-4-6");
580+
expect(auProfile).toMatchObject({
581+
id: "au.anthropic.claude-sonnet-4-6",
582+
name: "AU Anthropic Claude Sonnet 4.6",
583+
input: ["text", "image"],
584+
});
585+
586+
const apacProfile = models.find((m) => m.id === "apac.anthropic.claude-sonnet-4-6");
587+
expect(apacProfile).toMatchObject({
588+
id: "apac.anthropic.claude-sonnet-4-6",
589+
name: "APAC Anthropic Claude Sonnet 4.6",
590+
input: ["text", "image"],
591+
});
592+
});
436593
});

‎extensions/amazon-bedrock/discovery.ts‎

Lines changed: 119 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,121 @@ import {
2121
const log = createSubsystemLogger("bedrock-discovery");
2222

2323
const DEFAULT_REFRESH_INTERVAL_SECONDS = 3600;
24-
const DEFAULT_CONTEXT_WINDOW = 32000;
24+
const DEFAULT_CONTEXT_WINDOW = 32_000;
2525
const DEFAULT_MAX_TOKENS = 4096;
26+
27+
// ---------------------------------------------------------------------------
28+
// Known model context windows (Bedrock API does not expose token limits)
29+
// ---------------------------------------------------------------------------
30+
31+
/**
32+
* Bedrock's ListFoundationModels and GetFoundationModel APIs return no token
33+
* limit information — only model ID, name, modalities, and lifecycle status.
34+
* There is currently no Bedrock API to discover context windows or max output
35+
* tokens programmatically.
36+
*
37+
* This map provides correct context window values for known models so that
38+
* session management, compaction thresholds, and context overflow detection
39+
* work correctly. If AWS adds token metadata to the API in the future, this
40+
* table should become a fallback rather than the primary source.
41+
*
42+
* Inference profile prefixes (us., eu., ap., global.) are stripped before lookup.
43+
*
44+
* Sources: https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html
45+
* https://platform.claude.com/docs/en/about-claude/models
46+
*/
47+
const KNOWN_CONTEXT_WINDOWS: Record<string, number> = {
48+
// Anthropic Claude
49+
"anthropic.claude-3-7-sonnet-20250219-v1:0": 200_000,
50+
"anthropic.claude-opus-4-7": 1_000_000,
51+
"anthropic.claude-opus-4-6-v1": 1_000_000,
52+
"anthropic.claude-opus-4-6-v1:0": 1_000_000,
53+
"anthropic.claude-sonnet-4-6": 1_000_000,
54+
"anthropic.claude-sonnet-4-6-v1:0": 1_000_000,
55+
"anthropic.claude-sonnet-4-5-20250929-v1:0": 200_000,
56+
"anthropic.claude-sonnet-4-20250514-v1:0": 200_000,
57+
"anthropic.claude-opus-4-5-20251101-v1:0": 200_000,
58+
"anthropic.claude-opus-4-1-20250805-v1:0": 200_000,
59+
"anthropic.claude-haiku-4-5-20251001-v1:0": 200_000,
60+
"anthropic.claude-3-5-haiku-20241022-v1:0": 200_000,
61+
"anthropic.claude-3-haiku-20240307-v1:0": 200_000,
62+
// Amazon Nova
63+
"amazon.nova-premier-v1:0": 1_000_000,
64+
"amazon.nova-pro-v1:0": 300_000,
65+
"amazon.nova-lite-v1:0": 300_000,
66+
"amazon.nova-micro-v1:0": 128_000,
67+
"amazon.nova-2-lite-v1:0": 300_000,
68+
// MiniMax
69+
"minimax.minimax-m2.5": 1_000_000,
70+
"minimax.minimax-m2.1": 1_000_000,
71+
"minimax.minimax-m2": 1_000_000,
72+
// Meta Llama 4
73+
"meta.llama4-maverick-17b-instruct-v1:0": 1_000_000,
74+
"meta.llama4-scout-17b-instruct-v1:0": 512_000,
75+
// Meta Llama 3
76+
"meta.llama3-3-70b-instruct-v1:0": 128_000,
77+
"meta.llama3-2-90b-instruct-v1:0": 128_000,
78+
"meta.llama3-2-11b-instruct-v1:0": 128_000,
79+
"meta.llama3-2-3b-instruct-v1:0": 128_000,
80+
"meta.llama3-2-1b-instruct-v1:0": 128_000,
81+
"meta.llama3-1-405b-instruct-v1:0": 128_000,
82+
"meta.llama3-1-70b-instruct-v1:0": 128_000,
83+
"meta.llama3-1-8b-instruct-v1:0": 128_000,
84+
// NVIDIA Nemotron
85+
"nvidia.nemotron-super-3-120b": 256_000,
86+
"nvidia.nemotron-nano-3-30b": 128_000,
87+
"nvidia.nemotron-nano-12b-v2": 128_000,
88+
"nvidia.nemotron-nano-9b-v2": 128_000,
89+
// Mistral
90+
"mistral.mistral-large-3-675b-instruct": 128_000,
91+
"mistral.mistral-large-2407-v1:0": 128_000,
92+
"mistral.mistral-small-2402-v1:0": 32_000,
93+
// DeepSeek
94+
"deepseek.r1-v1:0": 128_000,
95+
"deepseek.v3.2": 128_000,
96+
// Cohere
97+
"cohere.command-r-plus-v1:0": 128_000,
98+
"cohere.command-r-v1:0": 128_000,
99+
// AI21
100+
"ai21.jamba-1-5-large-v1:0": 256_000,
101+
"ai21.jamba-1-5-mini-v1:0": 256_000,
102+
// Google Gemma
103+
"google.gemma-3-27b-it": 128_000,
104+
"google.gemma-3-12b-it": 128_000,
105+
"google.gemma-3-4b-it": 128_000,
106+
// GLM
107+
"zai.glm-5": 128_000,
108+
"zai.glm-4.7": 128_000,
109+
"zai.glm-4.7-flash": 128_000,
110+
// Qwen
111+
"qwen.qwen3-coder-next": 256_000,
112+
"qwen.qwen3-coder-30b-a3b-v1:0": 256_000,
113+
"qwen.qwen3-32b-v1:0": 128_000,
114+
"qwen.qwen3-vl-235b-a22b": 128_000,
115+
};
116+
117+
/**
118+
* Resolve the real context window for a Bedrock model ID.
119+
* Strips inference profile prefixes (us., eu., ap., global.) before lookup.
120+
*/
121+
function resolveKnownContextWindow(modelId: string): number | undefined {
122+
const stripped = modelId.replace(/^(?:us|eu|ap|apac|au|jp|global)\./, "");
123+
const candidates = [modelId, stripped];
124+
for (const candidate of candidates) {
125+
if (KNOWN_CONTEXT_WINDOWS[candidate] !== undefined) {
126+
return KNOWN_CONTEXT_WINDOWS[candidate];
127+
}
128+
const withoutVersionSuffix = candidate.replace(/:0$/, "");
129+
if (
130+
withoutVersionSuffix !== candidate &&
131+
KNOWN_CONTEXT_WINDOWS[withoutVersionSuffix] !== undefined
132+
) {
133+
return KNOWN_CONTEXT_WINDOWS[withoutVersionSuffix];
134+
}
135+
}
136+
return undefined;
137+
}
138+
26139
const DEFAULT_COST = {
27140
input: 0,
28141
output: 0,
@@ -163,7 +276,7 @@ function toModelDefinition(
163276
reasoning: inferReasoningSupport(summary),
164277
input: mapInputModalities(summary),
165278
cost: DEFAULT_COST,
166-
contextWindow: defaults.contextWindow,
279+
contextWindow: resolveKnownContextWindow(id) ?? defaults.contextWindow,
167280
maxTokens: defaults.maxTokens,
168281
};
169282
}
@@ -192,7 +305,7 @@ function resolveBaseModelId(profile: InferenceProfileSummary): string | undefine
192305
}
193306
if (profile.type === "SYSTEM_DEFINED") {
194307
const id = profile.inferenceProfileId ?? "";
195-
const prefixMatch = /^(?:us|eu|ap|jp|global)\.(.+)$/i.exec(id);
308+
const prefixMatch = /^(?:us|eu|ap|apac|au|jp|global)\.(.+)$/i.exec(id);
196309
if (prefixMatch) {
197310
return prefixMatch[1];
198311
}
@@ -282,7 +395,9 @@ function resolveInferenceProfiles(
282395
reasoning: baseModel?.reasoning ?? false,
283396
input: baseModel?.input ?? ["text"],
284397
cost: baseModel?.cost ?? DEFAULT_COST,
285-
contextWindow: baseModel?.contextWindow ?? defaults.contextWindow,
398+
contextWindow: baseModel?.contextWindow
399+
?? resolveKnownContextWindow(baseModelId ?? profile.inferenceProfileId ?? "")
400+
?? defaults.contextWindow,
286401
maxTokens: baseModel?.maxTokens ?? defaults.maxTokens,
287402
});
288403
}

0 commit comments

Comments
 (0)