Bug Description
Summary
When a cron job (or any agent run) is pinned to a non-Claude model on AWS Bedrock via provider: bedrock, every request fails with a 400 validation error before the model does any work. The same models work perfectly when called directly through the boto3 Converse API, so the bug is in Hermes' Bedrock request-building / routing layer — not in the models themselves.
Claude models (Opus, Haiku) on the same Bedrock provider are unaffected.
Environment
- Hermes version: 0.16.0
- Commit:
1c68f6f81 (2026-06-07)
- Provider:
bedrock (region us-east-1)
- Config:
model.provider: bedrock, model.base_url: https://bedrock-runtime.us-east-1.amazonaws.com
- Triggered from a scheduled cron job, but reproducible on any agent turn.
Two distinct failure modes
I tested 5 Bedrock models on the same job (a simple email-check task with the standard toolset). Each row was run via the cron runner and re-verified with a direct boto3 call.
| Model |
Model ID |
Result |
Error |
| Claude Opus 4 |
global.anthropic.claude-opus-4-8 |
✅ works |
— |
| Claude Haiku 4.5 |
us.anthropic.claude-haiku-4-5-20251001-v1:0 |
✅ works |
— |
| Kimi K2.5 |
moonshotai.kimi-k2.5 |
❌ fails 4/4 |
Group A |
| DeepSeek v3.2 |
deepseek.v3.2 |
❌ fails 2/2 |
Group A |
| Amazon Nova 2 Lite |
global.amazon.nova-2-lite-v1:0 |
❌ fails 2/2 |
Group B |
Group A — missing field 'type' (Kimi, DeepSeek):
RuntimeError: Error code: 400 - {'message': '{"error":{"code":"validation_error",
"message":"Failed to deserialize the JSON body into the target type: ?[0]:
Invalid \'tools\': missing field `type` at line 1 column 30848",
"param":null,"type":"invalid_request_error"}}'}
The error is an OpenAI-style invalid_request_error (not a boto3 ValidationException), which suggests these models are being routed to an OpenAI-compatible wire format where each tool must be wrapped as {"type": "function", "function": {...}}. At least one tool in the serialized payload is missing its top-level type field. The column offset shifts with payload size (22497 / 22692 / 28563 / 30848), and restricting the toolset to just terminal lowered it (28563 → 14858) but did not fix it — the offending tool is in the base toolset.
Group B — content: expected JSONArray, found String (Nova 2 Lite):
RuntimeError: Error code: 400 - {'message': 'Malformed input request:
#/messages/0/content: expected type: JSONArray, found: String,
reformat your input and try again.'}
Nova is stricter than Claude: it requires messages[].content to always be an array of content blocks, while Claude on Converse tolerates a bare string. Somewhere on this path a message's content is emitted as a string.
Proof the models are fine (control test via boto3)
All three "failing" models succeed when called directly through Converse, including with tools:
import boto3
c = boto3.client("bedrock-runtime", region_name="us-east-1")
tools = {"tools": [{"toolSpec": {"name": "terminal", "description": "run cmd",
"inputSchema": {"json": {"type": "object",
"properties": {"command": {"type": "string"}}, "required": ["command"]}}}]}
for m in ("deepseek.v3.2", "global.amazon.nova-2-lite-v1:0"):
r = c.converse(modelId=m,
messages=[{"role": "user", "content": [{"text": "list files"}]}],
toolConfig=tools, inferenceConfig={"maxTokens": 50})
print(m, "->", r["output"]["message"]["content"])
Both return a valid toolUse block. DeepSeek also passes with 15 tools in the payload.
Reproduction
- Set provider to
bedrock (region us-east-1).
- Pin a job/agent to
deepseek.v3.2 (or moonshotai.kimi-k2.5) and give it any task with tools enabled → Group A error.
- Pin to
global.amazon.nova-2-lite-v1:0 with the same task → Group B error.
- Pin to
global.anthropic.claude-opus-4-8 → works.
Regression note
This worked previously. A job pinned to Kimi K2.5 ran successfully ~30 times between 30 May and 9 Jun, then began failing 100% on 10 Jun — shortly after the host was updated to v0.16.0 (8 Jun). So something in the 0.16.0 tool/message serialization (or model routing) regressed for non-Claude Bedrock models.
Suspected root cause (pointers, not confirmed)
- Group A: Kimi/DeepSeek on Bedrock appear to be routed to an OpenAI-compatible wire path (the error shape is OpenAI's
invalid_request_error), and the tool serializer there drops the top-level type: "function" on at least one tool. Worth checking how agent/agent_init.py resolves api_mode for these model IDs vs. bedrock_converse, and the OpenAI-format tool builder.
- Group B: In
agent/bedrock_adapter.py, convert_messages_to_converse() / _convert_content_to_converse() can emit content as a string for some message shape; Nova rejects the non-array form that Claude accepts. Forcing content to always be a list of blocks should fix it.
Impact
Users who want to run cheaper/smaller models on Bedrock for lightweight tasks are forced back to Claude. Affects at least Kimi, DeepSeek, and Amazon Nova families.
Steps to Reproduce
-
Configure the bedrock provider (region us-east-1):
model.provider: bedrock
model.base_url: https://bedrock-runtime.us-east-1.amazonaws.com
-
Pin an agent/cron job to a non-Claude Bedrock model and give it any
task with tools enabled. Tested IDs:
- moonshotai.kimi-k2.5 → Group A error
- deepseek.v3.2 → Group A error
- global.amazon.nova-2-lite-v1:0 → Group B error
-
Run the turn. It fails with a 400 before the model produces output.
-
Control: pin global.anthropic.claude-opus-4-8 (or
us.anthropic.claude-haiku-4-5-20251001-v1:0) → works fine.
-
Control: call the same failing model directly via boto3 converse()
with a toolConfig → succeeds and returns a valid toolUse block,
proving the model itself is fine.
Expected Behavior
Non-Claude Bedrock models (Kimi, DeepSeek, Amazon Nova) should send a
well-formed request and complete the turn, exactly as they do when called
directly through the boto3 Converse API — and exactly as Claude models do
through the same Hermes bedrock provider.
Actual Behavior
Every request fails with a 400 validation error before any model work.
Two distinct failure modes:
Group A (Kimi K2.5, DeepSeek v3.2) — OpenAI-style invalid_request_error:
Invalid 'tools': missing field type at line 1 column 30848
→ a tool in the serialized payload is missing its {"type":"function"}
wrapper. Column offset shifts with payload size (22497 / 22692 /
28563 / 30848). Restricting the toolset to just terminal lowered
the offset (28563 → 14858) but did NOT fix it.
Group B (Amazon Nova 2 Lite):
Malformed input request: #/messages/0/content: expected type:
JSONArray, found: String
→ message content is emitted as a bare string; Nova requires an array
of content blocks (Claude tolerates the string, Nova does not).
Regression: a job pinned to Kimi K2.5 ran successfully ~30 times between
30 May and 9 Jun, then began failing 100% on 10 Jun — shortly after the
host updated to v0.16.0 (8 Jun).
Affected Component
Agent Core (conversation loop, context compression, memory)
Messaging Platform (if gateway-related)
No response
Debug Report
=== Hermes Debug Report (sanitized) ===
NOTE: Manually curated to exclude private chat transcripts, credentials, and
unrelated work data. Only Bedrock model-routing errors relevant to this bug
are included.
--- system ---
version: 0.16.0 (2026.6.5) [1c68f6f8]
os: AlmaLinux 9.7 (Linux 5.14.0 x86_64)
python: 3.11.15
openai_sdk: 2.24.0
profile: default
model: global.anthropic.claude-opus-4-8 (default - works)
provider: bedrock
region: us-east-1
terminal: local
--- features ---
toolsets: hermes-cli
mcp_servers: 0
memory_provider: built-in
cron_jobs: 2 active
skills: 110
--- config_overrides ---
compression.threshold: 0.8
display.streaming: true
--- relevant bedrock errors (agent.log, model-routing only) ---
2026-05-22 20:53:19 WARNING [session] API call failed (1/3) provider=bedrock model=moonshotai.kimi-k2.5 summary=HTTP 400: {"error":{"code":"validation_error","message":"... Invalid 'tools': missing field `type` at line 1 column 15856","type":"invalid_request_error"}}
2026-06-10 09:00:55 WARNING [cron_job] API call failed (1/3) provider=bedrock model=moonshotai.kimi-k2.5 summary=HTTP 400: {... Invalid 'tools': missing field `type` at line 1 column 22497 ...}
2026-06-11 09:46:59 WARNING [cron_job] API call failed (1/3) provider=bedrock model=global.amazon.nova-2-lite-v1:0 summary=HTTP 400: Malformed input request: #/messages/0/content: expected type: JSONArray, found: String
2026-06-11 09:52:59 WARNING [cron_job] API call failed (1/3) provider=bedrock model=deepseek.v3.2 summary=HTTP 400: {... Invalid 'tools': missing field `type` at line 1 column 30848 ...}
(full log: 16 entries — Kimi K2.5 ×9, DeepSeek v3.2 ×2, Nova 2 Lite ×2, plus 2 'model identifier is invalid' from a wrong-prefix probe. Claude Opus/Haiku on same provider never errored on serialization.)
Operating System
AlmaLinux 9.7
Python Version
3.11.15
Hermes Version
Hermes Agent v0.16.0 (2026.6.5) · upstream a72bb03
Additional Logs / Traceback (optional)
Root Cause Analysis (optional)
No response
Proposed Fix (optional)
No response
Are you willing to submit a PR for this?
Bug Description
Summary
When a cron job (or any agent run) is pinned to a non-Claude model on AWS Bedrock via
provider: bedrock, every request fails with a400validation error before the model does any work. The same models work perfectly when called directly through theboto3Converse API, so the bug is in Hermes' Bedrock request-building / routing layer — not in the models themselves.Claude models (Opus, Haiku) on the same Bedrock provider are unaffected.
Environment
1c68f6f81(2026-06-07)bedrock(regionus-east-1)model.provider: bedrock,model.base_url: https://bedrock-runtime.us-east-1.amazonaws.comTwo distinct failure modes
I tested 5 Bedrock models on the same job (a simple email-check task with the standard toolset). Each row was run via the cron runner and re-verified with a direct
boto3call.global.anthropic.claude-opus-4-8us.anthropic.claude-haiku-4-5-20251001-v1:0moonshotai.kimi-k2.5deepseek.v3.2global.amazon.nova-2-lite-v1:0Group A —
missing field 'type'(Kimi, DeepSeek):The error is an OpenAI-style
invalid_request_error(not a boto3ValidationException), which suggests these models are being routed to an OpenAI-compatible wire format where each tool must be wrapped as{"type": "function", "function": {...}}. At least one tool in the serialized payload is missing its top-leveltypefield. Thecolumnoffset shifts with payload size (22497 / 22692 / 28563 / 30848), and restricting the toolset to justterminallowered it (28563 → 14858) but did not fix it — the offending tool is in the base toolset.Group B —
content: expected JSONArray, found String(Nova 2 Lite):Nova is stricter than Claude: it requires
messages[].contentto always be an array of content blocks, while Claude on Converse tolerates a bare string. Somewhere on this path a message'scontentis emitted as a string.Proof the models are fine (control test via boto3)
All three "failing" models succeed when called directly through Converse, including with tools:
Both return a valid
toolUseblock. DeepSeek also passes with 15 tools in the payload.Reproduction
bedrock(regionus-east-1).deepseek.v3.2(ormoonshotai.kimi-k2.5) and give it any task with tools enabled → Group A error.global.amazon.nova-2-lite-v1:0with the same task → Group B error.global.anthropic.claude-opus-4-8→ works.Regression note
This worked previously. A job pinned to Kimi K2.5 ran successfully ~30 times between 30 May and 9 Jun, then began failing 100% on 10 Jun — shortly after the host was updated to v0.16.0 (8 Jun). So something in the 0.16.0 tool/message serialization (or model routing) regressed for non-Claude Bedrock models.
Suspected root cause (pointers, not confirmed)
invalid_request_error), and the tool serializer there drops the top-leveltype: "function"on at least one tool. Worth checking howagent/agent_init.pyresolvesapi_modefor these model IDs vs.bedrock_converse, and the OpenAI-format tool builder.agent/bedrock_adapter.py,convert_messages_to_converse()/_convert_content_to_converse()can emitcontentas a string for some message shape; Nova rejects the non-array form that Claude accepts. Forcingcontentto always be a list of blocks should fix it.Impact
Users who want to run cheaper/smaller models on Bedrock for lightweight tasks are forced back to Claude. Affects at least Kimi, DeepSeek, and Amazon Nova families.
Steps to Reproduce
Configure the bedrock provider (region us-east-1):
model.provider: bedrock
model.base_url: https://bedrock-runtime.us-east-1.amazonaws.com
Pin an agent/cron job to a non-Claude Bedrock model and give it any
task with tools enabled. Tested IDs:
Run the turn. It fails with a 400 before the model produces output.
Control: pin global.anthropic.claude-opus-4-8 (or
us.anthropic.claude-haiku-4-5-20251001-v1:0) → works fine.
Control: call the same failing model directly via boto3 converse()
with a toolConfig → succeeds and returns a valid toolUse block,
proving the model itself is fine.
Expected Behavior
Non-Claude Bedrock models (Kimi, DeepSeek, Amazon Nova) should send a
well-formed request and complete the turn, exactly as they do when called
directly through the boto3 Converse API — and exactly as Claude models do
through the same Hermes bedrock provider.
Actual Behavior
Every request fails with a 400 validation error before any model work.
Two distinct failure modes:
Group A (Kimi K2.5, DeepSeek v3.2) — OpenAI-style invalid_request_error:
Invalid 'tools': missing field
typeat line 1 column 30848→ a tool in the serialized payload is missing its {"type":"function"}
wrapper. Column offset shifts with payload size (22497 / 22692 /
28563 / 30848). Restricting the toolset to just
terminalloweredthe offset (28563 → 14858) but did NOT fix it.
Group B (Amazon Nova 2 Lite):
Malformed input request: #/messages/0/content: expected type:
JSONArray, found: String
→ message content is emitted as a bare string; Nova requires an array
of content blocks (Claude tolerates the string, Nova does not).
Regression: a job pinned to Kimi K2.5 ran successfully ~30 times between
30 May and 9 Jun, then began failing 100% on 10 Jun — shortly after the
host updated to v0.16.0 (8 Jun).
Affected Component
Agent Core (conversation loop, context compression, memory)
Messaging Platform (if gateway-related)
No response
Debug Report
Operating System
AlmaLinux 9.7
Python Version
3.11.15
Hermes Version
Hermes Agent v0.16.0 (2026.6.5) · upstream a72bb03
Additional Logs / Traceback (optional)
Root Cause Analysis (optional)
No response
Proposed Fix (optional)
No response
Are you willing to submit a PR for this?