Skip to content

[Bug]: Non-Claude Bedrock models fail with malformed tool/message payloads (Kimi, DeepSeek, Amazon Nova) #43946

@tatum163

Description

@tatum163

Bug Description

Summary

When a cron job (or any agent run) is pinned to a non-Claude model on AWS Bedrock via provider: bedrock, every request fails with a 400 validation error before the model does any work. The same models work perfectly when called directly through the boto3 Converse API, so the bug is in Hermes' Bedrock request-building / routing layer — not in the models themselves.

Claude models (Opus, Haiku) on the same Bedrock provider are unaffected.

Environment

  • Hermes version: 0.16.0
  • Commit: 1c68f6f81 (2026-06-07)
  • Provider: bedrock (region us-east-1)
  • Config: model.provider: bedrock, model.base_url: https://bedrock-runtime.us-east-1.amazonaws.com
  • Triggered from a scheduled cron job, but reproducible on any agent turn.

Two distinct failure modes

I tested 5 Bedrock models on the same job (a simple email-check task with the standard toolset). Each row was run via the cron runner and re-verified with a direct boto3 call.

Model Model ID Result Error
Claude Opus 4 global.anthropic.claude-opus-4-8 ✅ works
Claude Haiku 4.5 us.anthropic.claude-haiku-4-5-20251001-v1:0 ✅ works
Kimi K2.5 moonshotai.kimi-k2.5 ❌ fails 4/4 Group A
DeepSeek v3.2 deepseek.v3.2 ❌ fails 2/2 Group A
Amazon Nova 2 Lite global.amazon.nova-2-lite-v1:0 ❌ fails 2/2 Group B

Group A — missing field 'type' (Kimi, DeepSeek):

RuntimeError: Error code: 400 - {'message': '{"error":{"code":"validation_error",
"message":"Failed to deserialize the JSON body into the target type: ?[0]:
Invalid \'tools\': missing field `type` at line 1 column 30848",
"param":null,"type":"invalid_request_error"}}'}

The error is an OpenAI-style invalid_request_error (not a boto3 ValidationException), which suggests these models are being routed to an OpenAI-compatible wire format where each tool must be wrapped as {"type": "function", "function": {...}}. At least one tool in the serialized payload is missing its top-level type field. The column offset shifts with payload size (22497 / 22692 / 28563 / 30848), and restricting the toolset to just terminal lowered it (28563 → 14858) but did not fix it — the offending tool is in the base toolset.

Group B — content: expected JSONArray, found String (Nova 2 Lite):

RuntimeError: Error code: 400 - {'message': 'Malformed input request:
#/messages/0/content: expected type: JSONArray, found: String,
reformat your input and try again.'}

Nova is stricter than Claude: it requires messages[].content to always be an array of content blocks, while Claude on Converse tolerates a bare string. Somewhere on this path a message's content is emitted as a string.

Proof the models are fine (control test via boto3)

All three "failing" models succeed when called directly through Converse, including with tools:

import boto3
c = boto3.client("bedrock-runtime", region_name="us-east-1")
tools = {"tools": [{"toolSpec": {"name": "terminal", "description": "run cmd",
    "inputSchema": {"json": {"type": "object",
    "properties": {"command": {"type": "string"}}, "required": ["command"]}}}]}

for m in ("deepseek.v3.2", "global.amazon.nova-2-lite-v1:0"):
    r = c.converse(modelId=m,
        messages=[{"role": "user", "content": [{"text": "list files"}]}],
        toolConfig=tools, inferenceConfig={"maxTokens": 50})
    print(m, "->", r["output"]["message"]["content"])

Both return a valid toolUse block. DeepSeek also passes with 15 tools in the payload.

Reproduction

  1. Set provider to bedrock (region us-east-1).
  2. Pin a job/agent to deepseek.v3.2 (or moonshotai.kimi-k2.5) and give it any task with tools enabled → Group A error.
  3. Pin to global.amazon.nova-2-lite-v1:0 with the same task → Group B error.
  4. Pin to global.anthropic.claude-opus-4-8 → works.

Regression note

This worked previously. A job pinned to Kimi K2.5 ran successfully ~30 times between 30 May and 9 Jun, then began failing 100% on 10 Jun — shortly after the host was updated to v0.16.0 (8 Jun). So something in the 0.16.0 tool/message serialization (or model routing) regressed for non-Claude Bedrock models.

Suspected root cause (pointers, not confirmed)

  • Group A: Kimi/DeepSeek on Bedrock appear to be routed to an OpenAI-compatible wire path (the error shape is OpenAI's invalid_request_error), and the tool serializer there drops the top-level type: "function" on at least one tool. Worth checking how agent/agent_init.py resolves api_mode for these model IDs vs. bedrock_converse, and the OpenAI-format tool builder.
  • Group B: In agent/bedrock_adapter.py, convert_messages_to_converse() / _convert_content_to_converse() can emit content as a string for some message shape; Nova rejects the non-array form that Claude accepts. Forcing content to always be a list of blocks should fix it.

Impact

Users who want to run cheaper/smaller models on Bedrock for lightweight tasks are forced back to Claude. Affects at least Kimi, DeepSeek, and Amazon Nova families.

Steps to Reproduce

  1. Configure the bedrock provider (region us-east-1):
    model.provider: bedrock
    model.base_url: https://bedrock-runtime.us-east-1.amazonaws.com

  2. Pin an agent/cron job to a non-Claude Bedrock model and give it any
    task with tools enabled. Tested IDs:

    • moonshotai.kimi-k2.5 → Group A error
    • deepseek.v3.2 → Group A error
    • global.amazon.nova-2-lite-v1:0 → Group B error
  3. Run the turn. It fails with a 400 before the model produces output.

  4. Control: pin global.anthropic.claude-opus-4-8 (or
    us.anthropic.claude-haiku-4-5-20251001-v1:0) → works fine.

  5. Control: call the same failing model directly via boto3 converse()
    with a toolConfig → succeeds and returns a valid toolUse block,
    proving the model itself is fine.

Expected Behavior

Non-Claude Bedrock models (Kimi, DeepSeek, Amazon Nova) should send a
well-formed request and complete the turn, exactly as they do when called
directly through the boto3 Converse API — and exactly as Claude models do
through the same Hermes bedrock provider.

Actual Behavior

Every request fails with a 400 validation error before any model work.
Two distinct failure modes:

Group A (Kimi K2.5, DeepSeek v3.2) — OpenAI-style invalid_request_error:
Invalid 'tools': missing field type at line 1 column 30848
→ a tool in the serialized payload is missing its {"type":"function"}
wrapper. Column offset shifts with payload size (22497 / 22692 /
28563 / 30848). Restricting the toolset to just terminal lowered
the offset (28563 → 14858) but did NOT fix it.

Group B (Amazon Nova 2 Lite):
Malformed input request: #/messages/0/content: expected type:
JSONArray, found: String
→ message content is emitted as a bare string; Nova requires an array
of content blocks (Claude tolerates the string, Nova does not).

Regression: a job pinned to Kimi K2.5 ran successfully ~30 times between
30 May and 9 Jun, then began failing 100% on 10 Jun — shortly after the
host updated to v0.16.0 (8 Jun).

Affected Component

Agent Core (conversation loop, context compression, memory)

Messaging Platform (if gateway-related)

No response

Debug Report

=== Hermes Debug Report (sanitized) ===
NOTE: Manually curated to exclude private chat transcripts, credentials, and
unrelated work data. Only Bedrock model-routing errors relevant to this bug
are included.

--- system ---
version:     0.16.0 (2026.6.5) [1c68f6f8]
os:          AlmaLinux 9.7 (Linux 5.14.0 x86_64)
python:      3.11.15
openai_sdk:  2.24.0
profile:     default
model:       global.anthropic.claude-opus-4-8   (default - works)
provider:    bedrock
region:      us-east-1
terminal:    local

--- features ---
toolsets:        hermes-cli
mcp_servers:     0
memory_provider: built-in
cron_jobs:       2 active
skills:          110

--- config_overrides ---
compression.threshold: 0.8
display.streaming:     true

--- relevant bedrock errors (agent.log, model-routing only) ---
2026-05-22 20:53:19 WARNING [session] API call failed (1/3) provider=bedrock model=moonshotai.kimi-k2.5 summary=HTTP 400: {"error":{"code":"validation_error","message":"... Invalid 'tools': missing field `type` at line 1 column 15856","type":"invalid_request_error"}}

2026-06-10 09:00:55 WARNING [cron_job] API call failed (1/3) provider=bedrock model=moonshotai.kimi-k2.5 summary=HTTP 400: {... Invalid 'tools': missing field `type` at line 1 column 22497 ...}

2026-06-11 09:46:59 WARNING [cron_job] API call failed (1/3) provider=bedrock model=global.amazon.nova-2-lite-v1:0 summary=HTTP 400: Malformed input request: #/messages/0/content: expected type: JSONArray, found: String

2026-06-11 09:52:59 WARNING [cron_job] API call failed (1/3) provider=bedrock model=deepseek.v3.2 summary=HTTP 400: {... Invalid 'tools': missing field `type` at line 1 column 30848 ...}

(full log: 16 entries — Kimi K2.5 ×9, DeepSeek v3.2 ×2, Nova 2 Lite ×2, plus 2 'model identifier is invalid' from a wrong-prefix probe. Claude Opus/Haiku on same provider never errored on serialization.)

Operating System

AlmaLinux 9.7

Python Version

3.11.15

Hermes Version

Hermes Agent v0.16.0 (2026.6.5) · upstream a72bb03

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

No response

Proposed Fix (optional)

No response

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/agentCore agent loop, run_agent.py, prompt builderprovider/bedrockAWS Bedrock (boto3, IAM)provider/deepseekDeepSeek APIprovider/kimiKimi / Moonshottype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions