Skip to content

synthetic recovery markers leak to upstream API — Groq rejects with HTTP 400 #110

@PowerCreek

Description

@PowerCreek

Symptom

In v0.17.1 (post #108), running through Groq with HERMES_DEFER_PERSONA=1 + a tool-bearing prompt causes a cascade:

  1. Model emits finish_reason=tool_calls with empty content + empty tool_calls list
  2. PR fix(response): recover on finish_reason=tool_calls with empty list (closes #99) #108's recovery branch fires — appends synthetic assistant + user messages with _empty_recovery_synthetic: True markers
  3. Next API call serializes those messages verbatim
  4. Groq's strict OpenAI validation rejects: messages.2: property _empty_recovery_synthe... → HTTP 400
  5. Hermes retries, hits same 400, falls through to (empty) user-facing

Reproducible from sandbox session on duplex container (DEVAGENTIC_USER_ID=sandbox, HERMES_DEFER_PERSONA=1, v0.17.1).

The recovery currently makes the empty-content problem worse than no recovery — at least the original (empty) was a clean failure; now we get noisy 400s + cascading retries.

Root cause

conversation_loop.py builds the API payload from each in-memory message by api_msg = msg.copy(), then strips individual known internal fields (reasoning, finish_reason, _thinking_prefill). But other hermes-side bookkeeping markers — _empty_recovery_synthetic (PR #69 / #108), _empty_terminal_sentinel (PR #69) — are NOT in the strip list. They round-trip to upstream.

Strict OpenAI validators (Groq, Mistral, Fireworks) reject any unknown property. Permissive ones (OpenRouter, Anthropic) silently ignore — which is why the bug didn't surface until a Groq deployment exercised the new #108 path.

This is a pre-existing bug since v0.16.0 (PR #69 introduced _empty_recovery_synthetic), but #108 makes it fire much more often (any finish_reason=tool_calls with empty list, not just finish_reason=stop with no tool_calls and no prior tool turn).

Fix

Replace the individual _thinking_prefill strip with a generic leading-underscore strip in the API-message preparation loop:

# Strip ALL internal markers — keys starting with `_` are hermes-side
# bookkeeping and MUST NOT round-trip to upstream APIs.
for _k in [k for k in api_msg if isinstance(k, str) and k.startswith("_")]:
    api_msg.pop(_k, None)

The leading-underscore convention is stable; anything that genuinely needs to ride the API has a documented OpenAI shape. Future internal markers added with _ prefix get auto-stripped.

Acceptance

  • Groq-routed sessions with HERMES_DEFER_PERSONA=1 + tool-bearing prompts no longer cascade HTTP 400s on synthetic recovery.
  • Existing tests pass (no regression on Mistral / OpenRouter / Anthropic paths).
  • New unit test asserts each known marker (_empty_recovery_synthetic / _empty_terminal_sentinel / _thinking_prefill) is stripped + standard OpenAI fields are preserved.
  • v0.17.2 wheel cut + Deploy: container hermes binary is pre-G1-G4; needs redeploy to include new plugins #66 install instructions updated.

Composition

Devagentic#295 (tools-strip workaround) should NOT be lifted until this fix deploys.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions