Bug Description

Affects: v2026.4.3 (commit 77a2aad)
Models: Kimi K2.5, GLM-5, Qwen (via Ollama)

Summary

Models served through Ollama can return invalid surrogate code points (U+D800-U+DFFF) that crash json.dumps() during UTF-8 serialization. This causes a UnicodeEncodeError that retries 3 times and then fails completely.

API call failed after 3 retries: 'utf-8' codec can't encode characters in position 45559-45560: surrogates not allowed
    Model: glm-5:cloud

Steps to Reproduce

1. Set up Hermes with an Ollama-hosted model (Kimi K2.5, GLM-5, or Qwen)
2. Configure the model in config.yaml
3. Trigger a large payload that may contain surrogate characters in:
   • File read operations with binary-like content
   • Long tool outputs or session history
   • Any content where the model's response contains lone surrogates
4. The API call will fail with UnicodeEncodeError, retry 3 times, and crash

Minimal reproduction:

In Hermes CLI with glm-5:cloud or kimi-k2.5:cloud via Ollama:

hermes

Ask the model to process a file or have a long conversation

that produces a response with surrogate characters:

read a large binary-adjacent file and summarize it

OR

continue a long session where history contains surrogates

Error appears:

API call failed after 3 retries: 'utf-8' codec can't encode

characters in position XXXXX-XXXXX: surrogates not allowed

Why it reproduces intermittently: Surrogates appear in model outputs unpredictably. Some Ollama models occasionally emit them when processing certain content. The position number (e.g., 45559) indicates where in the serialized JSON payload the surrogates occur.

Expected Behavior

Commands should be processed.

Actual Behavior

API call failed after 3 retries: 'utf-8' codec can't encode characters in position 45559-45560: surrogates not allowed

..and it doesn't process the command.

Affected Component

CLI (interactive chat), Gateway (Telegram/Discord/Slack/WhatsApp)

Messaging Platform (if gateway-related)

No response

Operating System

macOS 26.2

Python Version

Python 3.11.15

Hermes Version

0.7.0 (2026.4.3)

Relevant Logs / Traceback

Root Cause Analysis (optional)

Models served through Ollama can return invalid surrogate code points (U+D800-U+DFFF) that crash json.dumps() during UTF-8 serialization. This causes a UnicodeEncodeError that retries 3 times and then fails completely.

Proposed Fix (optional)

Fix (3 call sites)

1. Add proactive sanitization before API call (line ~6823)

Insert after _sanitize_api_messages():

            api_messages = self._sanitize_api_messages(api_messages)

            # Proactive surrogate sanitization: clean any lone surrogates from
            # tool results, session history, or other sources before API call.
            # Prevents UnicodeEncodeError during json.dumps() serialization.
            _sanitize_messages_surrogates(api_messages)

            # Calculate approximate request size for logging

2. Sanitize API responses before storing (line ~5242)

Replace:

        msg = {
            "role": "assistant",
            "content": assistant_message.content or "",
            "reasoning": reasoning_text,
            "finish_reason": finish_reason,
        }

With:

        # Sanitize surrogates from API response content before storing. Some models
        # (e.g., Kimi through Ollama) can return invalid surrogates in their
        # output, which crashes json.dumps() when persisting or logging.
        raw_content = assistant_message.content or ""
        sanitized_content = _sanitize_surrogates(raw_content)
        if reasoning_text:
            reasoning_text = _sanitize_surrogates(reasoning_text)

        msg = {
            "role": "assistant",
            "content": sanitized_content,
            "reasoning": reasoning_text,
            "finish_reason": finish_reason,
        }

3. Fix retry path (line ~7336)

# Change:
if _sanitize_messages_surrogates(messages):

# To:
if _sanitize_messages_surrogates(api_messages):

Files Changed

• run_agent.py: 3 call sites (lines ~5242, ~6823, ~7336)

Are you willing to submit a PR for this?

• I'd like to fix this myself and submit a PR