Skip to content

[Bug] Custom OpenAI-compatible provider (Qwen/DashScope) fails with 400 in JSON mode due to missing JSON hint in messages #453

@veedaisme

Description

@veedaisme

🐞 Bug Report

Describe the bug

When Honcho is configured to use a custom OpenAI-compatible provider backed by
DashScope/Qwen (e.g. qwen3.5-plus), any background task that uses JSON mode
(response_format={"type":"json_object"}) fails immediately with a 400 error:

Error code: 400 - {'error': {'message': "<400> InternalError.Algo.InvalidParameter:
'messages' must contain the word 'json' in some form, to use 'response_format' of
type 'json_object'.", 'type': 'invalid_request_error', ...}}

The deriver starts up successfully but its first structured-output LLM call is
rejected before generation even begins. The worker retries 3 times then stays in a
broken loop.


To Reproduce

  1. Self-host Honcho and set the following in config.toml:
    [llm]
    OPENAI_COMPATIBLE_API_KEY = "your-dashscope-key"
    OPENAI_COMPATIBLE_BASE_URL = "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
    
    [deriver]
    PROVIDER = "custom"
    MODEL = "qwen3.5-plus"
  2. Start Honcho with docker compose up.
  3. Watch the deriver logs: docker compose logs --tail=100 -f deriver
  4. See the repeated 400 invalid_parameter_error on every structured-output attempt.

Expected behaviour

Honcho should be able to use DashScope/Qwen through the custom OpenAI-compatible
provider path for JSON-mode requests without request rejection. Qwen does support
response_format={"type":"json_object"}; it just additionally requires the request
messages to explicitly contain the word json.


Media prove

Deriver log output:

2026-03-26 14:48:07,442 - src.utils.clients - WARNING - Error on attempt 1/3 with
custom/qwen3.5-plus: Error code: 400 - {'error': {'message': "<400>
InternalError.Algo.InvalidParameter: 'messages' must contain the word 'json' in
some form, to use 'response_format' of type 'json_object'.", ...}}

Your environment

  • OS: Ubuntu (Docker self-hosted)
  • Honcho Server Version: latest main
  • Custom Provider: Alibaba Cloud DashScope — https://dashscope-intl.aliyuncs.com/compatible-mode/v1
  • Model: qwen3.5-plus

Additional context

Root cause: src/utils/clients.py sets response_format={"type":"json_object"}
in the shared AsyncOpenAI branch for JSON mode, but does not guarantee the outgoing
messages explicitly mention the word json. DashScope/Qwen validates this server-side
and rejects the request if the word is absent.

This is documented Qwen behaviour:
https://www.alibabacloud.com/help/en/model-studio/qwen-structured-output

Affected components: The bug affects any Honcho component that uses JSON mode
through the custom OpenAI-compatible path, not just the deriver — summary and dream
likely hit the same issue.

Suggested fix: In the AsyncOpenAI client path in src/utils/clients.py, when
json_mode=True and the provider is custom and the model name contains qwen,
prepend a system message containing a JSON instruction if none of the existing messages
already mention json:

if json_mode and provider != "vllm":
    if provider == "custom" and "qwen" in model.lower():
        has_json_hint = any(
            isinstance(m.get("content"), str)
            and "json" in m["content"].lower()
            for m in openai_params["messages"]
        )
        if not has_json_hint:
            openai_params["messages"] = [
                {
                    "role": "system",
                    "content": "Return valid JSON only. The output must be a JSON object.",
                },
                *openai_params["messages"],
            ]
    openai_params["response_format"] = {"type": "json_object"}

This needs to be applied to both the non-streaming and streaming AsyncOpenAI
branches. The same pattern applies for the elif json_mode: branch inside the
streaming path.

I have tested this fix locally and it resolves the deriver 400 failures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions