High-level issues on the cloud repo side

There is one function on the cloud side that is implicated in this bug:

apps/web/src/lib/ai-gateway/llm-proxy-helpers.ts:168 — makeErrorReadable()

It has three distinct problems, only one of which matters for the infinite-loop bug, but all of which deserve attention:

Problem 1 — it rewrites any upstream 4xx/5xx into context_length_exceeded

The structure of the check is:

if (response.status < 400) return undefined;             // only run on errors
if (isUserByok) { ... }                                  // BYOK branch
const model = kiloExclusiveModels.find(m => m.public_id === requestedModel);
if (model) {
  const estimatedTokenCount = estimateTokenCount(request);
  if (estimatedTokenCount >= model.context_length) {
    // REWRITE: replace upstream error with a synthesized context_length_exceeded
    return NextResponse.json(
      { error, error_type: ProxyErrorType.context_length_exceeded, message: error },
      { status: response.status }
    );
  }
}

The stated intent (in the comment immediately above) is:

Sometimes we get generic or nonsensical errors when the context length is exceeded (such as "Internal Server Error" or "No allowed providers are available for the selected model")

— i.e. translate ambiguous overflow errors into a clearer message.

In practice there is no check that the upstream error was actually overflow-related. The rewrite fires for:

• Genuine overflows ✅ correct
• Provider outages (Novita/Minimax 500/502/503) ❌ wrong
• Rate limits (429) ❌ wrong
• Parse errors / malformed upstream responses ❌ wrong
• Any other 4xx/5xx that happens to land on a large request ❌ wrong

So yes — we are wrongly transforming errors. Any time a Kilo-exclusive model has an upstream hiccup on a moderately-sized request, the client sees context_length_exceeded instead of the real cause. The CLI then does what it's designed to do for that signal: compact and retry. If the upstream keeps failing, the cloud keeps translating, and the CLI keeps compacting → the exact loop PR #9344 caps.

Problem 2 — the token estimate is badly inaccurate and over-counts

function estimateTokenCount(request: GatewayRequest) {
  return Math.round(JSON.stringify(request).length / 4 + (getMaxTokens(request) ?? 0));
}

Two issues:

1. JSON.stringify(...).length / 4 — the "4 chars ≈ 1 token" heuristic is for plaintext English. JSON carries 50–100% overhead from quotes, braces, escape sequences, tool-definition scaffolding, cache-control markers, etc. So the "character side" of the estimate over-counts real tokens by roughly 1.5–2×.
2. Adding max_tokens — the model's context_length is total budget (input + output). max_tokens is the output budget. Adding them is formally correct, but clients typically send max_tokens = model.max_completion_tokens (the model's advertised max output). For minimax/minimax-m2.5:free that's 131_072 out of a 204_800 context — so the comparison effectively becomes "is the estimated input alone ≥ 73,728 tokens?", and with the 2× over-count, an actual 37k-token input is already enough to trigger the rewrite.

Net effect: the rewrite fires on inputs that would actually fit in the model. On kilo-auto/free (minimax-m2.5:free, 204.8k context, 131k max output) this is very easy to reach after just a few tool calls.

Problem 3 — the rewrite preserves the upstream status code

return NextResponse.json(
  { error, error_type: ProxyErrorType.context_length_exceeded, message: error },
  { status: response.status }        // <-- upstream status preserved
);

So a 500 upstream error becomes a 500 response with error.code === "context_length_exceeded". That shape confuses the CLI's error classifier: parseAPICallError uses statusCode === 413 as one overflow signal, so the status is no guide, but it uses body.error.code === "context_length_exceeded" as another — and this path matches that regardless of the underlying status. The effect: the CLI treats a genuine 500 outage as recoverable via compaction.

Timeline / PRs on the cloud side

The bug itself — the transformation code

None of the recent cloud PRs introduced the buggy rewrite. It has been the dominant error-translation path for Kilo-exclusive models since February.

The amplifiers — what made the rewrite fire frequently in the last week

Existing cloud-repo issues about this

I searched Kilo-Org/cloud for open issues about context_length_exceeded, makeErrorReadable, or estimateTokenCount. There are none. The bug is tracked entirely on the CLI side (Kilo-Org/kilocode#9285 by Zindaar, confirmed by visonforcoding). The cloud repo does not have an issue filed for the wrong-error-transformation problem yet.

Diagram

flowchart TD
    U[Upstream provider] -->|any 4xx or 5xx:<br/>500, 502, 503, 429, 400, ...| M[makeErrorReadable]

    M --> C1{BYOK?}
    C1 -->|yes| R1[BYOK-specific message]
    C1 -->|no| C2{Kilo-exclusive model?}
    C2 -->|no| Pass[pass upstream error through]
    C2 -->|yes| C3["estimateTokenCount >= context_length?<br/>(JSON.stringify/4 + max_tokens)"]
    C3 -->|no| C4{Stealth model?}
    C3 -->|yes| Rewrite["REWRITE to<br/>error_type: context_length_exceeded<br/>status: upstream status"]

    Rewrite --> CLI[CLI classifies as<br/>context_overflow]
    CLI --> Compact[auto-compact]
    Compact --> U

    style Rewrite fill:#fee,stroke:#c00
    style C3 fill:#ffe,stroke:#cc0

The red box is where we're wrongly transforming. The yellow box is where the transformation decision is made on a badly inflated heuristic. Together they produce the loop that PR #9344 now caps on the CLI side.

Recommended cloud-side follow-ups (not fixed by #9344)

1. Gate the rewrite on upstream error content, not just status + size. Only rewrite if the upstream body actually contains an ambiguous overflow signature (e.g. matches /maximum context|context.*length|token.*exceed/i or is empty/generic) — not for every 4xx/5xx.
2. Fix the over-counting estimate. Either (a) use an actual tokenizer (tiktoken / model-specific) before rewriting, or (b) widen the trigger threshold to context_length * 1.5 to compensate for JSON overhead.
3. Don't preserve the upstream status code on rewrite. If we're confident enough to call it overflow, return 413 (the canonical overflow status) so clients can rely on status code alone.
4. Add a cloud-repo issue tracking this — right now the only record is on the CLI side.