Skip to content

Commit 605e7bf

Browse files
nvshaxienvshaxie
committed
fix(inference): wire ollama-local supportsUsageInStreaming via getSandboxInferenceConfig (#2747)
Follow-up to #3678. That PR added a Python-side conditional in `generate-openclaw-config.py` that toggled `supportsUsageInStreaming` when `NEMOCLAW_PROVIDER_KEY` was `"ollama"` or `"ollama-local"`. However, in real onboard runs `getSandboxInferenceConfig("model", "ollama-local")` hits the default switch arm and returns `providerKey = "inference"` — the same value used by all managed-inference providers — so the build-time `NEMOCLAW_PROVIDER_KEY` is never `"ollama"` and the conditional was dead code. The earlier regression tests in `test/generate-openclaw-config.test.ts` passed only because they set `NEMOCLAW_PROVIDER_KEY=ollama` directly, which never happens in production. Verified locally with the python script and the real env values NemoClaw passes through at docker build time (`NEMOCLAW_PROVIDER_KEY=inference`, `NEMOCLAW_INFERENCE_BASE_URL=https://inference.local/v1`, `NEMOCLAW_INFERENCE_COMPAT_B64=e30=` for an empty compat): the generated `openclaw.json` has `model.compat = null`, confirming the flag never landed. Move the wiring to the proper layer: add an explicit `"ollama-local"` case in `getSandboxInferenceConfig` that sets `inferenceCompat = { supportsUsageInStreaming: true }`. The existing encode-into-`NEMOCLAW_INFERENCE_COMPAT_B64` path then carries the flag into the python script's `inference_compat` dict, which writes it onto `model.compat` exactly as before — except now it actually fires for ollama-local builds. Cloud providers and other local backends keep their existing `inferenceCompat` (or absence thereof). Drop the now-redundant python conditional, drop the three unrealistic-fixture vitest cases from `test/generate-openclaw-config.test.ts`, and add two new ones in `src/lib/inference/config.test.ts` that exercise `getSandboxInferenceConfig` with realistic provider names. Test plan - `npx vitest run src/lib/inference/config.test.ts` — 24/24 pass, including the two new ones (`forces supportsUsageInStreaming for ollama-local`, `does not force supportsUsageInStreaming for non-ollama providers`) - `npx vitest run test/generate-openclaw-config.test.ts` — 67/67 pass (was 70/70 with the three deleted unrealistic cases) - Direct script run with prod-shaped env values: `NEMOCLAW_INFERENCE_COMPAT_B64={"supportsUsageInStreaming":true}` (the shape TypeScript will emit for ollama-local) → `model.compat = {'supportsUsageInStreaming': True}` in generated openclaw.json ✓ Signed-off-by: Shawn Xie <shaxie@nvidia.com>
1 parent 50ad764 commit 605e7bf

4 files changed

Lines changed: 54 additions & 82 deletions

File tree

‎scripts/generate-openclaw-config.py‎

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -423,24 +423,6 @@ def build_config(env: dict | None = None) -> dict:
423423
setup, inference_compat, openclaw_plugins, openclaw_plugin_ids
424424
)
425425

426-
# Ollama's OpenAI-compatible /v1/chat/completions stream omits the
427-
# `usage` chunk by default; OpenAI clients have to send
428-
# `stream_options.include_usage: true` to receive it. OpenClaw gates
429-
# that request flag on `model.compat.supportsUsageInStreaming`
430-
# (src/agents/openai-transport-stream.ts) and its Ollama extension
431-
# only opts in when its own detector recognises the endpoint as
432-
# Ollama. NemoClaw routes ollama-local traffic via the standardised
433-
# `https://inference.local/v1` URL through the OpenShell gateway, so
434-
# the upstream detector misses it and the TUI token counter stays
435-
# `?` indefinitely (#2747). Set the flag here so the request is sent
436-
# with `stream_options.include_usage: true` regardless of how
437-
# OpenClaw resolves the provider id. Mirrors the LM Studio extension
438-
# workaround (`withLmstudioUsageCompat` in
439-
# extensions/lmstudio/src/stream.ts). Keep the set of provider keys
440-
# in sync with `_bundled_provider_plugins["ollama"]` below.
441-
if provider_key in {"ollama", "ollama-local"}:
442-
inference_compat.setdefault("supportsUsageInStreaming", True)
443-
444426
msg_channels = json.loads(
445427
base64.b64decode(
446428
env.get("NEMOCLAW_MESSAGING_CHANNELS_B64", "W10=") or "W10="

‎src/lib/inference/config.test.ts‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -282,6 +282,43 @@ describe("getSandboxInferenceConfig", () => {
282282
});
283283
});
284284

285+
// #2747 followup to #3678: Ollama's OpenAI-compat stream omits the usage
286+
// chunk unless the request sets `stream_options.include_usage: true`.
287+
// OpenClaw gates that on `model.compat.supportsUsageInStreaming`. The
288+
// ollama-local path runs through `inference.local`, which OpenClaw's
289+
// own Ollama detector doesn't recognise — so the compat flag has to
290+
// travel into openclaw.json via `inferenceCompat`. The previous Python-
291+
// side conditional in `generate-openclaw-config.py` could never fire
292+
// because `providerKey` resolves to `MANAGED_PROVIDER_ID` ("inference")
293+
// for ollama-local, not "ollama" / "ollama-local". Wiring it here is
294+
// the right layer.
295+
it("forces supportsUsageInStreaming for ollama-local (#2747)", () => {
296+
expect(getSandboxInferenceConfig("qwen2.5:7b", "ollama-local")).toEqual({
297+
providerKey: MANAGED_PROVIDER_ID,
298+
primaryModelRef: `${MANAGED_PROVIDER_ID}/qwen2.5:7b`,
299+
inferenceBaseUrl: INFERENCE_ROUTE_URL,
300+
inferenceApi: "openai-completions",
301+
inferenceCompat: {
302+
supportsUsageInStreaming: true,
303+
},
304+
});
305+
});
306+
307+
it("does not force supportsUsageInStreaming for non-ollama providers", () => {
308+
for (const provider of [
309+
"nvidia-prod",
310+
"nvidia-nim",
311+
"nvidia-router",
312+
"openai-api",
313+
"compatible-endpoint",
314+
"anthropic-prod",
315+
"gemini-api",
316+
]) {
317+
const compat = getSandboxInferenceConfig("model", provider).inferenceCompat;
318+
expect(compat?.supportsUsageInStreaming).toBeUndefined();
319+
}
320+
});
321+
285322
it("uses a probed Responses API override when one is available", () => {
286323
expect(getSandboxInferenceConfig("gpt-5.4", "openai-api", "openai-responses")).toEqual({
287324
providerKey: "openai",

‎src/lib/inference/config.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,23 @@ export function getSandboxInferenceConfig(
218218
providerKey = MANAGED_PROVIDER_ID;
219219
primaryModelRef = `${MANAGED_PROVIDER_ID}/${model}`;
220220
break;
221+
case "ollama-local":
222+
// Ollama's OpenAI-compatible /v1/chat/completions stream omits the
223+
// `usage` chunk unless the request sets `stream_options.include_usage:
224+
// true`. OpenClaw gates that on `model.compat.supportsUsageInStreaming`
225+
// (src/agents/openai-transport-stream.ts upstream); its own Ollama
226+
// extension auto-detects the flag only when the endpoint reads as
227+
// localhost:11434 or `model.provider === "ollama"` exactly, which
228+
// doesn't match the standardised `inference.local` URL NemoClaw routes
229+
// through. Pin the compat flag here so the TUI token counter updates
230+
// after each turn instead of staying `?` (#2747). Mirrors the LM Studio
231+
// extension's `withLmstudioUsageCompat` workaround.
232+
providerKey = MANAGED_PROVIDER_ID;
233+
primaryModelRef = `${MANAGED_PROVIDER_ID}/${model}`;
234+
inferenceCompat = {
235+
supportsUsageInStreaming: true,
236+
};
237+
break;
221238
case "nvidia-prod":
222239
case "nvidia-nim":
223240
default:

‎test/generate-openclaw-config.test.ts‎

Lines changed: 0 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -414,70 +414,6 @@ describe("generate-openclaw-config.py: config generation", () => {
414414
});
415415
});
416416

417-
// #2747: Ollama's OpenAI-compatible streaming API omits the usage chunk
418-
// unless `stream_options.include_usage` is set on the request. OpenClaw
419-
// gates that on `model.compat.supportsUsageInStreaming`. NemoClaw routes
420-
// ollama-local through the standardised `inference.local` URL, which
421-
// OpenClaw's own Ollama detector does not recognise — so we force the
422-
// flag here. Cloud providers and other local backends must not be
423-
// affected.
424-
it("enables supportsUsageInStreaming for Ollama provider keys (#2747)", () => {
425-
for (const providerKey of ["ollama", "ollama-local"]) {
426-
const config = runConfigScript({
427-
NEMOCLAW_MODEL: "qwen2.5:7b",
428-
NEMOCLAW_PROVIDER_KEY: providerKey,
429-
NEMOCLAW_PRIMARY_MODEL_REF: "qwen2.5:7b",
430-
NEMOCLAW_INFERENCE_BASE_URL: "https://inference.local/v1",
431-
NEMOCLAW_INFERENCE_API: "openai-completions",
432-
});
433-
const model = config.models.providers[providerKey].models[0];
434-
expect(model.compat?.supportsUsageInStreaming).toBe(true);
435-
}
436-
});
437-
438-
it("does not enable supportsUsageInStreaming for non-Ollama providers (#2747)", () => {
439-
const cases = [
440-
{ NEMOCLAW_PROVIDER_KEY: "openai", NEMOCLAW_INFERENCE_BASE_URL: "https://api.openai.com/v1" },
441-
{
442-
NEMOCLAW_PROVIDER_KEY: "anthropic",
443-
NEMOCLAW_INFERENCE_BASE_URL: "https://api.anthropic.com",
444-
},
445-
{ NEMOCLAW_PROVIDER_KEY: "vllm", NEMOCLAW_INFERENCE_BASE_URL: "https://inference.local/v1" },
446-
{
447-
NEMOCLAW_PROVIDER_KEY: "nim-local",
448-
NEMOCLAW_INFERENCE_BASE_URL: "https://inference.local/v1",
449-
},
450-
];
451-
452-
for (const envCase of cases) {
453-
const config = runConfigScript({
454-
NEMOCLAW_MODEL: "test-model",
455-
NEMOCLAW_PRIMARY_MODEL_REF: "test-ref",
456-
NEMOCLAW_INFERENCE_API: "openai-completions",
457-
...envCase,
458-
});
459-
const model = config.models.providers[envCase.NEMOCLAW_PROVIDER_KEY].models[0];
460-
expect(model.compat?.supportsUsageInStreaming).toBeUndefined();
461-
}
462-
});
463-
464-
// If a future model-specific-setup manifest declares
465-
// supportsUsageInStreaming explicitly, that decision should win over our
466-
// ollama-keyed default — including when a manifest opts the flag *off*.
467-
it("respects existing supportsUsageInStreaming from inference compat (#2747)", () => {
468-
const config = runConfigScript({
469-
NEMOCLAW_MODEL: "qwen2.5:7b",
470-
NEMOCLAW_PROVIDER_KEY: "ollama",
471-
NEMOCLAW_PRIMARY_MODEL_REF: "qwen2.5:7b",
472-
NEMOCLAW_INFERENCE_BASE_URL: "https://inference.local/v1",
473-
NEMOCLAW_INFERENCE_API: "openai-completions",
474-
NEMOCLAW_INFERENCE_COMPAT_B64: Buffer.from(
475-
JSON.stringify({ supportsUsageInStreaming: false }),
476-
).toString("base64"),
477-
});
478-
expect(config.models.providers.ollama.models[0].compat.supportsUsageInStreaming).toBe(false);
479-
});
480-
481417
it("does not activate the OpenClaw Kimi setup for non-matching routes", () => {
482418
const cases = [
483419
{ NEMOCLAW_MODEL: "deepseek-ai/DeepSeek-V4-Flash" },

0 commit comments

Comments
 (0)