Skip to content

fix: support reasoning compatible endpoints (Fixes #3279)#3286

Open
deepujain wants to merge 3 commits into
NVIDIA:mainfrom
deepujain:fix/3279-compatible-reasoning-flag
Open

fix: support reasoning compatible endpoints (Fixes #3279)#3286
deepujain wants to merge 3 commits into
NVIDIA:mainfrom
deepujain:fix/3279-compatible-reasoning-flag

Conversation

@deepujain

@deepujain deepujain commented May 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Custom OpenAI-compatible providers can opt into reasoning-mode validation through NEMOCLAW_REASONING. When that flag is enabled, NemoClaw skips the Responses/tool-call probe that reasoning-only wrappers often reject, and the sandbox smoke check accepts reasoning text as valid output.

Fixes #3279.

Changes

  • Normalized NEMOCLAW_REASONING aliases such as yes, 1, no, and 0.
  • Persisted compatible-endpoint reasoning state through onboarding session updates and provider resume.
  • Skipped Responses/tool-call and streaming probes for reasoning-mode compatible endpoints.
  • Increased the sandbox smoke response budget and accepted reasoning_content or reasoning when message.content is empty.
  • Added focused tests for flag normalization, provider state handoff, session persistence, and the custom endpoint reasoning path.

Testing

  • npm run build:cli passed.
  • npm run typecheck:cli passed.
  • npm test -- test/onboard-selection.test.ts -t "honors NEMOCLAW_REASONING" passed.
  • npm test -- src/lib/onboard/machine/handlers/provider-inference.test.ts src/lib/state/onboard-session.test.ts src/lib/onboard/compatible-endpoint-smoke.test.ts passed.
  • HOME=/private/tmp/nemoclaw-test-home-3279-rebase npm test -- --reporter=dot was attempted; it timed out locally after 180s before completing.

Evidence it works

The focused custom endpoint test simulates a reasoning-only chat response with empty content and non-empty reasoning_content. With NEMOCLAW_REASONING=yes, onboarding accepts the provider, avoids /responses and streaming probes, and keeps preferredInferenceApi on chat completions. Session tests cover persisting and clearing the reasoning flag during provider handoff and resume.

Signed-off-by: Deepak Jain deepujain@gmail.com

@copy-pr-bot

copy-pr-bot Bot commented May 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented May 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds normalization and configuration helpers for a compatible-endpoint reasoning mode, persists the flag in onboarding session state, applies it during provider selection/resume, conditions custom OpenAI-like endpoint validation and smoke probes on the flag (including larger token budget and reasoning-field fallbacks), and adds unit/integration tests.

Changes

Reasoning mode support for OpenAI-compatible endpoints

Layer / File(s) Summary
Reasoning helpers
src/lib/onboard.ts
Adds normalizeReasoningFlag() and configureCompatibleEndpointReasoning() and exports them; they normalize common aliases and mirror the resolved value into process.env.NEMOCLAW_REASONING.
Session persistence
src/lib/state/onboard-session.ts
Adds `compatibleEndpointReasoning: string
Provider selection & probe wiring
src/lib/onboard.ts
Calls configureCompatibleEndpointReasoning() during custom endpoint setup and resume, stores/clears the flag in setup flows, includes it in toSessionUpdates(), and modifies validateCustomOpenAiLikeSelection() to branch probe options on NEMOCLAW_REASONING === "true" (tool-calling, responses probe, streaming).
Smoke test
src/lib/onboard.ts
Increases smoke probe max_tokens to 512 and accepts response text from message.content with fallbacks to message.reasoning_content and message.reasoning.
Helper exports
src/lib/onboard.ts
Exports normalizeReasoningFlag and configureCompatibleEndpointReasoning via module.exports.
Unit tests
test/onboard.test.ts
Extends OnboardTestInternals to include the two helpers and adds a unit test for alias normalization and environment config behavior.
Integration test
test/onboard-selection.test.ts
Adds a test that sets NEMOCLAW_REASONING=yes with a custom endpoint, verifies the reasoning-model selection, that validation uses /chat/completions (not /responses) and does not use streaming, and that the reasoning-mode onboarding prompt is suppressed.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 I nudge the flag from yes to true,
I hop where reasoning fields accrue,
Custom endpoints hear the call,
content or reasoning — we accept them all,
A little normalization, that's my cue.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically summarizes the main change: adding support for reasoning-mode compatible endpoints, directly addressing the primary fix objective.
Linked Issues check ✅ Passed The PR fully implements the requirements from #3279: NEMOCLAW_REASONING is now configurable for Option 3 providers, reasoning responses are correctly extracted, and reasoning-mode probes are properly skipped.
Out of Scope Changes check ✅ Passed All changes are directly scoped to implementing reasoning-mode support for compatible endpoints; no unrelated modifications are present in onboard.ts, test files, or session state.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
test/onboard-selection.test.ts (1)

2075-2094: ⚡ Quick win

Assert probe path behavior explicitly in reasoning mode.

Line [2077] and Line [2150] currently validate end-state only; this test can still pass if /responses (or streaming probe) is called and then fallback succeeds. Please log curl args and assert reasoning mode avoids /responses/-N while still hitting /chat/completions.

Suggested test hardening
@@
     const scriptPath = path.join(tmpDir, "custom-openai-reasoning-check.js");
+    const curlArgsLog = path.join(tmpDir, "custom-openai-reasoning-curl-args.log");
@@
       path.join(fakeBin, "curl"),
       `#!/usr/bin/env bash
+args_log=${JSON.stringify(curlArgsLog)}
+printf '%s\\n' "$*" >> "$args_log"
 body='{"error":{"message":"bad request"}}'
 status="400"
 outfile=""
 url=""
@@
     assert.equal(payload.result.model, "reasoning-model");
     assert.equal(payload.result.preferredInferenceApi, "openai-completions");
     assert.equal(payload.reasoning, "true");
+    const curlInvocations = fs.readFileSync(curlArgsLog, "utf-8");
+    assert.match(curlInvocations, /chat\/completions/);
+    assert.doesNotMatch(curlInvocations, /\/responses/);
+    assert.doesNotMatch(curlInvocations, /(^|\s)-N(\s|$)/);
     assert.ok(
       payload.messages.every(
         (message: string) => !/Enable reasoning mode for this model/.test(message),
       ),
     );

Also applies to: 2150-2160

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard-selection.test.ts` around lines 2075 - 2094, The fake curl
script written to path.join(fakeBin, "curl") should record its received
arguments so the test can assert probe behavior: modify the heredoc to append
the full "$@" or each processed arg into a temp log file (e.g.,
"$outfile.args.log" or a known capture file in the test dir) before handling
-o/url logic, and keep returning the same body/status logic; then update the
test assertions to read that captured args log and explicitly assert that in
reasoning mode the captured args do not include '/responses' nor the '-N' flag
and do include a call to '/chat/completions'. Ensure you reference the same
fakeBin/curl generation code and the test's reasoning-mode invocation when
adding the new assertions.
test/onboard.test.ts (1)

391-410: ⚡ Quick win

Add an explicit assertion for the unset/default reasoning path.

This test validates aliases, but not the key default behavior when NEMOCLAW_REASONING is unset. Locking that in will better guard the PR’s intended fallback semantics.

✅ Suggested test addition
   expect(normalizeReasoningFlag("maybe")).toBeNull();

+  delete process.env.NEMOCLAW_REASONING;
+  await expect(configureCompatibleEndpointReasoning()).resolves.toBe("false");
+  expect(process.env.NEMOCLAW_REASONING).toBe("false");
+
   process.env.NEMOCLAW_REASONING = "yes";
   await expect(configureCompatibleEndpointReasoning()).resolves.toBe("true");
   expect(process.env.NEMOCLAW_REASONING).toBe("true");
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard.test.ts` around lines 391 - 410, Add an explicit assertion that
the "unset/default" path is covered: before setting NEMOCLAW_REASONING to "yes",
delete process.env.NEMOCLAW_REASONING (or set it to undefined) and call await
expect(configureCompatibleEndpointReasoning()).resolves.toBe(normalizeReasoningFlag(undefined));
this ensures the test asserts the function's fallback behavior when
NEMOCLAW_REASONING is not present and references the existing
normalizeReasoningFlag and configureCompatibleEndpointReasoning symbols.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard.ts`:
- Around line 1508-1511: The configureCompatibleEndpointReasoning function
currently stores the normalized NEMOCLAW_REASONING in process.env only, causing
stale or missing values across onboarding flows; update it to persist the
normalized value into the selected provider state object (use the same
normalization via normalizeReasoningFlag) instead of relying solely on
process.env, and ensure that when provider selection changes the stored flag is
cleared from that provider state and process.env is restored/updated
accordingly; locate configureCompatibleEndpointReasoning and related calls to
normalizeReasoningFlag and modify the provider selection/restore logic where
provider state is mutated so the flag is saved with the provider, removed on
selection change, and used to seed process.env when a provider is active (also
apply the same change pattern to the other occurrence referenced near the file's
later provider-state handling).

---

Nitpick comments:
In `@test/onboard-selection.test.ts`:
- Around line 2075-2094: The fake curl script written to path.join(fakeBin,
"curl") should record its received arguments so the test can assert probe
behavior: modify the heredoc to append the full "$@" or each processed arg into
a temp log file (e.g., "$outfile.args.log" or a known capture file in the test
dir) before handling -o/url logic, and keep returning the same body/status
logic; then update the test assertions to read that captured args log and
explicitly assert that in reasoning mode the captured args do not include
'/responses' nor the '-N' flag and do include a call to '/chat/completions'.
Ensure you reference the same fakeBin/curl generation code and the test's
reasoning-mode invocation when adding the new assertions.

In `@test/onboard.test.ts`:
- Around line 391-410: Add an explicit assertion that the "unset/default" path
is covered: before setting NEMOCLAW_REASONING to "yes", delete
process.env.NEMOCLAW_REASONING (or set it to undefined) and call await
expect(configureCompatibleEndpointReasoning()).resolves.toBe(normalizeReasoningFlag(undefined));
this ensures the test asserts the function's fallback behavior when
NEMOCLAW_REASONING is not present and references the existing
normalizeReasoningFlag and configureCompatibleEndpointReasoning symbols.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 78ba4585-9297-4c89-b6ae-5ad64ecf87cb

📥 Commits

Reviewing files that changed from the base of the PR and between f1568f6 and 077cb4b.

📒 Files selected for processing (3)
  • src/lib/onboard.ts
  • test/onboard-selection.test.ts
  • test/onboard.test.ts

Comment thread src/lib/onboard.ts Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lib/state/onboard-session.ts (1)

707-717: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Persist compatibleEndpointReasoning in the safe update whitelist.

SessionUpdates adds this field (Line 131), but filterSafeUpdates() never forwards it, so updates passed via markStepComplete() / completeSession() are silently dropped. That breaks both setting and clearing (null) this value during provider transitions.

Suggested fix
 export function filterSafeUpdates(updates: SessionUpdates): Partial<Session> {
   const safe: Partial<Session> = {};
   if (!isObject(updates)) return safe;
   if (typeof updates.sandboxName === "string") safe.sandboxName = updates.sandboxName;
   if (typeof updates.provider === "string") safe.provider = updates.provider;
   if (typeof updates.model === "string") safe.model = updates.model;
   if (typeof updates.endpointUrl === "string") safe.endpointUrl = redactUrl(updates.endpointUrl);
   if (typeof updates.credentialEnv === "string") safe.credentialEnv = updates.credentialEnv;
   if (typeof updates.preferredInferenceApi === "string")
     safe.preferredInferenceApi = updates.preferredInferenceApi;
+  if (typeof updates.compatibleEndpointReasoning === "string") {
+    safe.compatibleEndpointReasoning = updates.compatibleEndpointReasoning;
+  } else if (updates.compatibleEndpointReasoning === null) {
+    safe.compatibleEndpointReasoning = null;
+  }
   if (typeof updates.nimContainer === "string") safe.nimContainer = updates.nimContainer;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/state/onboard-session.ts` around lines 707 - 717, filterSafeUpdates
currently omits the SessionUpdates field compatibleEndpointReasoning so updates
from markStepComplete/completeSession are dropped; update the function
filterSafeUpdates to accept and propagate compatibleEndpointReasoning (including
allowing null to clear it) into the returned Partial<Session> when typeof
updates.compatibleEndpointReasoning is "string" or when
updates.compatibleEndpointReasoning === null, mirroring how other
nullable/optional fields are handled so provider transitions can set and clear
this value.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lib/state/onboard-session.ts`:
- Around line 707-717: filterSafeUpdates currently omits the SessionUpdates
field compatibleEndpointReasoning so updates from
markStepComplete/completeSession are dropped; update the function
filterSafeUpdates to accept and propagate compatibleEndpointReasoning (including
allowing null to clear it) into the returned Partial<Session> when typeof
updates.compatibleEndpointReasoning is "string" or when
updates.compatibleEndpointReasoning === null, mirroring how other
nullable/optional fields are handled so provider transitions can set and clear
this value.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 06d7e88a-98bf-4a0d-89a9-ddd0edb6c09a

📥 Commits

Reviewing files that changed from the base of the PR and between 077cb4b and 66bbe29.

📒 Files selected for processing (4)
  • src/lib/onboard.ts
  • src/lib/state/onboard-session.ts
  • test/onboard-selection.test.ts
  • test/onboard.test.ts
🚧 Files skipped from review as they are similar to previous changes (3)
  • test/onboard-selection.test.ts
  • test/onboard.test.ts
  • src/lib/onboard.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/lib/state/onboard-session.ts (1)

145-166: ⚡ Quick win

Include compatibleEndpointReasoning in debug summary output.

The new persisted field is not exposed by DebugSessionSummary/summarizeForDebug, which makes reasoning-mode resume issues harder to diagnose from debug dumps.

Proposed patch
 export interface DebugSessionSummary {
   version: number;
   sessionId: string;
   status: string;
   resumable: boolean;
   mode: string;
   startedAt: string;
   updatedAt: string;
   sandboxName: string | null;
   provider: string | null;
   model: string | null;
   endpointUrl: string | null;
   credentialEnv: string | null;
   preferredInferenceApi: string | null;
+  compatibleEndpointReasoning: string | null;
   nimContainer: string | null;
   policyPresets: string[] | null;
   gpuPassthrough: boolean;
   lastStepStarted: string | null;
   lastCompletedStep: string | null;
   failure: SessionFailure | null;
   steps: Record<string, StepState>;
 }
@@
     endpointUrl: redactUrl(session.endpointUrl),
     credentialEnv: session.credentialEnv,
     preferredInferenceApi: session.preferredInferenceApi,
+    compatibleEndpointReasoning: session.compatibleEndpointReasoning,
     nimContainer: session.nimContainer,
     policyPresets: session.policyPresets,

Also applies to: 852-867

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/state/onboard-session.ts` around lines 145 - 166, DebugSessionSummary
and the summarizeForDebug output are missing the new persisted field
compatibleEndpointReasoning, so add compatibleEndpointReasoning: string | null
(matching the persisted type) to the DebugSessionSummary interface and update
the summarizeForDebug function to read the session-compatibleEndpointReasoning
value and include it in the returned summary object (alongside fields like
failure, steps, nimContainer, preferredInferenceApi, etc.) so debug dumps expose
reasoning-mode compatibility; update any related spots mentioned (e.g., the
summarizeForDebug implementation around the lastStep/lastCompletedStep logic) to
populate this field.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/state/onboard-session.ts`:
- Around line 145-166: DebugSessionSummary and the summarizeForDebug output are
missing the new persisted field compatibleEndpointReasoning, so add
compatibleEndpointReasoning: string | null (matching the persisted type) to the
DebugSessionSummary interface and update the summarizeForDebug function to read
the session-compatibleEndpointReasoning value and include it in the returned
summary object (alongside fields like failure, steps, nimContainer,
preferredInferenceApi, etc.) so debug dumps expose reasoning-mode compatibility;
update any related spots mentioned (e.g., the summarizeForDebug implementation
around the lastStep/lastCompletedStep logic) to populate this field.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f8e80300-8c3d-41f3-be29-8da2e7314cd8

📥 Commits

Reviewing files that changed from the base of the PR and between 66bbe29 and 2c1672b.

📒 Files selected for processing (1)
  • src/lib/state/onboard-session.ts

@wscurran wscurran added enhancement: provider provider: openai OpenAI API or OpenAI-compatible provider behavior and removed v0.0.39 labels May 11, 2026
@wscurran

Copy link
Copy Markdown
Contributor

✨ Thanks for submitting this detailed PR about supporting reasoning-compatible endpoints for custom OpenAI providers. This change aims to improve the compatibility of NemoClaw with reasoning-mode validation through the NEMOCLAW_REASONING flag.


Related open issues:

@deepujain

Copy link
Copy Markdown
Contributor Author

Rebased on current main and trimmed the accidental test-file expansion so the diff stays focused. build:cli plus focused reasoning/session/provider tests pass.

@deepujain

Copy link
Copy Markdown
Contributor Author

Follow-up: moved the reasoning-mode helpers under src/lib/onboard/ so the top-level onboard entrypoint is net-neutral for the growth guard. build:cli and the focused reasoning/session/provider tests pass.

@wscurran wscurran added area: inference Inference routing, serving, model selection, or outputs area: providers Inference provider integrations and provider behavior bug-fix PR fixes a bug or regression feature PR adds or expands user-visible functionality needs: rebase PR needs rebase or conflict resolution and removed enhancement: provider labels Jun 3, 2026
@deepujain deepujain force-pushed the fix/3279-compatible-reasoning-flag branch from f1a0314 to 17b9109 Compare June 4, 2026 19:40
@deepujain

Copy link
Copy Markdown
Contributor Author

Rebased onto current main and kept the onboard entrypoint net-neutral. build:cli plus the focused reasoning, session, provider, and onboard-selection tests pass.

@wscurran wscurran removed the feature PR adds or expands user-visible functionality label Jun 9, 2026
deepujain added 2 commits June 8, 2026 21:36
Fixes NVIDIA#3279

Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
@deepujain deepujain force-pushed the fix/3279-compatible-reasoning-flag branch from 17b9109 to a33d3d8 Compare June 9, 2026 04:42
@deepujain

Copy link
Copy Markdown
Contributor Author

Rebased on current main, re-signed the commit stack, and cleaned up the PR body DCO sign-off. Build, typecheck, and focused reasoning/session tests pass locally. Ready for another look.

@deepujain deepujain force-pushed the fix/3279-compatible-reasoning-flag branch from a33d3d8 to 8a84fb3 Compare June 9, 2026 04:45
@deepujain

Copy link
Copy Markdown
Contributor Author

Follow-up: fixed the growth guard by keeping the top-level onboard entrypoint net-negative against main. Build, typecheck, and the focused reasoning/session tests still pass locally.

Signed-off-by: Deepak Jain <deepujain@gmail.com>
@deepujain deepujain force-pushed the fix/3279-compatible-reasoning-flag branch from 8a84fb3 to 0a86ee8 Compare June 9, 2026 04:48
@deepujain

Copy link
Copy Markdown
Contributor Author

One more cleanup: moved the reasoning selection coverage into a focused test file so the legacy test-size budget stays at its limit. Build, typecheck, and the focused tests pass locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: inference Inference routing, serving, model selection, or outputs area: providers Inference provider integrations and provider behavior bug-fix PR fixes a bug or regression needs: rebase PR needs rebase or conflict resolution provider: openai OpenAI API or OpenAI-compatible provider behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NEMOCLAW_REASONING not configurable for Option 3 providers; reasoning-only models fail silently

4 participants