docs: search agent dev note by dhruvnathawani · Pull Request #350 · NVIDIA-NeMo/DataDesigner

dhruvnathawani · 2026-02-23T19:39:49Z

Summary

Add a dev note documenting the search agent SFT data pipeline used to generate tool-use trajectories for training Nemotron Super's web browsing capabilities.

What's in the post

Motivation: Why training search agents requires trajectory data capturing the full thought-action-observation loop, not just QA pairs. BrowseComp benchmark context.
Pipeline walkthrough: Wikidata KG random walks (50k seeds) -> two-stage riddle generation (draft + BrowseComp-style obfuscation) via Data Designer LLM columns -> search trajectory rollouts with live Tavily web search via MCP -> post-processing to SFT-ready JSONL
ASCII pipeline diagram showing the 4-stage flow (seed data -> riddle generation -> trajectory rollouts -> SFT dataset)
Example Wikidata paths (NVIDIA -> Jensen Huang -> Oregon State -> Benton County -> Thomas Hart Benton) with draft-to-obfuscated question transformation
Seed filtering heuristics: anti-meta filters, hop range constraints (4-8), generic entity removal
Full trajectory example in OpenAI-messages format showing system prompt, tool calls, tool responses, and final answer
Production yield analysis: 50k seeds -> 37k valid seeds (74%) -> 24k valid questions (65%) -> 7k valid trajectories (29%), 14% end-to-end yield
Correctness challenge: multi-answer validity, stale Wikidata ground truth (U.S. Steel/Nippon Steel example), 27.5% raw accuracy before filtering
Data Designer MCP integration walkthrough: LocalStdioMCPProvider, ToolConfig (allowlists, turn budgets, timeouts), tool_alias + with_trace=TraceType.ALL_MESSAGES for full conversation capture
Collapsible full source script with Tavily MCP server, 3-column DAG (draft -> obfuscated -> agent trajectory), and trace capture
Next steps: scale to 25k questions, push difficulty higher, explore fresher knowledge bases, search RL environment

Files changed

docs/devnotes/posts/search-agent.md (updated with code dropdown)
docs/devnotes/.authors.yml (pulled latest from main with dnathawani)
mkdocs.yml (added Search Agent to Dev Notes nav)

greptile-apps · 2026-02-26T20:32:23Z

Greptile Summary

This PR adds a new dev note documenting the search agent SFT data pipeline used to generate tool-use trajectories for training Nemotron Super's web browsing capabilities, along with two supporting images and a nav entry in mkdocs.yml.

The post is thorough and well-written, covering motivation, the 4-stage pipeline (Wikidata KG walks → riddle generation → trajectory rollouts → SFT post-processing), yield analysis, correctness challenges, and a Data Designer MCP integration walkthrough. Previously flagged issues from earlier review rounds have been largely resolved — the wikidata-graph-walk.png image and search_agent.py asset file are both now present, the trajectory example acknowledges the simplified OpenAI-spec format explicitly, and the yield numbers are self-consistent.

Key changes:

docs/devnotes/posts/search-agent.md — new 587-line dev note with code snippets, pipeline diagrams, benchmark results, and a collapsible full-source recipe
docs/devnotes/posts/images/browsecomp-benchmark-results.jpg and wikidata-graph-walk.png — supporting images referenced from the post
mkdocs.yml — "Search Agent" added to the Dev Notes nav with correct most-recent-first ordering

Confidence Score: 5/5

This is a documentation-only PR with no runtime logic changes; it is safe to merge.
All changes are documentation, images, and a nav entry. Previously flagged blockers (missing image, missing script file) have been resolved. The only remaining issue is a cosmetic double-slash in one URL, which does not prevent the build from succeeding.
No files require special attention beyond the minor URL typo in docs/devnotes/posts/search-agent.md line 585.

Important Files Changed

Filename	Overview
docs/devnotes/posts/search-agent.md	New dev note documenting the search agent SFT pipeline; comprehensive and well-structured. Previously flagged issues (missing image, missing script file, OpenAI-spec clarification, yield numbers) have been resolved. One minor typo remains: a double slash in the GTC 2026 URL.
mkdocs.yml	Adds "Search Agent" to the Dev Notes nav and reorders RQA Dataset below Design Principles, with an explanatory comment about most-recent-first ordering. Change is minimal and correct.
docs/devnotes/posts/images/browsecomp-benchmark-results.jpg	New benchmark results image referenced from search-agent.md; no issues.
docs/devnotes/posts/images/wikidata-graph-walk.png	New Wikidata graph walk illustration referenced from search-agent.md; previously flagged as missing, now included. No issues.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["🌐 Wikidata Knowledge Graph\n(100M+ entities)"] -->|SPARQL random walks\n4–8 hops, anti-meta filters| B["Stage 1: Seed Data\n50,000 JSONL records\nseed_entity → final_answer_entity"]

    B -->|74% pass filter| C["Stage 2a: Draft Riddle\nLLM — chain clues,\nhide intermediate nodes"]
    C -->|65% valid| D["Stage 2b: BrowseComp Obfuscation\nLLM — concise, no breadcrumbs,\n1–2 sentences"]

    D -->|24,000 valid questions| E["Stage 3: Trajectory Rollouts\nMiniMax-M2 + Tavily MCP\nThought → Tool Call → Observation loop\n~12 tool calls/sample"]

    E -->|29% correct & complete| F["Stage 4: Post-Processing\nNormalize tool outputs\nDrop truncated\nSelect best rollout\nWrite OpenAI-messages JSONL"]

    F --> G["~7,000 SFT Records\n14% end-to-end yield\nNemotron Super 0% → 31.28% BrowseComp"]

    style A fill:#1a1a2e,color:#eee
    style B fill:#16213e,color:#eee
    style C fill:#16213e,color:#eee
    style D fill:#16213e,color:#eee
    style E fill:#0f3460,color:#eee
    style F fill:#0f3460,color:#eee
    style G fill:#533483,color:#eee

Prompt To Fix All With AI

This is a comment left during a code review.
Path: docs/devnotes/posts/search-agent.md
Line: 585

Comment:
**Double slash in GTC 2026 URL**

The URL contains a double slash after the domain (`nvidia.com//gtc/`), which is a typo. Most servers will redirect it, but it looks unprofessional in published documentation and may break for some link validators or users on strict proxies.

```suggestion
6. [GTC 2026 Workshop: Building Search Agents with NeMo Data Designer](https://www.nvidia.com/gtc/session-catalog/sessions/gtc26-dlit81572/)
```

How can I resolve this? If you propose a fix, please make it concise.

_{Last reviewed commit: ecec9a7}

- Update date to 2026-03-11 (mvansegbroeck) - Rewrite intro: keep agentic shift opener, replace middle paragraphs with BrowseComp-first framing per mvansegbroeck suggestion - Fix trajectory example: "hour-angle coordinate system" is incorrect, changed to "first equatorial coordinate system" (mvansegbroeck) - Move stale Wikidata ground truth discussion earlier into Step 1 as motivation, with forward link to Correctness Challenge (mvansegbroeck) - Move safety controls from Key Takeaway #7 into MCP Integration section where it fits better (mvansegbroeck) - Add GTC 2026 workshop link to Key Resources (mvansegbroeck) - Replace outdated Try For Yourself code (LocalStdioMCPProvider, Qwen model, no Stage 4) with minimal inline snippet + full recipe include using hosted Tavily MCP endpoint (same pattern as text-to-sql)

PR feedback: - Update date to 2026-03-12 - Rewrite intro: keep agentic shift opener, replace middle paragraphs with BrowseComp-first framing - Fix trajectory example: "hour-angle coordinate system" -> "first equatorial coordinate system" - Move stale Wikidata ground truth into Step 1 as early motivation - Move safety controls from Key Takeaway #7 into MCP Integration section - Add GTC 2026 workshop link to Key Resources New content: - Add Wikidata graph walk diagram (wikidata-graph-walk.png) to Step 1 - Add BrowseComp Benchmark Results section with bar chart (JPG): Nemotron Super 0% -> 31.28% (SFT + RL), vs GPT-OSS-120B 33.89% - Replace outdated Try For Yourself code with minimal inline snippet + full recipe include via pymdownx.snippets

dhruvnathawani · 2026-03-12T03:40:19Z

greptile is just wrong in this case with it's suggestions

Co-authored-by: Johnny Greco <jogreco@nvidia.com>

…IA-NeMo/DataDesigner into dhruv/devnotes/search-agent

johnnygreco

This is great, @dhruvnathawani !

Co-authored-by: Johnny Greco <jogreco@nvidia.com>

johnnygreco

🙌

dhruvnathawani added 2 commits February 23, 2026 11:28

search agent dev notes

877d7ee

docs: add code dropdown and mkdocs nav for search agent devnote

c387432

dhruvnathawani changed the title ~~search agent dev notes~~ docs: search agent dev note Feb 26, 2026

Merge branch 'main' into dhruv/devnotes/search-agent

0620a60

dhruvnathawani marked this pull request as ready for review February 26, 2026 20:30

dhruvnathawani requested a review from a team as a code owner February 26, 2026 20:30

dhruvnathawani requested review from kirit93 and mvansegbroeck March 9, 2026 19:57

mvansegbroeck reviewed Mar 11, 2026

View reviewed changes

Comment thread docs/devnotes/posts/search-agent.md Outdated

mvansegbroeck reviewed Mar 11, 2026

View reviewed changes

Comment thread docs/devnotes/posts/search-agent.md

mvansegbroeck reviewed Mar 11, 2026

View reviewed changes

Comment thread docs/devnotes/posts/search-agent.md Outdated

Comment thread docs/devnotes/posts/search-agent.md Outdated

Comment thread docs/devnotes/posts/search-agent.md

Comment thread docs/devnotes/posts/search-agent.md Outdated

dhruvnathawani added 2 commits March 11, 2026 19:35

Merge branch 'main' into dhruv/devnotes/search-agent

c4f2726

resolve merge conflict

e985a7c

greptile-apps Bot reviewed Mar 12, 2026

View reviewed changes

Comment thread docs/devnotes/posts/search-agent.md

Comment thread docs/devnotes/posts/search-agent.md

Comment thread docs/devnotes/posts/search-agent.md

greptile-apps Bot reviewed Mar 12, 2026

View reviewed changes

Comment thread docs/devnotes/posts/search-agent.md

Comment thread docs/devnotes/posts/search-agent.md

greptile-apps Bot reviewed Mar 12, 2026

View reviewed changes

Comment thread docs/devnotes/posts/search-agent.md

dhruvnathawani added 4 commits March 11, 2026 20:17

add search agent dev note images

2e25f0e

remove image

e70a9b2

changes

a01c603

address pr comments

0307af1

greptile-apps Bot reviewed Mar 12, 2026

View reviewed changes

Comment thread docs/devnotes/posts/search-agent.md

address pr comments

c849c56

greptile-apps Bot reviewed Mar 12, 2026

View reviewed changes

Comment thread docs/devnotes/posts/search-agent.md Outdated

Comment thread docs/devnotes/posts/search-agent.md

mvansegbroeck previously approved these changes Mar 12, 2026

View reviewed changes