Greptile Summary

This PR adds a new devnote documenting the prompt sensitivity SDG pipeline — a Data Designer workflow that generates diverse MCQ preambles and format instructions to reduce LLM brittleness to prompt phrasing, used in Nemotron training mixtures for both SFT and RL.

• New post published in both MkDocs (docs/devnotes/posts/prompt-sensitivity.md) and Fern (fern/versions/latest/pages/devnotes/posts/prompt-sensitivity.mdx), with navigation wired up in mkdocs.yml and fern/versions/latest.yml.
• The double_asterisks regex template (\\*\\*([A-Za-z])\\*\\*) has a first-match extraction problem: MCQ chain-of-thought responses routinely bold individual option letters in reasoning text, causing re.search() to return the first such match rather than the intended final-answer marker — the same failure mode as the angle_brackets pattern that was already fixed in this PR.

Important Files Changed

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["STEP 1: Seed Examples\n10 format templates × 5 preamble anchors = 50 rows\nDataFrameSeedSource + SamplingStrategy.SHUFFLE"] --> B
    B["STEP 2: Diversity Samplers\nsentence_type(3) × tone(5) × strictness(3)\n× verbosity(3) × domain(3) × order(8) = 3,240"] --> C
    C["STEP 3: LLM Generation Columns\npreamble | format_instruction | user_prompt"] --> D
    D["STEP 4: Quality Judges\nformat_compliance + regex_alignment + order_coherence (binary)\npreamble_quality 0–3 rubric\n~15–20% fail at least one gate"] --> E
    E["STEP 5: Training Mixture Integration\nYAML-driven: majority_preamble 25% canonical\n75% diverse variations → 1M packed records"]

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
docs/devnotes/posts/prompt-sensitivity.md:481
**`double_asterisks` regex has the same first-match problem as the previously fixed `angle_brackets` pattern**

`re.search(r"\*\*([A-Za-z])\*\*", response)` returns the *first* `**X**` in the response. In a typical MCQ chain-of-thought, the model discusses each option by letter — e.g. "**A** is incorrect because…, **B** also fails…, therefore **C**" — so `re.search` extracts `A` rather than the intended final answer `**C**`. The fix applied to `angle_brackets` (switching from `[A-Za-z]` to `[A-Z]`) reduces false positives from lowercase emphasis but doesn't address the positional issue for uppercase letters; using `re.findall()` and taking the last match, or anchoring the search to the final line, is the reliable fix for RL reward extraction with this format.

### Issue 2 of 2
fern/versions/latest/pages/devnotes/posts/prompt-sensitivity.mdx:168
**`double_asterisks` regex has the same first-match problem as the previously fixed `angle_brackets` pattern**

Same issue as in the `.md` version: `re.search(r"\*\*([A-Za-z])\*\*", response)` returns the first bolded single letter in the response. MCQ chain-of-thought reasoning commonly bolds option labels ("**A** is wrong, **B** is also wrong, **C** is correct"), so the first match will be a discarded option rather than the final answer. Using `re.findall()[-1]` or anchoring to the last line avoids this false extraction.

_{Reviews (9): Last reviewed commit: "docs(fern): recompress prompt sensitivit..." | Re-trigger Greptile}

MkDocs preview: https://69eed809.dd-docs-preview.pages.dev

Fern preview: https://nvidia-preview-pr-351.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.

Review: PR #351 — docs: add prompt sensitivity devnote

Summary

Docs-only PR. Adds a new MkDocs devnote (docs/devnotes/posts/prompt-sensitivity.md, 594 lines) plus an inline image (prompt_anatomy.png), and registers the post in mkdocs.yml. The post explains the prompt-sensitivity SDG pipeline used for Nemotron training data: motivation, anatomy diagram, six diversity samplers, dual judges, regex-paired format templates, training-mixture YAML, and a collapsible full-source script.

API symbols cited in the code blocks (DataDesignerConfigBuilder, DataFrameSeedSource, LLMTextColumnConfig, LLMJudgeColumnConfig, Score, SamplerColumnConfig, CategorySamplerParams, ModelConfig, SamplingStrategy) all exist in packages/data-designer-config/. Heading style (# **Title**, ## **Section**) matches the other devnotes in docs/devnotes/posts/.

Findings

1. Inconsistency between narrative pipeline and the "full source" script (medium)

The body of the post and the appendix script describe different pipelines, but the post does not flag this. Readers who scroll through Step 2 and then expand "Try For Yourself" will see two non-matching configurations.

• Step 2 narrative (around line 196 onward) defines six samplers including answer_format with 3 values: \boxed{}, \boxed{LETTER}, \boxed{<letter>}. The diagram on line 124 advertises the combinatorial space as 3 × 5 × 3 × 3 × 3 × 3 = 1,215.
• The full source (line 481 onward) replaces answer_format with preamble_format_order (8 values for component ordering). Format diversity in the full source comes from the cross-product seed data (MCQ_FORMAT_TEMPLATES × SEED_PREAMBLES), not from a sampler. There is no answer_format sampler in the full source.
• That means the actual combinatorial space of the full source is 3 × 5 × 3 × 3 × 3 × 8 = 3,240, sampled jointly with 50 seed rows — and the "1,215 combinations" claim in the diagram is for a different pipeline than the one shipped in the appendix.

Recommendation: either (a) align the narrative samplers to the full source, or (b) explicitly note that the narrative is a simplified didactic pipeline and the appendix is the production version, calling out the differences (cross-product seeds, the placement-order sampler).

2. "25+ format templates" claim vs. 10 templates in the source (low–medium)

Multiple places in the post advertise "25+ distinct format templates":

• Line 384: "We curated 25+ distinct format templates spanning ..."
• PR description: "25+ answer formats (boxed, brackets, XML tags, asterisks, arrows, etc.)"

The full source MCQ_FORMAT_TEMPLATES list (lines 421–452) contains only 10 entries. If the production templates list really is 25+, consider adding the rest (or trimming to a representative subset and softening the claim to e.g. "10 representative templates from a 25+ catalogue").

3. Unrelated nav reordering in mkdocs.yml (low)

The diff also moves Deep Research Trajectories from before Design Principles to after RQA Dataset:

-      - Deep Research Trajectories: devnotes/posts/deep-research-trajectories.md
       - Design Principles: devnotes/posts/design-principles.md
       - RQA Dataset: devnotes/posts/rqa.md
+      - Deep Research Trajectories: devnotes/posts/deep-research-trajectories.md
+      - Prompt Sensitivity: devnotes/posts/prompt-sensitivity.md

The PR description doesn't mention this reordering, only the new file. If the intent is purely to keep newest posts at the bottom (Deep Research is dated 2026-02-10 and the new post is 2026-02-18, both newer than Design Principles), that's fine — but worth mentioning in the PR description to avoid looking like an accident. Otherwise, consider reverting the move.

4. Step 1 walkthrough understates what seed_data actually is (low)

Step 1 shows a simple pd.DataFrame of 5 preambles loaded with SamplingStrategy.SHUFFLE. The full source instead constructs a cross-product DataFrame of 50 rows (5 preambles × 10 format templates), which is the basis for the regex-pairing trick described later. The walkthrough never reconciles this — readers might miss that the seed dataset itself is doing meaningful structural work in the production pipeline. A one-line bridge ("In production we cross-product these with format templates — see the appendix") would close the gap.

5. Minor

• Line 8 frontmatter date 2026-02-18 is consistent with the previous devnote (2026-02-10); fine.
• Line 33 image is referenced as ![Prompt Anatomy](prompt_anatomy.png) — relative path next to the post; matches MkDocs Material conventions used elsewhere in docs/devnotes/posts/.
• Line 600 closes with a sentence pointing to the GitHub repo as "documentation," which is technically inaccurate (the docs site is at nvidia-nemo.github.io/DataDesigner/). Worth fixing the link target.
• The two judges shown in Step 3 (format_compliance, preamble_quality) are described as "dual" judges, but the appendix script actually defines four judges (format_compliance, regex_alignment, order_coherence, preamble_quality). The pipeline diagram on line 137 also says "DUAL QUALITY JUDGES" but lists only two. Either rename the section ("Quality judges") or update the count to 4 to match the source.
• The post mentions add_prompt_variations.py (line 316) but this script isn't in this PR or referenced from anywhere in the repo. If it's an internal/external script, consider adding a footnote so readers don't go searching for it.

Verdict

Docs-only, low-risk merge. No blocking issues, but I'd recommend addressing the narrative-vs-appendix mismatches before merging — particularly (1) the "1,215 combinations" claim that doesn't match the full source, (2) the "25+ templates" claim that doesn't match the 10 in the script, and (5) the "dual judges" framing when the appendix has four. These are the kind of inconsistencies a careful reader will flag, and they're cheap to fix.

Thanks @dhruvnathawani! Can you ask your agent to copy the post to the Fern documentation as well? We are getting ready to migrate and for the moment need to write the docs in both places.

Fern

Done — ported the post to fern/versions/latest/pages/devnotes/posts/prompt-sensitivity.mdx and registered it at the top of the Dev Notes section in fern/versions/latest.yml. Both MkDocs and Fern previews should pick it up, let me know if this looks okay

Want your agent to iterate on Greptile's feedback? Try greploops.