Skip to content

fix: strengthen prd.json schema instructions to prevent AI hallucination#142

Merged
subsy merged 2 commits intosubsy:mainfrom
anthonymq:fix/prd-json-schema-hallucination
Jan 18, 2026
Merged

fix: strengthen prd.json schema instructions to prevent AI hallucination#142
subsy merged 2 commits intosubsy:mainfrom
anthonymq:fix/prd-json-schema-hallucination

Conversation

@anthonymq
Copy link
Copy Markdown
Contributor

@anthonymq anthonymq commented Jan 18, 2026

Summary

Fixes #141

The AI agent was generating invalid prd.json files with nested structures (e.g., wrapping everything in a "prd" object) instead of the flat schema expected by the JSON tracker.

Changes

  • PrdChatApp.tsx: Added explicit WRONG examples and stronger "DO NOT USE" guidance in the skillPrompt
  • ralph-tui-create-json/SKILL.md: Added comprehensive anti-patterns section with visual ❌ WRONG and ✅ CORRECT examples

Problem

When using ralph-tui create-prd with chat mode, the AI agent would sometimes generate:

{
  "prd": {
    "metadata": {...},
    "overview": {...},
    ...
  }
}

Instead of the correct flat structure:

{
  "name": "Feature Name",
  "userStories": [...]
}

Solution

The validation in json.ts already correctly rejects these invalid schemas, but this fix prevents the AI from generating them in the first place by:

  1. Adding explicit examples of WRONG patterns in the skill prompt
  2. Using visual markers (❌ ✅) to make correct/incorrect patterns obvious
  3. Listing specific forbidden fields and patterns

Testing

  • bun run typecheck passes
  • bun run build succeeds

Summary by CodeRabbit

Documentation

  • Clarified PRD document structure requirements with improved guidance and detailed specifications
  • Added comprehensive validation rules and compliance guidelines to prevent errors
  • Enhanced instructional examples demonstrating proper formatting and structure
  • Documented common mistakes and forbidden patterns with clear prevention guidance
  • Improved field requirement documentation for better user understanding and compliance

✏️ Tip: You can customize this high-level summary in your review settings.

@vercel
Copy link
Copy Markdown

vercel bot commented Jan 18, 2026

@anthonymq is attempting to deploy a commit to the plgeek Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Jan 18, 2026

Walkthrough

The PR strengthens JSON schema validation guidance across documentation and code by mandating a flat root-level structure with "name" field, explicitly prohibiting invalid patterns like "prd" wrapper objects and "tasks" arrays, and providing comprehensive schema requirements and anti-patterns to prevent AI hallucination of invalid schemas.

Changes

Cohort / File(s) Summary
Schema Documentation and Guidance
skills/ralph-tui-create-json/SKILL.md, src/tui/components/PrdChatApp.tsx
Updated documentation and prompt instructions to enforce flat root JSON structure with explicit "name" field. Added CRITICAL sections detailing forbidden patterns (wrapper objects, "tasks" arrays, nested structures), required fields, validation rules, and anti-pattern examples. Removed references to "project" field at root and "prd" wrapper objects. Enhanced tracker prompt with comprehensive schema requirements and "DO NOT USE" examples.

Estimated Code Review Effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly Related PRs

Poem

🐰 Hop, hop! The schema springs so flat,
No nested "prd" wrappers—mind that!
With "name" at root and stories true,
Invalid patterns we bid adieu!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: strengthening prd.json schema instructions to prevent AI hallucination, which aligns with the primary objective of the PR.
Linked Issues check ✅ Passed The PR fully addresses all coding requirements from issue #141: strengthening schema instructions in PrdChatApp.tsx with explicit DO NOT examples, adding anti-hallucination sections to SKILL.md with WRONG vs CORRECT examples, and maintaining existing validation logic.
Out of Scope Changes check ✅ Passed All changes are directly scoped to issue #141: documentation updates in SKILL.md and prompt guidance enhancements in PrdChatApp.tsx to prevent AI schema hallucination. No unrelated modifications detected.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 markdownlint-cli2 (0.18.1)
skills/ralph-tui-create-json/SKILL.md

markdownlint-cli2 v0.18.1 (markdownlint v0.38.0)
Finding: skills/ralph-tui-create-json/SKILL.md
Linting: 1 file(s)
Summary: 5 error(s)
Error: EACCES: permission denied, open '/markdownlint-cli2-results.json'
at async open (node:internal/fs/promises:640:25)
at async Object.writeFile (node:internal/fs/promises:1214:14)
at async Promise.all (index 0)
at async outputSummary (file:///usr/local/lib/node_modules/markdownlint-cli2/markdownlint-cli2.mjs:877:5)
at async main (file:///usr/local/lib/node_modules/markdownlint-cli2/markdownlint-cli2.mjs:1053:25)
at async file:///usr/local/lib/node_modules/markdownlint-cli2/markdownlint-cli2-bin.mjs:12:22 {
errno: -13,
code: 'EACCES',
syscall: 'open',
path: '/markdownlint-cli2-results.json'
}


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Fixes subsy#141

The AI agent was generating invalid prd.json files with nested structures
(e.g., wrapping everything in a 'prd' object) instead of the flat schema
expected by the JSON tracker.

Changes:
- PrdChatApp.tsx: Added explicit WRONG examples and stronger DO NOT USE
  guidance in the skillPrompt
- ralph-tui-create-json/SKILL.md: Added comprehensive anti-patterns section
  with visual ❌ WRONG and ✅ CORRECT examples

The validation in json.ts already correctly rejects these invalid schemas,
but this fix prevents the AI from generating them in the first place.
@codecov
Copy link
Copy Markdown

codecov bot commented Jan 18, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.11%. Comparing base (25f8a38) to head (3f05020).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #142   +/-   ##
=======================================
  Coverage   44.11%   44.11%           
=======================================
  Files          62       62           
  Lines       15716    15716           
=======================================
  Hits         6933     6933           
  Misses       8783     8783           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@anthonymq anthonymq force-pushed the fix/prd-json-schema-hallucination branch from 311d19f to 0258339 Compare January 18, 2026 11:53
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
skills/ralph-tui-create-json/SKILL.md (1)

300-356: Fix field name inconsistency in example.

The example uses "project" field (line 303), but the schema definition requires "name" field (line 56, line 12). This inconsistency contradicts the updated schema and could confuse users or the AI agent.

📝 Proposed fix
 {
-  "project": "my-app",
+  "name": "my-app",
   "branchName": "ralph/task-priority",
   "description": "Add priority levels to tasks",

@vercel
Copy link
Copy Markdown

vercel bot commented Jan 18, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
ralph-tui Ready Ready Preview, Comment Jan 18, 2026 2:39pm

Request Review

@subsy
Copy link
Copy Markdown
Owner

subsy commented Jan 18, 2026

@anthonymq thanks - nice clean PR - good enhancement!

@subsy subsy merged commit b57bcee into subsy:main Jan 18, 2026
9 checks passed
sakaman pushed a commit to sakaman/ralph-tui that referenced this pull request Feb 15, 2026
…cination

fix: strengthen prd.json schema instructions to prevent AI hallucination
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI agent generates invalid prd.json schema with nested 'prd' wrapper

2 participants