fix: validate prd.json schema and improve AI prompt to prevent hallucinated formats#12
Conversation
…inated formats - Add validatePrdJsonSchema() with specific detection for common AI mistakes - Include explicit schema example and DO NOT list in create-prd prompt - Validate generated JSON before writing in convert command Fixes subsy#11
WalkthroughAdds runtime PRD JSON schema validation with structured errors, integrates pre-write validation into the convert command to avoid saving invalid prd.json, and tightens the TUI AI prompt by embedding an explicit PRD JSON schema and strict output instructions. Changes
Sequence Diagram(s)sequenceDiagram
actor User
participant ConvertCmd as convert.ts
participant Validator as validatePrdJsonSchema
participant FileSystem as File System
participant ErrorOutput as Error Output
User->>ConvertCmd: run convert --to json
ConvertCmd->>ConvertCmd: generate prdJson
ConvertCmd->>Validator: validatePrdJsonSchema(prdJson, outputPath)
alt Schema valid
Validator-->>ConvertCmd: validated PrdJson
ConvertCmd->>FileSystem: create output dir & write prd.json
FileSystem-->>ConvertCmd: write success
ConvertCmd-->>User: exit success
else Schema invalid
Validator-->>ConvertCmd: PrdJsonSchemaError (details + suggestion)
ConvertCmd->>ErrorOutput: print structured error + fix suggestion
ConvertCmd-->>User: exit with code 1
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In @src/plugins/trackers/builtin/json.ts:
- Around line 147-151: The current assignment const name = (obj.name as string)
|| (obj.project as string) incorrectly treats falsy but present values (e.g.,
"") as missing; change the logic in the name resolution to first check property
existence (e.g., Object.prototype.hasOwnProperty.call(obj, 'name')) and then
verify typeof obj.name === 'string' to use it, otherwise check for 'project'
similarly; update the subsequent error push to trigger only when neither
property exists as a string.
🧹 Nitpick comments (3)
src/plugins/trackers/builtin/json.ts (2)
403-414: Consider extracting duplicated error handling logic.The
PrdJsonSchemaErrorhandling block is duplicated in bothgetTasks(lines 403-412) andgetEpics(lines 615-624). This could be extracted to a private helper method for consistency and maintainability.♻️ Suggested refactor
private logPrdSchemaError(err: PrdJsonSchemaError): void { console.error(`\n${err.message}\n`); console.error('Issues found:'); for (const detail of err.details) { console.error(` - ${detail}`); } console.error(`\nHow to fix:\n${err.suggestion}\n`); }Then use in both methods:
} catch (err) { if (err instanceof PrdJsonSchemaError) { - console.error(`\n${err.message}\n`); - console.error('Issues found:'); - for (const detail of err.details) { - console.error(` - ${detail}`); - } - console.error(`\nHow to fix:\n${err.suggestion}\n`); + this.logPrdSchemaError(err); } else {Also applies to: 615-626
84-93: Optional: Consider adding prototype chain safeguard for custom Error subclass.With your current ES2022 transpilation target, the
instanceofchecks forPrdJsonSchemaErrorwork correctly without additional fixes. However, if the codebase is transpiled to older JavaScript versions (ES5 or early ES2015), the prototype chain may not be correctly set, causinginstanceofchecks to fail. AddingObject.setPrototypeOfis a defensive measure that remains harmless in modern environments while future-proofing against transpilation target changes.🔧 Optional safeguard
export class PrdJsonSchemaError extends Error { constructor( message: string, public readonly details: string[], public readonly suggestion: string ) { super(message); this.name = 'PrdJsonSchemaError'; + Object.setPrototypeOf(this, PrdJsonSchemaError.prototype); } }src/commands/convert.ts (1)
508-520: Validation placement is correct, but consider the error message context.The pre-write validation is a good safety net. However, if validation fails here, it indicates a bug in the internal
convertToPrdJsonfunction (since the PRD parser should produce valid structures). The error message fromPrdJsonSchemaError.suggestionmentions "regenerate with convert command", which would be confusing since this is the convert command.Consider adding context-specific messaging for this case:
🔧 Suggested improvement
try { validatePrdJsonSchema(prdJson, outputPath); } catch (err) { if (err instanceof PrdJsonSchemaError) { - printError('Generated prd.json failed schema validation:'); + printError('Internal error: Generated prd.json failed schema validation.'); + printError('This indicates a bug in the PRD parser. Please report this issue.'); for (const detail of err.details) { console.error(` - ${detail}`); } - console.error(`\n${err.suggestion}`); + // Don't print the generic suggestion as it refers to re-running this command process.exit(1); } throw err; }
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/commands/convert.tssrc/plugins/trackers/builtin/json.tssrc/tui/components/PrdChatApp.tsx
🧰 Additional context used
🧬 Code graph analysis (1)
src/commands/convert.ts (1)
src/plugins/trackers/builtin/json.ts (2)
validatePrdJsonSchema(636-636)PrdJsonSchemaError(84-93)
🔇 Additional comments (5)
src/tui/components/PrdChatApp.tsx (2)
86-100: LGTM!The JSON schema example is well-structured and consistent with the
PrdJsoninterface defined injson.ts. It correctly demonstrates all required fields (id,title,passes,dependsOn) and optional fields, providing clear guidance for the AI agent.
106-125: LGTM!The skill prompt is comprehensive and well-aligned with the schema validation in
json.ts. The explicit "DO NOT use" list effectively addresses the common AI hallucination patterns identified in the linked issue. The prompt clearly specifies required fields and references the exact schema example.src/plugins/trackers/builtin/json.ts (2)
211-223: LGTM!The normalisation logic correctly handles the
projectalias and ensures a consistentPrdJsonstructure. The defensive fallback forpasseson line 219 is reasonable as a safeguard.
636-636: LGTM!The export enables reuse of
validatePrdJsonSchemainconvert.tsfor pre-write validation, which aligns with the PR objectives.src/commands/convert.ts (1)
22-25: LGTM!The imports correctly reference the newly exported validation utilities from the JSON tracker module.
- Use property existence check for name field to handle empty strings - Update error message for unsupported fields to say 'Remove these fields'
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
src/plugins/trackers/builtin/json.ts (3)
190-198: Minor: Potential duplicate error message forstatusfield.When a story has a
statusfield but nopassesfield, the user receives two error messages: one from thepassesvalidation (lines 180-184) and another from the unsupported fields check here. Consider removing'status'fromunsupportedFieldssince it's already handled with a more specific message above.🔧 Suggested fix
- const unsupportedFields = ['subtasks', 'estimated_hours', 'files', 'status']; + const unsupportedFields = ['subtasks', 'estimated_hours', 'files'];
221-221: Consider validating metadata fields.The
metadataobject is cast directly without validating its internal structure. If an AI hallucinates incorrect types forcreatedAt,updatedAt, orversion(e.g., numbers instead of strings), they would pass through. This is low-risk since metadata is optional, but could cause subtle issues if downstream code assumes string types.🔧 Optional validation
- metadata: obj.metadata as PrdJson['metadata'], + metadata: obj.metadata && typeof obj.metadata === 'object' + ? { + createdAt: typeof (obj.metadata as Record<string, unknown>).createdAt === 'string' + ? (obj.metadata as Record<string, unknown>).createdAt as string + : undefined, + updatedAt: typeof (obj.metadata as Record<string, unknown>).updatedAt === 'string' + ? (obj.metadata as Record<string, unknown>).updatedAt as string + : undefined, + version: typeof (obj.metadata as Record<string, unknown>).version === 'string' + ? (obj.metadata as Record<string, unknown>).version as string + : undefined, + } + : undefined,
403-412: Consider extracting duplicate error handling to a helper.The error formatting logic for
PrdJsonSchemaErroris duplicated in bothgetTasks(lines 403-412) andgetEpics(lines 615-624). If the error format needs to change, both locations must be updated.🔧 Suggested refactor
Add a method to
PrdJsonSchemaErroror a standalone helper:// Option 1: Add to PrdJsonSchemaError class printError(): void { console.error(`\n${this.message}\n`); console.error('Issues found:'); for (const detail of this.details) { console.error(` - ${detail}`); } console.error(`\nHow to fix:\n${this.suggestion}\n`); }Then simplify the catch blocks:
} catch (err) { if (err instanceof PrdJsonSchemaError) { - console.error(`\n${err.message}\n`); - console.error('Issues found:'); - for (const detail of err.details) { - console.error(` - ${detail}`); - } - console.error(`\nHow to fix:\n${err.suggestion}\n`); + err.printError(); } else {Also applies to: 615-624
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/plugins/trackers/builtin/json.ts
🔇 Additional comments (4)
src/plugins/trackers/builtin/json.ts (4)
81-93: LGTM! Well-structured custom error class.The error class properly extends
Error, sets thenameproperty for identification, and exposes structured details and suggestions as readonly properties. This design enables clean error handling downstream.
112-145: Good detection of AI hallucination patterns with actionable guidance.Early detection of common AI-generated schema mistakes (top-level
prdortaskswrappers) with a clear example of the correct structure is helpful. Throwing immediately here prevents confusing cascading validation errors.
354-369: LGTM! Clean integration of schema validation.Validation occurs before caching, ensuring only valid data enters the cache. Letting exceptions propagate to callers (
getTasks,getEpics) is appropriate since they have context-specific error handling.
636-636: LGTM!The named export enables
convert.tsto perform pre-write validation, completing the defence-in-depth strategy described in the PR objectives.
- Add Object.setPrototypeOf safeguard for PrdJsonSchemaError class - Extract duplicated error handling to logPrdSchemaError helper - Fix convert.ts error message to indicate internal bug vs user error
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/plugins/trackers/builtin/json.ts (1)
229-234: Misleading comment and unvalidated metadata cast.Two minor observations:
The comment on line 231 mentions "convert from status if needed as fallback", but status conversion isn't actually performed here—the fallback simply defaults to
false.
obj.metadatais cast directly without structural validation. If the metadata contains unexpected types (e.g., non-string values forcreatedAt), downstream code might encounter type mismatches.Suggested improvements
userStories: (obj.userStories as PrdUserStory[]).map((s) => ({ ...s, - // Ensure passes is a boolean (convert from status if needed as fallback) + // Ensure passes is a boolean (default to false if somehow missing after validation) passes: typeof s.passes === 'boolean' ? s.passes : false, })), - metadata: obj.metadata as PrdJson['metadata'], + metadata: typeof obj.metadata === 'object' && obj.metadata !== null + ? (obj.metadata as PrdJson['metadata']) + : undefined,
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
src/commands/convert.tssrc/plugins/trackers/builtin/json.ts
🧰 Additional context used
🧬 Code graph analysis (1)
src/commands/convert.ts (1)
src/plugins/trackers/builtin/json.ts (2)
validatePrdJsonSchema(639-639)PrdJsonSchemaError(84-94)
🔇 Additional comments (9)
src/commands/convert.ts (2)
22-25: LGTM!The imports are correctly structured and aligned with the validation utilities exported from
json.ts.
508-520: LGTM!The pre-write validation is well-placed and the error handling correctly distinguishes between schema validation failures (indicating an internal bug) and other unexpected errors. The messaging appropriately directs users to report the issue since this validates internally-generated JSON.
src/plugins/trackers/builtin/json.ts (7)
81-94: LGTM!The custom error class is well-designed with
Object.setPrototypeOfto ensureinstanceofchecks work correctly across different TypeScript compilation targets. The readonly properties are appropriate for immutable error context.
96-106: LGTM!Good extraction of the error formatting logic into a reusable helper. The output format provides clear, actionable guidance.
126-158: LGTM!Excellent early detection of common AI hallucination patterns with actionable schema guidance. The detailed suggestion including the exact expected format will help users correct their files quickly.
376-378: LGTM!Good defensive typing—parsing to
unknownand validating before assignment ensures type safety. The validation placement inreadPrdcentralises schema checks for all read operations.
416-422: LGTM!The error handling appropriately distinguishes schema errors from other failures and provides helpful output via the extracted helper. Returning an empty array maintains backward compatibility while logging guides the user to fix the issue.
623-629: LGTM!Consistent error handling pattern with
getTasks. Good use of the extractedlogPrdSchemaErrorhelper to maintain DRY.
639-639: LGTM!The named export enables the validation function to be reused in
convert.tsfor pre-write validation.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/tui/components/PrdChatApp.tsx (1)
87-126: Schema example includes fields beyond what is listed as required, potentially confusing the AI.The
jsonSchemaExample(lines 87-101) includesdescription,acceptanceCriteria, andpriorityfields, but the "Required fields" section (lines 113-117) only listsid,title,passes, anddependsOn. Additionally, the PR objective from issue #11 specifies the expected schema as:{ name, branchName, userStories: [ { id, title, passes, dependsOn } ] }This mismatch may lead to inconsistent AI output — the model might include the extra fields from the example or omit them based on the required list. Consider either:
- Aligning the example to only show the truly required fields, or
- Expanding the "Required fields" list to explicitly document which fields are optional.
♻️ Suggested alignment with issue #11's expected schema
const jsonSchemaExample = `{ "name": "Feature Name", "branchName": "feature/my-feature", "userStories": [ { "id": "US-001", "title": "Story title", - "description": "As a user, I want...", - "acceptanceCriteria": ["Criterion 1"], - "priority": 1, "passes": false, "dependsOn": [] } ] }`;
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/tui/components/PrdChatApp.tsx
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Start all code files with a file-level JSDoc comment section explaining the file's purpose, prefixed with 'ABOUTME: '
Files:
src/tui/components/PrdChatApp.tsx
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Start all code files with a file-level JSDoc comment section explaining the file's purpose, prefixed with "ABOUTME: "
Files:
src/tui/components/PrdChatApp.tsx
🔇 Additional comments (1)
src/tui/components/PrdChatApp.tsx (1)
1-5: LGTM!The file has a proper ABOUTME comment that clearly explains its purpose, satisfying the coding guidelines.
|
@anthonymq - Thanks for the first community PR! Solid approach, and the error messages with actionable suggestions are helpful. One minor note: the JSDoc on getEpics() was removed. Not blocking, but worth restoring if you get a chance. Merging - welcome aboard! |
The JSDoc was accidentally removed in PR subsy#12. Restoring per reviewer feedback.
|
Great ! Thanks @subsy i'm using the tui as we speak and it's a really great tool ! |
|
Great to hear @anthonymq ! |
…tion fix: validate prd.json schema and improve AI prompt to prevent hallucinated formats
The JSDoc was accidentally removed in PR subsy#12. Restoring per reviewer feedback.
Summary
Fixes #11 - AI-generated
prd.jsonfiles sometimes use an incorrect schema that ralph-tui cannot load.Changes
1. Runtime Schema Validation (
json.ts)validatePrdJsonSchema()function that detects invalid schemas on load"prd"wrapper,"tasks"instead of"userStories")2. Explicit Schema in AI Prompt (
PrdChatApp.tsx)tasks,status,subtasks,estimated_hours)tasks/prd.json3. Pre-Write Validation (
convert.ts)ralph-tui convert --to jsonnow validates before writingTesting
bun run typecheckbun run buildbun run lintSummary by CodeRabbit
New Features
Bug Fixes
✏️ Tip: You can customize this high-level summary in your review settings.