Skip to content

fix: validate prd.json schema and improve AI prompt to prevent hallucinated formats#12

Merged
subsy merged 4 commits intosubsy:mainfrom
anthonymq:fix/prd-json-schema-validation
Jan 13, 2026
Merged

fix: validate prd.json schema and improve AI prompt to prevent hallucinated formats#12
subsy merged 4 commits intosubsy:mainfrom
anthonymq:fix/prd-json-schema-validation

Conversation

@anthonymq
Copy link
Copy Markdown
Contributor

@anthonymq anthonymq commented Jan 13, 2026

Summary

Fixes #11 - AI-generated prd.json files sometimes use an incorrect schema that ralph-tui cannot load.

Changes

1. Runtime Schema Validation (json.ts)

  • Added validatePrdJsonSchema() function that detects invalid schemas on load
  • Specific detection for common AI hallucination patterns ("prd" wrapper, "tasks" instead of "userStories")
  • Clear error messages with exact fix instructions

2. Explicit Schema in AI Prompt (PrdChatApp.tsx)

  • The prompt now includes the exact schema example
  • Explicit "DO NOT use" list for common mistakes (tasks, status, subtasks, estimated_hours)
  • Specifies output path: tasks/prd.json

3. Pre-Write Validation (convert.ts)

  • ralph-tui convert --to json now validates before writing
  • Prevents corrupt files from being saved

Testing

  • bun run typecheck
  • bun run build
  • bun run lint

Summary by CodeRabbit

  • New Features

    • Built-in runtime validation for PRD JSON with structured, actionable error reporting integrated into import and conversion flows.
    • Tracker option now enforces an exact PRD JSON output schema to improve reliability of generated task files.
    • Normalises inputs (accepts "project" as name alias, coerces fields) before writing.
  • Bug Fixes

    • Validation now runs before output creation; clearer messages, detailed error lines, and an exit path for unrecoverable schema errors.

✏️ Tip: You can customize this high-level summary in your review settings.

…inated formats

- Add validatePrdJsonSchema() with specific detection for common AI mistakes
- Include explicit schema example and DO NOT list in create-prd prompt
- Validate generated JSON before writing in convert command

Fixes subsy#11
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Jan 13, 2026

Walkthrough

Adds runtime PRD JSON schema validation with structured errors, integrates pre-write validation into the convert command to avoid saving invalid prd.json, and tightens the TUI AI prompt by embedding an explicit PRD JSON schema and strict output instructions.

Changes

Cohort / File(s) Summary
Schema validation & runtime parsing
src/plugins/trackers/builtin/json.ts
Adds PrdJsonSchemaError, validatePrdJsonSchema(data, filePath) and logPrdSchemaError; enforces and normalises PRD JSON, detects common AI-hallucination patterns, integrates validation into readPrd(), and exports the validator.
Convert command integration
src/commands/convert.ts
Imports validatePrdJsonSchema and PrdJsonSchemaError; validates generated prdJson before creating output directory or writing; prints structured schema errors with detail lines and a fix suggestion, and exits with code 1 on validation failure.
TUI prompt tightening
src/tui/components/PrdChatApp.tsx
Adds jsonSchemaExample and updates the JSON tracker option’s skillPrompt to include a CRITICAL block with an exact PRD JSON schema example, enumerated required/forbidden fields, and required output path (tasks/prd.json).

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant ConvertCmd as convert.ts
    participant Validator as validatePrdJsonSchema
    participant FileSystem as File System
    participant ErrorOutput as Error Output

    User->>ConvertCmd: run convert --to json
    ConvertCmd->>ConvertCmd: generate prdJson
    ConvertCmd->>Validator: validatePrdJsonSchema(prdJson, outputPath)
    alt Schema valid
        Validator-->>ConvertCmd: validated PrdJson
        ConvertCmd->>FileSystem: create output dir & write prd.json
        FileSystem-->>ConvertCmd: write success
        ConvertCmd-->>User: exit success
    else Schema invalid
        Validator-->>ConvertCmd: PrdJsonSchemaError (details + suggestion)
        ConvertCmd->>ErrorOutput: print structured error + fix suggestion
        ConvertCmd-->>User: exit with code 1
    end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I found a JSON tale askew and thin,
I nudged the schema, chased the stray bits in,
"Name and userStories — please align!" I cry,
I hop, I check, I warn — then watch the build fly,
A tidy prd, and off we spin.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main changes: adding schema validation and improving the AI prompt to prevent malformed prd.json files.
Linked Issues check ✅ Passed The pull request fully addresses all coding requirements from issue #11: runtime schema validation with error detection, improved AI prompt with explicit schema and dos/don'ts, and pre-write validation.
Out of Scope Changes check ✅ Passed All changes are directly scoped to addressing issue #11: schema validation logic, AI prompt improvements, and validation integration.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In @src/plugins/trackers/builtin/json.ts:
- Around line 147-151: The current assignment const name = (obj.name as string)
|| (obj.project as string) incorrectly treats falsy but present values (e.g.,
"") as missing; change the logic in the name resolution to first check property
existence (e.g., Object.prototype.hasOwnProperty.call(obj, 'name')) and then
verify typeof obj.name === 'string' to use it, otherwise check for 'project'
similarly; update the subsequent error push to trigger only when neither
property exists as a string.
🧹 Nitpick comments (3)
src/plugins/trackers/builtin/json.ts (2)

403-414: Consider extracting duplicated error handling logic.

The PrdJsonSchemaError handling block is duplicated in both getTasks (lines 403-412) and getEpics (lines 615-624). This could be extracted to a private helper method for consistency and maintainability.

♻️ Suggested refactor
private logPrdSchemaError(err: PrdJsonSchemaError): void {
  console.error(`\n${err.message}\n`);
  console.error('Issues found:');
  for (const detail of err.details) {
    console.error(`  - ${detail}`);
  }
  console.error(`\nHow to fix:\n${err.suggestion}\n`);
}

Then use in both methods:

    } catch (err) {
      if (err instanceof PrdJsonSchemaError) {
-       console.error(`\n${err.message}\n`);
-       console.error('Issues found:');
-       for (const detail of err.details) {
-         console.error(`  - ${detail}`);
-       }
-       console.error(`\nHow to fix:\n${err.suggestion}\n`);
+       this.logPrdSchemaError(err);
      } else {

Also applies to: 615-626


84-93: Optional: Consider adding prototype chain safeguard for custom Error subclass.

With your current ES2022 transpilation target, the instanceof checks for PrdJsonSchemaError work correctly without additional fixes. However, if the codebase is transpiled to older JavaScript versions (ES5 or early ES2015), the prototype chain may not be correctly set, causing instanceof checks to fail. Adding Object.setPrototypeOf is a defensive measure that remains harmless in modern environments while future-proofing against transpilation target changes.

🔧 Optional safeguard
export class PrdJsonSchemaError extends Error {
  constructor(
    message: string,
    public readonly details: string[],
    public readonly suggestion: string
  ) {
    super(message);
    this.name = 'PrdJsonSchemaError';
+    Object.setPrototypeOf(this, PrdJsonSchemaError.prototype);
  }
}
src/commands/convert.ts (1)

508-520: Validation placement is correct, but consider the error message context.

The pre-write validation is a good safety net. However, if validation fails here, it indicates a bug in the internal convertToPrdJson function (since the PRD parser should produce valid structures). The error message from PrdJsonSchemaError.suggestion mentions "regenerate with convert command", which would be confusing since this is the convert command.

Consider adding context-specific messaging for this case:

🔧 Suggested improvement
   try {
     validatePrdJsonSchema(prdJson, outputPath);
   } catch (err) {
     if (err instanceof PrdJsonSchemaError) {
-      printError('Generated prd.json failed schema validation:');
+      printError('Internal error: Generated prd.json failed schema validation.');
+      printError('This indicates a bug in the PRD parser. Please report this issue.');
       for (const detail of err.details) {
         console.error(`  - ${detail}`);
       }
-      console.error(`\n${err.suggestion}`);
+      // Don't print the generic suggestion as it refers to re-running this command
       process.exit(1);
     }
     throw err;
   }
📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b2461ca and 86fa64b.

📒 Files selected for processing (3)
  • src/commands/convert.ts
  • src/plugins/trackers/builtin/json.ts
  • src/tui/components/PrdChatApp.tsx
🧰 Additional context used
🧬 Code graph analysis (1)
src/commands/convert.ts (1)
src/plugins/trackers/builtin/json.ts (2)
  • validatePrdJsonSchema (636-636)
  • PrdJsonSchemaError (84-93)
🔇 Additional comments (5)
src/tui/components/PrdChatApp.tsx (2)

86-100: LGTM!

The JSON schema example is well-structured and consistent with the PrdJson interface defined in json.ts. It correctly demonstrates all required fields (id, title, passes, dependsOn) and optional fields, providing clear guidance for the AI agent.


106-125: LGTM!

The skill prompt is comprehensive and well-aligned with the schema validation in json.ts. The explicit "DO NOT use" list effectively addresses the common AI hallucination patterns identified in the linked issue. The prompt clearly specifies required fields and references the exact schema example.

src/plugins/trackers/builtin/json.ts (2)

211-223: LGTM!

The normalisation logic correctly handles the project alias and ensures a consistent PrdJson structure. The defensive fallback for passes on line 219 is reasonable as a safeguard.


636-636: LGTM!

The export enables reuse of validatePrdJsonSchema in convert.ts for pre-write validation, which aligns with the PR objectives.

src/commands/convert.ts (1)

22-25: LGTM!

The imports correctly reference the newly exported validation utilities from the JSON tracker module.

- Use property existence check for name field to handle empty strings
- Update error message for unsupported fields to say 'Remove these fields'
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
src/plugins/trackers/builtin/json.ts (3)

190-198: Minor: Potential duplicate error message for status field.

When a story has a status field but no passes field, the user receives two error messages: one from the passes validation (lines 180-184) and another from the unsupported fields check here. Consider removing 'status' from unsupportedFields since it's already handled with a more specific message above.

🔧 Suggested fix
-      const unsupportedFields = ['subtasks', 'estimated_hours', 'files', 'status'];
+      const unsupportedFields = ['subtasks', 'estimated_hours', 'files'];

221-221: Consider validating metadata fields.

The metadata object is cast directly without validating its internal structure. If an AI hallucinates incorrect types for createdAt, updatedAt, or version (e.g., numbers instead of strings), they would pass through. This is low-risk since metadata is optional, but could cause subtle issues if downstream code assumes string types.

🔧 Optional validation
-    metadata: obj.metadata as PrdJson['metadata'],
+    metadata: obj.metadata && typeof obj.metadata === 'object'
+      ? {
+          createdAt: typeof (obj.metadata as Record<string, unknown>).createdAt === 'string'
+            ? (obj.metadata as Record<string, unknown>).createdAt as string
+            : undefined,
+          updatedAt: typeof (obj.metadata as Record<string, unknown>).updatedAt === 'string'
+            ? (obj.metadata as Record<string, unknown>).updatedAt as string
+            : undefined,
+          version: typeof (obj.metadata as Record<string, unknown>).version === 'string'
+            ? (obj.metadata as Record<string, unknown>).version as string
+            : undefined,
+        }
+      : undefined,

403-412: Consider extracting duplicate error handling to a helper.

The error formatting logic for PrdJsonSchemaError is duplicated in both getTasks (lines 403-412) and getEpics (lines 615-624). If the error format needs to change, both locations must be updated.

🔧 Suggested refactor

Add a method to PrdJsonSchemaError or a standalone helper:

// Option 1: Add to PrdJsonSchemaError class
printError(): void {
  console.error(`\n${this.message}\n`);
  console.error('Issues found:');
  for (const detail of this.details) {
    console.error(`  - ${detail}`);
  }
  console.error(`\nHow to fix:\n${this.suggestion}\n`);
}

Then simplify the catch blocks:

     } catch (err) {
       if (err instanceof PrdJsonSchemaError) {
-        console.error(`\n${err.message}\n`);
-        console.error('Issues found:');
-        for (const detail of err.details) {
-          console.error(`  - ${detail}`);
-        }
-        console.error(`\nHow to fix:\n${err.suggestion}\n`);
+        err.printError();
       } else {

Also applies to: 615-624

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 86fa64b and 36d5edd.

📒 Files selected for processing (1)
  • src/plugins/trackers/builtin/json.ts
🔇 Additional comments (4)
src/plugins/trackers/builtin/json.ts (4)

81-93: LGTM! Well-structured custom error class.

The error class properly extends Error, sets the name property for identification, and exposes structured details and suggestions as readonly properties. This design enables clean error handling downstream.


112-145: Good detection of AI hallucination patterns with actionable guidance.

Early detection of common AI-generated schema mistakes (top-level prd or tasks wrappers) with a clear example of the correct structure is helpful. Throwing immediately here prevents confusing cascading validation errors.


354-369: LGTM! Clean integration of schema validation.

Validation occurs before caching, ensuring only valid data enters the cache. Letting exceptions propagate to callers (getTasks, getEpics) is appropriate since they have context-specific error handling.


636-636: LGTM!

The named export enables convert.ts to perform pre-write validation, completing the defence-in-depth strategy described in the PR objectives.

- Add Object.setPrototypeOf safeguard for PrdJsonSchemaError class
- Extract duplicated error handling to logPrdSchemaError helper
- Fix convert.ts error message to indicate internal bug vs user error
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
src/plugins/trackers/builtin/json.ts (1)

229-234: Misleading comment and unvalidated metadata cast.

Two minor observations:

  1. The comment on line 231 mentions "convert from status if needed as fallback", but status conversion isn't actually performed here—the fallback simply defaults to false.

  2. obj.metadata is cast directly without structural validation. If the metadata contains unexpected types (e.g., non-string values for createdAt), downstream code might encounter type mismatches.

Suggested improvements
     userStories: (obj.userStories as PrdUserStory[]).map((s) => ({
       ...s,
-      // Ensure passes is a boolean (convert from status if needed as fallback)
+      // Ensure passes is a boolean (default to false if somehow missing after validation)
       passes: typeof s.passes === 'boolean' ? s.passes : false,
     })),
-    metadata: obj.metadata as PrdJson['metadata'],
+    metadata: typeof obj.metadata === 'object' && obj.metadata !== null
+      ? (obj.metadata as PrdJson['metadata'])
+      : undefined,
📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 36d5edd and 509ac38.

📒 Files selected for processing (2)
  • src/commands/convert.ts
  • src/plugins/trackers/builtin/json.ts
🧰 Additional context used
🧬 Code graph analysis (1)
src/commands/convert.ts (1)
src/plugins/trackers/builtin/json.ts (2)
  • validatePrdJsonSchema (639-639)
  • PrdJsonSchemaError (84-94)
🔇 Additional comments (9)
src/commands/convert.ts (2)

22-25: LGTM!

The imports are correctly structured and aligned with the validation utilities exported from json.ts.


508-520: LGTM!

The pre-write validation is well-placed and the error handling correctly distinguishes between schema validation failures (indicating an internal bug) and other unexpected errors. The messaging appropriately directs users to report the issue since this validates internally-generated JSON.

src/plugins/trackers/builtin/json.ts (7)

81-94: LGTM!

The custom error class is well-designed with Object.setPrototypeOf to ensure instanceof checks work correctly across different TypeScript compilation targets. The readonly properties are appropriate for immutable error context.


96-106: LGTM!

Good extraction of the error formatting logic into a reusable helper. The output format provides clear, actionable guidance.


126-158: LGTM!

Excellent early detection of common AI hallucination patterns with actionable schema guidance. The detailed suggestion including the exact expected format will help users correct their files quickly.


376-378: LGTM!

Good defensive typing—parsing to unknown and validating before assignment ensures type safety. The validation placement in readPrd centralises schema checks for all read operations.


416-422: LGTM!

The error handling appropriately distinguishes schema errors from other failures and provides helpful output via the extracted helper. Returning an empty array maintains backward compatibility while logging guides the user to fix the issue.


623-629: LGTM!

Consistent error handling pattern with getTasks. Good use of the extracted logPrdSchemaError helper to maintain DRY.


639-639: LGTM!

The named export enables the validation function to be reused in convert.ts for pre-write validation.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
src/tui/components/PrdChatApp.tsx (1)

87-126: Schema example includes fields beyond what is listed as required, potentially confusing the AI.

The jsonSchemaExample (lines 87-101) includes description, acceptanceCriteria, and priority fields, but the "Required fields" section (lines 113-117) only lists id, title, passes, and dependsOn. Additionally, the PR objective from issue #11 specifies the expected schema as:

{ name, branchName, userStories: [ { id, title, passes, dependsOn } ] }

This mismatch may lead to inconsistent AI output — the model might include the extra fields from the example or omit them based on the required list. Consider either:

  1. Aligning the example to only show the truly required fields, or
  2. Expanding the "Required fields" list to explicitly document which fields are optional.
♻️ Suggested alignment with issue #11's expected schema
   const jsonSchemaExample = `{
   "name": "Feature Name",
   "branchName": "feature/my-feature",
   "userStories": [
     {
       "id": "US-001",
       "title": "Story title",
-      "description": "As a user, I want...",
-      "acceptanceCriteria": ["Criterion 1"],
-      "priority": 1,
       "passes": false,
       "dependsOn": []
     }
   ]
 }`;
📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 509ac38 and 27ca73f.

📒 Files selected for processing (1)
  • src/tui/components/PrdChatApp.tsx
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}

📄 CodeRabbit inference engine (CLAUDE.md)

Start all code files with a file-level JSDoc comment section explaining the file's purpose, prefixed with 'ABOUTME: '

Files:

  • src/tui/components/PrdChatApp.tsx
**/*.{js,ts,tsx,jsx}

📄 CodeRabbit inference engine (AGENTS.md)

Start all code files with a file-level JSDoc comment section explaining the file's purpose, prefixed with "ABOUTME: "

Files:

  • src/tui/components/PrdChatApp.tsx
🔇 Additional comments (1)
src/tui/components/PrdChatApp.tsx (1)

1-5: LGTM!

The file has a proper ABOUTME comment that clearly explains its purpose, satisfying the coding guidelines.

@subsy
Copy link
Copy Markdown
Owner

subsy commented Jan 13, 2026

@anthonymq - Thanks for the first community PR!

Solid approach, and the error messages with actionable suggestions are helpful.

One minor note: the JSDoc on getEpics() was removed. Not blocking, but worth restoring if you get a chance.

Merging - welcome aboard!

@subsy subsy merged commit b6f98f0 into subsy:main Jan 13, 2026
2 checks passed
anthonymq added a commit to anthonymq/ralph-tui that referenced this pull request Jan 13, 2026
The JSDoc was accidentally removed in PR subsy#12. Restoring per reviewer feedback.
@anthonymq
Copy link
Copy Markdown
Contributor Author

Great ! Thanks @subsy i'm using the tui as we speak and it's a really great tool !

@subsy
Copy link
Copy Markdown
Owner

subsy commented Jan 13, 2026

Great to hear @anthonymq !

sakaman pushed a commit to sakaman/ralph-tui that referenced this pull request Feb 15, 2026
…tion

fix: validate prd.json schema and improve AI prompt to prevent hallucinated formats
sakaman pushed a commit to sakaman/ralph-tui that referenced this pull request Feb 15, 2026
The JSDoc was accidentally removed in PR subsy#12. Restoring per reviewer feedback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: create-prd generates invalid prd.json schema that fails to load

2 participants