Wire up dead Zod schemas for LLM output validation (vibe-kanban)#13
Conversation
Replaces manual JSON.parse + duck-typing chains (obj.riskLevel || obj.risk_level || ...) with LlmAssessmentSchema.parse() in three of four parse sites. Adds a transform on the schema to centralize the snake_case to camelCase mapping into LlmAssessment shape. Triage batch site (llm.ts) left with TODO(human) for indexed variant. Tests updated: removed camelCase-tolerance test and lenient-defaults test that contradicted the new strict validation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the schema rollout — replaces the final duck-typed parse loop in triageBatchAssess with IndexedLlmAssessmentSchema.safeParse(). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request replaces manual JSON parsing and mapping with type-safe Zod schema validation for LLM assessments. It introduces LlmAssessmentSchema and IndexedLlmAssessmentSchema to handle the transformation from snake_case wire format to internal camelCase types, simplifying the logic in llm-batch.ts and llm.ts. Tests were updated to reflect stricter validation requirements. A review comment suggests refactoring the schema transformations to reduce code duplication by extracting a shared mapping function.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 057ffba4ed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex P1: schema rejected `false_positive_reason: null`, which prompts.yaml lines 40 and 432 explicitly allow. Coerce null/undefined to '' via .nullable().optional().transform() so prompt-compliant LLM output isn't silently dropped. Gemini: factored the snake_case -> camelCase mapping into a shared mapToInternal() helper consumed by both LlmAssessmentSchema and IndexedLlmAssessmentSchema, eliminating the duplicated transform body. Added a regression test asserting null false_positive_reason parses to ''. 214/213 tests passing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
src/analyzer/schemas.tsdefinesAssessmentSchema,BatchAssessmentSchema, and others with full Zod validation. This file is effectively dead code — bothllm.tsandllm-batch.tsuse manualJSON.parse()with duck-typing fallbacks (obj.riskLevel || obj.risk_level || ...) instead.\n\nThe providers already exposegenerateObject<T>(schema, ...)for structured output validated at the SDK level, but this is never used for assessments.\n\nFix: replace the manual JSON parsing in both files withAssessmentSchema.parse()to catch malformed LLM output reliably and eliminate the duck-typing chains.