π Error Message Quality Analysis
Analysis Date: 2026-04-05
Test Cases: 3
Average Score: 68.3/100
Status: β οΈ Needs Improvement
Method: Analysis performed via compiler source-code tracing (the sandbox environment restricts binary execution). Error outputs were derived by tracing the full call chains through pkg/parser/, pkg/workflow/, and pkg/console/.
Executive Summary
Three workflows were analysed across three error categories. The compiler's schema-validation path (engine typos, invalid permission scopes) produces well-structured, IDE-parseable errors with source context. However, YAML syntax errors expose a structural inconsistency that causes IDEs to navigate to the wrong line, and several patterns introduce avoidable noise that reduces readability.
Key Findings:
- Strengths: "Did you mean?" suggestions for engine typos; rich source context with
^ pointer for schema errors; permissions docs link; YAML error messages translated from cryptic goccy text to plain English
- Weaknesses: YAML parse errors wrap with
file:1:1: instead of the actual error line; schema validation errors repeat position information redundantly in both the IDE prefix and the message body; invalid-permissions errors may list valid scopes twice (once from appendKnownFieldValidValuesHint, once from generateSchemaBasedSuggestions)
- Critical Issues: None β all scores are above the critical threshold (55)
Test Case Results
Test Case 1: YAML Syntax Error (missing colon) β Score: 63/100 β οΈ
Test Configuration
Workflow: .github/workflows/artifacts-summary.md (97 lines, simple)
Error Type: Category A β Frontmatter YAML syntax error
Error Introduced: Line 9 β engine copilot (: removed β engine copilot)
Predicted Compiler Output
.github/workflows/test-1.md:1:1: error: failed to parse frontmatter:
[9:1] missing ':' after key β YAML mapping entries require 'key: value' format
> 9 | engine copilot
^
```
_Source trace_: `frontmatter_content.go:67` wraps the goccy error string in `fmt.Errorf("failed to parse frontmatter:\n%s", formattedErr)`. This is NOT a `FormattedParserError`, so `compiler.go:96` calls `formatCompilerError(markdownPath, "error", err.Error(), err)` which hard-codes position `1:1`.
#### Evaluation Scores
| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 18/25 | Good |
| Actionability | 20/25 | Good |
| Context | 13/20 | Limited |
| Examples | 3/15 | Poor |
| Consistency | 9/15 | Inconsistent |
| **Total** | **63/100** | **Acceptable** |
#### Strengths
- β
Excellent YAML error translation: "missing ':' after key β YAML mapping entries require 'key: value' format" (from `yaml_error.go` translation table)
- β
Goccy-style source context with `^` pointer embedded in message body
- β
Correct inner line reference `[9:1]` visible in message
#### Weaknesses
- β **Outer IDE position is wrong**: `file:1:1:` instead of `file:9:1:` β IDE "click-to-jump" navigates to the wrong line. Root cause: `frontmatter_content.go:67` uses `fmt.Errorf` (not `FormattedParserError`), so `compiler.go:96` falls back to `1:1`
- β οΈ Generic wrapper `"failed to parse frontmatter:"` adds noise without value
- β οΈ Mixed format: outer `file:N:M:` style and inner `[line:col]` goccy style in same output
- β οΈ No example of correct syntax (`engine: copilot`)
#### Improvement Suggestions
1. **Fix outer position by returning `FormattedParserError` from frontmatter parser**:
Instead of `fmt.Errorf("failed to parse frontmatter:\n%s", formattedErr)`, extract the line/column from the goccy error and return a `FormattedParserError` with the exact position, so the compiler gets a properly-positioned error at `file:9:1:` not `file:1:1:`.
2. **Remove the "failed to parse frontmatter:" wrapper**:
The goccy output already contains full context. The wrapper message is redundant noise.
3. **Add a corrected syntax example** (following the `mcp.go` pattern):
```
Correct usage:
engine: copilot
```
</details>
<details>
<summary>Test Case 2: Invalid Engine Name (typo) β Score: 76/100 β
</summary>
#### Test Configuration
**Workflow**: `.github/workflows/agentic-observability-kit.md` (265 lines, medium)
**Error Type**: Category B β Configuration error (invalid value)
**Error Introduced**: `engine: copiilot` (double-`i` typo)
#### Predicted Compiler Output
```
.github/workflows/test-2.md:11:1: error: 'engine' (line 11, col 1): value must be one of 'claude', 'codex', 'copilot', 'gemini'. Did you mean 'copilot'?
9 | permissions:
10 | contents: read
11 | engine: copiilot
^^^^^^
12 | strict: true
13 | tracker-id: agentic-observability-kit
```
_Source trace_: `validateWithSchemaAndLocation` β `cleanOneOfMessage` strips the `oneOf` jargon β `LocateJSONPathInYAMLWithAdditionalProperties` finds the precise line β `generateSchemaBasedSuggestions` computes Levenshtein distance between "copiilot" and enum values (distance 1 to "copilot") β returns `Did you mean 'copilot'?`
#### Evaluation Scores
| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 21/25 | Excellent |
| Actionability | 22/25 | Excellent |
| Context | 17/20 | Good |
| Examples | 5/15 | Minimal |
| Consistency | 11/15 | Generally consistent |
| **Total** | **76/100** | **Good** |
#### Strengths
- β
"Did you mean 'copilot'?" β highly actionable, pinpoints the fix instantly
- β
Lists all valid engine names in the error message
- β
Precise line/column with source context and `^` pointer
- β
IDE-parseable format (`file:line:col:`)
#### Weaknesses
- β οΈ **Redundant position** in message body: `'engine' (line 11, col 1):` duplicates what the IDE-format prefix already shows. When there is only one failure the `formatSchemaFailureDetail` path-prefix adds noise
- β οΈ "value must be one of" is JSON-schema vocabulary; plainer phrasing (e.g. "unknown engine") would be more accessible
- β οΈ No pointer to documentation or custom engine reference
#### Improvement Suggestions
1. **Suppress the `'field' (line N, col M):` prefix when there is only a single failure** β it duplicates the IDE prefix and clutters the message
2. **Replace "value must be one of" with plain language**, e.g. `invalid engine 'copiilot'. Valid engines: claude, codex, copilot, gemini`
3. **Add documentation link**: "For custom engines see: https://github.com/github/gh-aw#custom-engines"
</details>
<details>
<summary>Test Case 3: Invalid Permissions Scope β Score: 66/100 β οΈ</summary>
#### Test Configuration
**Workflow**: `.github/workflows/security-compliance.md` (301 lines, complex)
**Error Type**: Category C β Semantic / unknown property
**Error Introduced**: `unknown-scope: read` added to `permissions:` block
#### Predicted Compiler Output
```
.github/workflows/test-3.md:N:3: error: 'permissions/unknown-scope' (line N, col 3):
Unknown property: unknown-scope (Valid permission scopes: actions, all, attestations,
checks, contents, deployments, discussions, id-token, issues, metadata, models,
organization-projects, packages, pages, pull-requests, repository-projects,
security-events, statuses, vulnerability-alerts) See: https://docs.github.com/....
Valid fields are: actions, all, attestations, checks, contents, deployments,
discussions, id-token, issues, metadata, ...
[context lines + ^ pointer]
Source trace: formatSchemaFailureDetail calls appendKnownFieldValidValuesHint (adds the full scopes list + docs link), then calls generateSchemaBasedSuggestions β generateFieldSuggestions (no close match β adds "Valid fields are: β¦" listing the scopes again up to 10 items).
Evaluation Scores
| Dimension |
Score |
Rating |
| Clarity |
15/25 |
Unclear |
| Actionability |
19/25 |
Moderate |
| Context |
16/20 |
Good |
| Examples |
4/15 |
Poor |
| Consistency |
12/15 |
Generally consistent |
| Total |
66/100 |
Acceptable |
Strengths
- β
Documentation link to GitHub Actions permissions docs
- β
Precise source location with
^ pointer
- β
Lists the valid permission scope names
Weaknesses
- β Duplicate valid-scopes list:
appendKnownFieldValidValuesHint adds the full scopes list, then generateSchemaBasedSuggestions β generateFieldSuggestions adds it again (as "Valid fields are: β¦") because no close match is found. The message becomes extremely long
- β οΈ Redundant position in message body (
'permissions/unknown-scope' (line N, col M):)
- β οΈ No "Did you mean?" for scopes that are even moderately close (threshold may be too strict)
- β οΈ No example of correct usage
Improvement Suggestions
- Prevent duplicate scope listing: In
formatSchemaFailureDetail, skip the generateSchemaBasedSuggestions call (or suppress its "Valid fields" branch) when appendKnownFieldValidValuesHint has already added valid-values content to the message. A simple guard: check whether message already contains "Valid" before calling the suggestions generator
- Shorten the scope list in
appendKnownFieldValidValuesHint: Cap at the same maxAcceptedFields = 10 limit used by generateFieldSuggestions
- Add a corrected-syntax example:
permissions:
contents: read
issues: write
Overall Statistics
| Metric |
Value |
| Tests Run |
3 |
| Average Score |
68.3/100 |
| Good (70β84) |
1 (TC2) |
| Acceptable (55β69) |
2 (TC1, TC3) |
| Poor (<55) |
0 |
| Below threshold (70) |
β
Yes β issue created |
Quality Assessment: β οΈ Needs Improvement β Average score 68.3/100, below the 70 threshold. Two of three test cases fell in the "Acceptable" range. No critical failures were found, but several patterns consistently degrade developer experience.
Priority Improvement Recommendations
π΄ High Priority
1. Fix IDE position for YAML syntax errors (pkg/parser/frontmatter_content.go)
Currently: frontmatter_content.go:67 returns a plain fmt.Errorf, causing the compiler to fall back to file:1:1:.
// CURRENT β loses precise position
return nil, fmt.Errorf("failed to parse frontmatter:\n%s", formattedErr)
// IMPROVED β extract line/col from goccy error, return FormattedParserError
line, col := extractLineColFromYAMLError(err)
compilerErr := console.CompilerError{
Position: console.ErrorPosition{File: filePath, Line: line + frontmatterOffset, Column: col},
Type: "error",
Message: translateYAMLError(yaml.FormatError(err, false, false)), // message only, no source
Context: contextLinesAroundLine(source, line, 3),
}
return nil, &FormattedParserError{formatted: console.FormatError(compilerErr)}
Impact: IDE "click to jump" will take developers directly to the offending line instead of line 1.
2. Prevent duplicate valid-scopes output (pkg/parser/schema_compiler.go:formatSchemaFailureDetail)
message = appendKnownFieldValidValuesHint(message, pathInfo.Path)
// Guard: only add suggestions if the hint didn't already add a valid-values list
if !strings.Contains(message, "Valid ") {
suggestions := generateSchemaBasedSuggestions(...)
if suggestions != "" {
message = message + ". " + suggestions
}
}
Impact: Eliminates the extremely long double-listing of all permission scopes.
π‘ Medium Priority
3. Remove redundant 'field' (line N, col M): prefix for single-failure errors (pkg/parser/schema_compiler.go:validateWithSchemaAndLocation)
// Only prefix with path+position for multi-failure output
message := detailLines[0]
if len(detailLines) > 1 {
message = "Multiple schema validation failures:\n- " + strings.Join(detailLines, "\n- ")
} else {
// For single failure, strip the path prefix since IDE format already shows position
message = stripPathPositionPrefix(detailLines[0])
}
4. Simplify schema jargon in engine error β Replace "value must be one of 'claude', 'codex', 'copilot', 'gemini'" with "invalid engine '%s'. Valid engines: claude, codex, copilot, gemini" by extracting the user-typed value from frontmatter.
π’ Low Priority (Nice to Have)
5. Add correct-syntax examples following the mcp.go pattern β mcp.go already includes Example:\n YAML blocks in error messages. The same pattern could be adopted for the most common schema validation errors (engine, permissions) to make errors immediately self-sufficient.
6. Remove the "failed to parse frontmatter:" wrapper β Once fix #1 is applied, this wrapper message becomes redundant. The formatted error with position already describes the problem.
Implementation Notes
The // Hints removed as per requirements comments in pkg/parser/schema_compiler.go (lines 352 and 380) indicate the CompilerError.Hint field was deliberately removed. If that decision is revisited, the Hint field already exists in pkg/console/console_types.go and is rendered by pkg/console/console.go (with hint: prefix in cyan). This would be the natural place to add correct-syntax examples without cluttering the main message.
The MCP-related errors in pkg/parser/mcp.go serve as a positive example: they include Example:\n YAML blocks that give developers the exact syntax they need. This pattern is worth adopting more broadly for schema-validated fields.
References:
Generated by Daily Syntax Error Quality Check Β· β 9.2M Β· β·
π Error Message Quality Analysis
Analysis Date: 2026-04-05β οΈ Needs Improvement
Test Cases: 3
Average Score: 68.3/100
Status:
Executive Summary
Three workflows were analysed across three error categories. The compiler's schema-validation path (engine typos, invalid permission scopes) produces well-structured, IDE-parseable errors with source context. However, YAML syntax errors expose a structural inconsistency that causes IDEs to navigate to the wrong line, and several patterns introduce avoidable noise that reduces readability.
Key Findings:
^pointer for schema errors; permissions docs link; YAML error messages translated from cryptic goccy text to plain Englishfile:1:1:instead of the actual error line; schema validation errors repeat position information redundantly in both the IDE prefix and the message body; invalid-permissions errors may list valid scopes twice (once fromappendKnownFieldValidValuesHint, once fromgenerateSchemaBasedSuggestions)Test Case Results
Test Case 1: YAML Syntax Error (missing colon) β Score: 63/100β οΈ
Test Configuration
Workflow:
.github/workflows/artifacts-summary.md(97 lines, simple)Error Type: Category A β Frontmatter YAML syntax error
Error Introduced: Line 9 β
engine copilot(:removed βengine copilot)Predicted Compiler Output
Source trace:
formatSchemaFailureDetailcallsappendKnownFieldValidValuesHint(adds the full scopes list + docs link), then callsgenerateSchemaBasedSuggestionsβgenerateFieldSuggestions(no close match β adds "Valid fields are: β¦" listing the scopes again up to 10 items).Evaluation Scores
Strengths
^pointerWeaknesses
appendKnownFieldValidValuesHintadds the full scopes list, thengenerateSchemaBasedSuggestionsβgenerateFieldSuggestionsadds it again (as "Valid fields are: β¦") because no close match is found. The message becomes extremely long'permissions/unknown-scope' (line N, col M):)Improvement Suggestions
formatSchemaFailureDetail, skip thegenerateSchemaBasedSuggestionscall (or suppress its "Valid fields" branch) whenappendKnownFieldValidValuesHinthas already added valid-values content to the message. A simple guard: check whethermessagealready contains "Valid" before calling the suggestions generatorappendKnownFieldValidValuesHint: Cap at the samemaxAcceptedFields = 10limit used bygenerateFieldSuggestionsOverall Statistics
Quality Assessment:β οΈ Needs Improvement β Average score 68.3/100, below the 70 threshold. Two of three test cases fell in the "Acceptable" range. No critical failures were found, but several patterns consistently degrade developer experience.
Priority Improvement Recommendations
π΄ High Priority
1. Fix IDE position for YAML syntax errors (
pkg/parser/frontmatter_content.go)Currently:
frontmatter_content.go:67returns a plainfmt.Errorf, causing the compiler to fall back tofile:1:1:.Impact: IDE "click to jump" will take developers directly to the offending line instead of line 1.
2. Prevent duplicate valid-scopes output (
pkg/parser/schema_compiler.go:formatSchemaFailureDetail)Impact: Eliminates the extremely long double-listing of all permission scopes.
π‘ Medium Priority
3. Remove redundant
'field' (line N, col M):prefix for single-failure errors (pkg/parser/schema_compiler.go:validateWithSchemaAndLocation)4. Simplify schema jargon in engine error β Replace
"value must be one of 'claude', 'codex', 'copilot', 'gemini'"with"invalid engine '%s'. Valid engines: claude, codex, copilot, gemini"by extracting the user-typed value from frontmatter.π’ Low Priority (Nice to Have)
5. Add correct-syntax examples following the
mcp.gopattern βmcp.goalready includesExample:\nYAML blocks in error messages. The same pattern could be adopted for the most common schema validation errors (engine, permissions) to make errors immediately self-sufficient.6. Remove the
"failed to parse frontmatter:"wrapper β Once fix #1 is applied, this wrapper message becomes redundant. The formatted error with position already describes the problem.Implementation Notes
The
// Hints removed as per requirementscomments inpkg/parser/schema_compiler.go(lines 352 and 380) indicate theCompilerError.Hintfield was deliberately removed. If that decision is revisited, theHintfield already exists inpkg/console/console_types.goand is rendered bypkg/console/console.go(withhint:prefix in cyan). This would be the natural place to add correct-syntax examples without cluttering the main message.The MCP-related errors in
pkg/parser/mcp.goserve as a positive example: they includeExample:\nYAML blocks that give developers the exact syntax they need. This pattern is worth adopting more broadly for schema-validated fields.References: