Skip to content

Commit 511d9bd

Browse files
semdsemd
authored
Merge branch 'main' into workflows/step_registry_completion_improvement
2 parents 8a01d27 + c2dd103 commit 511d9bd

8,627 files changed

Lines changed: 387314 additions & 138774 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/debug-oas/SKILL.md‎

Lines changed: 83 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: debug-oas
3-
description: Use when debugging OpenAPI (OAS) issues for a specific API area in Kibana by scoping validation output with one or more --path filters.
3+
description: Use when debugging OpenAPI (OAS) issues for a specific API area in Kibana by scoping validation output with one or more --path filters, then separating structural invalid-OAS failures from quality or documentation gaps such as missing descriptions.
44
user-invocable: true
55
disable-model-invocation: true
66
---
@@ -11,12 +11,24 @@ disable-model-invocation: true
1111

1212
Use `node ./scripts/validate_oas_docs.js` to validate Kibana OAS, but always scope output with `--path` so developers can focus on the API area they are actively changing.
1313

14+
Use this skill when the developer needs issue breakdown, categorization, or representative examples. For quick pass/fail only, use `validate-oas` first.
15+
16+
If results look stale or surprising, refresh generated OAS first using the environment setup flow in `validate-oas`. Treat stale `oas_docs` as a setup problem, not a debugging conclusion.
17+
18+
The validator can surface two broad categories of issues:
19+
- `structural`: invalid OAS problems that usually block correctness, such as schema violations, invalid shapes, unresolved references, or mismatches between path definitions and the spec structure.
20+
- `quality`: documentation completeness problems such as missing `description`, `summary`, `example`, or `examples`.
21+
22+
When reporting results, always separate these categories. Lead with structural issues first.
23+
1424
## Required interaction flow
1525

1626
1. Ask what APIs the developer is working on.
1727
2. Ask for one or more HTTP API paths (for example `/api/fleet/agent_policies`).
18-
3. Run validation with those route-style `--path` filters.
19-
4. Display the command output directly so the developer can see current issues.
28+
3. Refresh generated OAS first when needed by following the environment setup flow in `validate-oas`.
29+
4. Run validation with those route-style `--path` filters.
30+
5. Classify the resulting issues into `structural` vs `quality`.
31+
6. Display the command output directly so the developer can see current issues.
2032

2133
Do not skip questions (1) and (2) unless the developer already provided the API paths.
2234

@@ -53,12 +65,79 @@ node ./scripts/validate_oas_docs.js --only serverless --path <api_route_prefix>
5365

5466
Default behavior:
5567
- Unless the developer asks otherwise, always include `--only traditional` so validation runs against a single OAS output file.
68+
- Prefer `--only traditional` by default because it matches the common local debugging path and keeps output narrower.
69+
- Use `--only serverless` only when the developer explicitly asks for it.
70+
71+
Optional structural-only summary when the raw output is noisy:
72+
73+
```bash
74+
node ./scripts/validate_oas_docs.js --only traditional --path <api_route_prefix> 2>&1 \
75+
| node .agents/skills/debug-oas/scripts/extract_structural_oas_issues.js
76+
```
77+
78+
If the developer wants to keep documentation issues in that helper output:
79+
80+
```bash
81+
node ./scripts/validate_oas_docs.js --only traditional --path <api_route_prefix> 2>&1 \
82+
| node .agents/skills/debug-oas/scripts/extract_structural_oas_issues.js --include-docs
83+
```
84+
85+
Only use the helper summary as a supplement. Keep the raw validator output available for exact debugging.
86+
87+
## Issue categorization
88+
89+
Classify issues using these defaults:
90+
91+
- `quality`:
92+
- missing `description`
93+
- missing `summary`
94+
- missing `example`
95+
- missing `examples`
96+
- `structural`:
97+
- everything else by default
98+
- examples include invalid schemas, type mismatches, missing required non-doc fields, invalid parameter definitions, unresolved refs, or malformed response/request structures
99+
100+
Heuristic:
101+
- If the message is a docs-completeness complaint about `description`, `summary`, `example`, or `examples`, treat it as `quality`.
102+
- Otherwise treat it as `structural` unless the developer explicitly asks for a finer split.
103+
104+
Severity guidance:
105+
- `structural` = blocking invalid OAS
106+
- `quality` = docs completeness or polish problems
107+
108+
If a run mixes both categories, report structural counts first, then quality counts.
109+
110+
## Scope narrowing
111+
112+
If the initial scope is noisy, narrow in this order:
113+
- start with a broad product area such as `/api/fleet`
114+
- narrow to a feature area such as `/api/fleet/epm`
115+
- narrow again to a route family such as `/api/fleet/epm/packages`
116+
117+
Suggest narrower scopes when one area dominates the structural issues.
56118

57119
## Output behavior
58120

59121
- Use the CLI line `Found N errors in ...` as the source of truth for issue count.
60122
- If the run has 25 or fewer issues, show the exact CLI output with no summarization.
61123
- If the run has more than 25 issues, summarize key patterns and suggest a narrower `--path` scope.
62-
- When summarizing, include a few issues copied verbatim from CLI output (for example one sample per dominant issue type such as missing `examples` and missing `description`).
124+
- When summarizing, include:
125+
- structural issue count
126+
- quality issue count
127+
- a few issues copied verbatim from CLI output, ideally one sample per dominant category
128+
- When summarizing, use language like:
129+
- `Structural issues (invalid OAS): ...`
130+
- `Quality issues (docs gaps): ...`
63131
- Prefer full issue output (do not use `--skip-printing-issues`) when debugging.
64132
- If no issues are shown for the selected path, suggest widening or adjusting the `--path` prefix.
133+
- If only quality issues remain, say that explicitly so the developer knows the remaining work is documentation-oriented rather than structural.
134+
135+
## Output template
136+
137+
When summarizing, use this shape:
138+
139+
```text
140+
Total issues: N
141+
Structural issues: X
142+
Quality issues: Y
143+
```

‎.agents/skills/debug-oas/scripts/extract_structural_oas_issues.js‎

Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
#!/usr/bin/env node
2+
/*
3+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
4+
* or more contributor license agreements. Licensed under the "Elastic License
5+
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
6+
* Public License v 1"; you may not use this file except in compliance with, at
7+
* your election, the "Elastic License 2.0", the "GNU Affero General Public
8+
* License v3.0 only", or the "Server Side Public License, v 1".
9+
*/
10+
11+
require('@kbn/setup-node-env');
12+
13+
var Fs = require('node:fs');
14+
var Path = require('node:path');
15+
16+
var DOC_MESSAGE_PATTERNS = [
17+
/required property 'example'/,
18+
/required property 'examples'/,
19+
/required property 'description'/,
20+
/required property 'summary'/,
21+
];
22+
23+
main();
24+
25+
function main() {
26+
var args = parseArgs(process.argv.slice(2));
27+
28+
readInput(args.inputPath)
29+
.then(function (input) {
30+
var parsedIssues = parseIssues(input);
31+
var filteredIssues = parsedIssues.filter(function (issue) {
32+
return args.includeDocs || !isDocumentationIssue(issue.message);
33+
});
34+
var groupedIssues = groupByMessage(filteredIssues);
35+
36+
printSummary({
37+
inputPath: args.inputPath,
38+
parsedIssues: parsedIssues,
39+
filteredIssues: filteredIssues,
40+
groupedIssues: groupedIssues,
41+
json: args.json,
42+
});
43+
})
44+
.catch(function (error) {
45+
console.error(error instanceof Error ? error.message : String(error));
46+
process.exit(1);
47+
});
48+
}
49+
50+
function parseArgs(argv) {
51+
var parsed = {
52+
includeDocs: false,
53+
inputPath: null,
54+
json: false,
55+
};
56+
var index;
57+
58+
for (index = 0; index < argv.length; index += 1) {
59+
var arg = argv[index];
60+
61+
if (arg === '--help' || arg === '-h') {
62+
printHelp();
63+
process.exit(0);
64+
}
65+
66+
if (arg === '--include-docs') {
67+
parsed.includeDocs = true;
68+
continue;
69+
}
70+
71+
if (arg === '--json') {
72+
parsed.json = true;
73+
continue;
74+
}
75+
76+
if (arg.startsWith('-')) {
77+
printHelp('Unknown argument "' + arg + '"');
78+
process.exit(1);
79+
}
80+
81+
if (parsed.inputPath !== null) {
82+
printHelp('Only one input path may be provided.');
83+
process.exit(1);
84+
}
85+
86+
parsed.inputPath = arg;
87+
}
88+
89+
return parsed;
90+
}
91+
92+
function printHelp(errorMessage) {
93+
if (errorMessage) {
94+
console.error(errorMessage);
95+
console.error('');
96+
}
97+
98+
console.log(`Extract structural OAS issues from validate_oas_docs output.
99+
100+
Usage:
101+
node .agents/skills/debug-oas/scripts/extract_structural_oas_issues.js [validate-output.txt]
102+
node scripts/validate_oas_docs.js --only traditional 2>&1 | node .agents/skills/debug-oas/scripts/extract_structural_oas_issues.js
103+
104+
Options:
105+
--include-docs Keep example/description/summary issues in the output.
106+
--json Print machine-readable JSON.
107+
`);
108+
}
109+
110+
function readInput(inputPath) {
111+
if (inputPath) {
112+
return Promise.resolve(Fs.readFileSync(Path.resolve(inputPath), 'utf8'));
113+
}
114+
115+
if (process.stdin.isTTY) {
116+
printHelp('Provide a path to saved CLI output or pipe input on stdin.');
117+
process.exit(1);
118+
}
119+
120+
return new Promise(function (resolve, reject) {
121+
var chunks = [];
122+
123+
process.stdin.setEncoding('utf8');
124+
process.stdin.on('data', function (chunk) {
125+
chunks.push(chunk);
126+
});
127+
process.stdin.on('end', function () {
128+
resolve(chunks.join(''));
129+
});
130+
process.stdin.on('error', reject);
131+
});
132+
}
133+
134+
function parseIssues(input) {
135+
var issues = [];
136+
var lines = stripAnsi(input).split(/\r?\n/);
137+
var currentPath = null;
138+
var currentMessage = null;
139+
var currentSchemaPath = null;
140+
var index;
141+
142+
function flushIssue() {
143+
if (!currentPath || !currentMessage) {
144+
currentPath = null;
145+
currentMessage = null;
146+
currentSchemaPath = null;
147+
return;
148+
}
149+
150+
issues.push({
151+
path: currentPath,
152+
message: currentMessage,
153+
schemaPath: currentSchemaPath,
154+
});
155+
156+
currentPath = null;
157+
currentMessage = null;
158+
currentSchemaPath = null;
159+
}
160+
161+
for (index = 0; index < lines.length; index += 1) {
162+
var content = normalizeIssueLine(lines[index]);
163+
164+
if (!content) {
165+
flushIssue();
166+
continue;
167+
}
168+
169+
if (content.startsWith('/')) {
170+
flushIssue();
171+
currentPath = content;
172+
continue;
173+
}
174+
175+
if (!currentPath) {
176+
continue;
177+
}
178+
179+
if (content.startsWith('Failed check @ schema path:')) {
180+
currentSchemaPath = content.slice('Failed check @ schema path:'.length).trim();
181+
flushIssue();
182+
continue;
183+
}
184+
185+
if (currentMessage === null) {
186+
currentMessage = content;
187+
continue;
188+
}
189+
}
190+
191+
flushIssue();
192+
193+
return issues;
194+
}
195+
196+
function normalizeIssueLine(line) {
197+
var match = line.match(/^\s*│\s*(.*)$/);
198+
var content = match ? match[1] : line;
199+
200+
return content.trim();
201+
}
202+
203+
function stripAnsi(value) {
204+
return value.replace(/\u001B\[[0-9;]*m/g, '');
205+
}
206+
207+
function isDocumentationIssue(message) {
208+
return DOC_MESSAGE_PATTERNS.some(function (pattern) {
209+
return pattern.test(message);
210+
});
211+
}
212+
213+
function groupByMessage(issues) {
214+
var groups = {};
215+
var sortedEntries;
216+
var index;
217+
218+
for (index = 0; index < issues.length; index += 1) {
219+
var issue = issues[index];
220+
if (!groups[issue.message]) {
221+
groups[issue.message] = [];
222+
}
223+
groups[issue.message].push(issue);
224+
}
225+
226+
sortedEntries = Object.entries(groups).sort(function (left, right) {
227+
if (right[1].length !== left[1].length) {
228+
return right[1].length - left[1].length;
229+
}
230+
231+
return left[0].localeCompare(right[0]);
232+
});
233+
234+
return sortedEntries;
235+
}
236+
237+
function printSummary(options) {
238+
var result = {
239+
inputPath: options.inputPath || '(stdin)',
240+
parsedIssueCount: options.parsedIssues.length,
241+
structuralIssueCount: options.filteredIssues.length,
242+
uniqueStructuralMessages: options.groupedIssues.length,
243+
issuesByMessage: options.groupedIssues.map(function (entry) {
244+
return {
245+
message: entry[0],
246+
count: entry[1].length,
247+
paths: entry[1].map(function (issue) {
248+
return issue.path;
249+
}),
250+
};
251+
}),
252+
};
253+
254+
if (options.json) {
255+
console.log(JSON.stringify(result, null, 2));
256+
return;
257+
}
258+
259+
console.log('Input: ' + result.inputPath);
260+
console.log('Parsed issues: ' + result.parsedIssueCount);
261+
console.log('Structural issues: ' + result.structuralIssueCount);
262+
console.log('Unique structural messages: ' + result.uniqueStructuralMessages);
263+
264+
if (result.structuralIssueCount === 0) {
265+
console.log('');
266+
console.log('No structural issues found.');
267+
return;
268+
}
269+
270+
console.log('');
271+
result.issuesByMessage.forEach(function (entry) {
272+
console.log(entry.count + 'x ' + entry.message);
273+
entry.paths.forEach(function (issuePath) {
274+
console.log(' - ' + issuePath);
275+
});
276+
console.log('');
277+
});
278+
}

0 commit comments

Comments
 (0)