Initialize repository with checkin validation and ci-analysis skill by danmoseley · Pull Request #2 · dotnet/arcade-skills

danmoseley · 2026-03-10T22:51:16Z

Initialize repository infrastructure with checkin validation modeled on dotnet/skills, without AI-based evaluation. Also adds the first skill: ci-analysis from lewing/agent-plugins, which enables closing dotnet/runtime#125149.

What's included

ci-analysis skill (from lewing/agent-plugins)

The dotnet-dnceng/ci-analysis skill analyzes CI build status and test failures in Azure DevOps and Helix for dotnet repositories. Includes:

SKILL.md with full workflow documentation
Get-CIStatus.ps1 script (PR, build, and Helix analysis modes)
12 reference docs (analysis workflow, failure interpretation, binlog comparison, build progression, delegation patterns, etc.)
plugin.json with MCP server configurations (AzDO, Helix, binlog)

With this skill in arcade-skills, dotnet/runtime can reference it here instead of maintaining a local copy (see dotnet/runtime#125149).

Workflows

CODEOWNERS validation (codeowners-folder-validation.yml) -- copied verbatim from dotnet/skills. Ensures every skill/test folder has a CODEOWNERS entry with 2+ individuals or 1+ team. Creates GitHub issues for violations on main branch pushes.
Structural PR validation (pr-validation.yml) -- new, PowerShell-based. Posts a PR comment listing errors/warnings when validation fails. Validates all non-AI structural checks from dotnet/skills' skill-validator:
- plugin.json: name (kebab-case, matches directory), version, description, skills/agents path with traversal protection
- SKILL.md frontmatter: name + description per agentskills.io spec
- SKILL.md body: line count (max 500), token size analysis (chars/4), file reference depth (max 1 dir), structural warnings (sections, code blocks, steps)
- compatibility field validation (max 500 chars)
- eval.yaml schema: scenarios with name + prompt, known assertion types, per-assertion required fields (path, value, pattern), eval prompt bias detection
- Marketplace JSON consistency (both copies must match, entries point to real dirs)
- Orphaned test directories (tests/ without matching plugins/)
- Agent file frontmatter (.agent.md)

Repository structure

plugins/dotnet-dnceng/   # CI analysis plugin
  plugin.json
  skills/ci-analysis/    # SKILL.md, scripts/, references/
tests/                   # Test fixtures (empty, ready for content)
docs/                    # Documentation (empty)
.github/plugin/          # Plugin marketplace config
.claude-plugin/          # Claude plugin marketplace config (must match .github/plugin/)

Configuration

global.json -- .NET 10 SDK
.gitattributes -- LF line endings
dependabot.yml -- weekly GitHub Actions updates
CODEOWNERS -- @lewing @danmoseley @ViktorHofer (default + workflows), @lewing @danmoseley (ci-analysis skill)
LICENSE -- MIT

Documentation

README.md, CONTRIBUTING.md (adapted from dotnet/skills, no AI eval references), AGENTS.md

What's NOT included (by design)

No AI evaluation pipeline (evaluation.yml, evaluation-run.yml, evaluation-fork-pr.yml)
No skill-validator tool (eng/skill-validator/)
No benchmark dashboard (eng/dashboard/)
No agentic workflows (devops-health-*, .agents/, agentic-workflows/)

These can be added later when AI evaluation is desired.

Action needed

Verify GitHub Actions is enabled on this repo (Settings > Actions > General)
Set workflow permissions to "Read and write" (needed for CODEOWNERS issue creation)

Add checkin validation modeled on dotnet/skills (without AI evaluation): - CODEOWNERS validation workflow (from dotnet/skills, verbatim) - Structural PR validation workflow: plugin.json, SKILL.md frontmatter, eval.yaml schema, marketplace.json consistency, orphaned test detection - Plugin marketplace configuration (.github/plugin + .claude-plugin) - Dependabot for GitHub Actions updates - Repository docs: README, CONTRIBUTING, AGENTS, LICENSE - Root configs: global.json (.NET 10 SDK), .gitattributes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds all non-AI validation from dotnet/skills' skill-validator: - Path traversal protection on skills/agents paths in plugin.json - Compatibility field validation (<=500 chars) - Body line count limit (<=500 lines) - Token size analysis (chars/4 approximation) with complexity warnings - File reference depth check (max 1 dir deep, no parent traversal) - Structural warnings (sections, code blocks, numbered steps) - Eval prompt bias detection (skill name in scenario prompt) - Per-assertion required fields (path, value, pattern per type) - Unknown assertion type detection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds the dotnet-dnceng plugin with the ci-analysis skill, synced from lewing/agent-plugins. This skill analyzes CI build status and test failures in Azure DevOps and Helix for dotnet repositories. Includes: - SKILL.md with full workflow documentation - Get-CIStatus.ps1 script (PR, build, and Helix analysis modes) - 12 reference docs (analysis workflow, failure interpretation, binlog comparison, build progression, delegation patterns, etc.) - plugin.json with MCP server configurations - Updated marketplace.json to register the plugin This enables dotnet/runtime and other repos to reference ci-analysis from dotnet/arcade-skills instead of maintaining local copies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@lewing

Adds @lewing and @danmoseley as code owners for the ci-analysis skill, workflows, and repo default. Reverts the deliberate SKILL.md frontmatter violation used to verify pr-validation catches errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When structural validation finds errors or warnings, posts a comment on the PR listing them (updates existing comment on re-push). Uses a marker comment for idempotent updates. Follows the same pattern as dotnet/skills' CODEOWNERS validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-10T22:58:35Z

Warnings (1)

plugins/dotnet-dnceng/skills/ci-analysis — ~2863 tokens (chars/4) — approaching 'comprehensive' range where gains diminish.

View workflow run

Copilot

Pull request overview

Initializes the repository scaffolding for “arcade-skills” with structural validation workflows and a first plugin (dotnet-dnceng) containing the ci-analysis skill for analyzing Azure DevOps + Helix CI failures.

Changes:

Adds the dotnet-dnceng plugin with MCP server configuration and the ci-analysis skill (script + references).
Introduces PR structural validation and CODEOWNERS folder validation GitHub workflows.
Adds baseline repo configuration/docs (README, contributing guidance, global.json, marketplaces, CODEOWNERS, dependabot, license).

Reviewed changes

Copilot reviewed 27 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/.gitkeep	Placeholder to keep `tests/` directory in the repo.
docs/.gitkeep	Placeholder to keep `docs/` directory in the repo.
plugins/dotnet-dnceng/plugin.json	Defines the `dotnet-dnceng` plugin and MCP server configurations.
plugins/dotnet-dnceng/skills/ci-analysis/SKILL.md	Skill definition and operating guidance for CI analysis.
plugins/dotnet-dnceng/skills/ci-analysis/scripts/Get-CIStatus.ps1	Main CI analysis PowerShell script for AzDO/Helix/gh queries and summary output.
plugins/dotnet-dnceng/skills/ci-analysis/references/analysis-workflow.md	Step-by-step workflow for using the skill output.
plugins/dotnet-dnceng/skills/ci-analysis/references/azdo-helix-reference.md	Reference on AzDO/Helix orgs, repos, and common pipelines.
plugins/dotnet-dnceng/skills/ci-analysis/references/azure-cli.md	Azure CLI fallback guidance for deeper investigations.
plugins/dotnet-dnceng/skills/ci-analysis/references/binlog-comparison.md	Pattern for binlog comparison and delegation guidance.
plugins/dotnet-dnceng/skills/ci-analysis/references/build-progression-analysis.md	Pattern for PR build progression analysis and tracking.
plugins/dotnet-dnceng/skills/ci-analysis/references/delegation-patterns.md	Subagent delegation patterns for scaling investigations.
plugins/dotnet-dnceng/skills/ci-analysis/references/failure-interpretation.md	Guidance for classifying and validating CI failure causes.
plugins/dotnet-dnceng/skills/ci-analysis/references/helix-artifacts.md	How to locate and interpret Helix work item artifacts.
plugins/dotnet-dnceng/skills/ci-analysis/references/manual-investigation.md	Manual commands for investigation when automation is insufficient.
plugins/dotnet-dnceng/skills/ci-analysis/references/recommendation-generation.md	How to synthesize recommendations from the script’s JSON summary.
plugins/dotnet-dnceng/skills/ci-analysis/references/script-modes.md	Documents script modes and key parameters.
plugins/dotnet-dnceng/skills/ci-analysis/references/sql-tracking.md	SQL tracking schemas/queries for complex investigations.
global.json	Pins .NET SDK version and roll-forward behavior.
README.md	Repo intro and installation instructions.
LICENSE	Adds MIT license text.
CONTRIBUTING.md	Contribution guidance and repo conventions for plugins/skills/tests.
AGENTS.md	High-level repo structure and validation overview.
.github/workflows/pr-validation.yml	PowerShell-based structural validation workflow for PRs and main pushes.
.github/workflows/codeowners-folder-validation.yml	Validates CODEOWNERS coverage/ownership requirements for skill/test folders.
.github/plugin/marketplace.json	GitHub plugin marketplace definition.
.claude-plugin/marketplace.json	Claude plugin marketplace definition (intended to match GitHub marketplace).
.github/dependabot.yml	Dependabot configuration for GitHub Actions updates.
.github/CODEOWNERS	CODEOWNERS defaults + explicit skill ownership entry.
.gitattributes	Enforces LF endings via `eol=lf`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-10T23:03:12Z

plugins/dotnet-dnceng/skills/ci-analysis/scripts/Get-CIStatus.ps1

+                # Fetch console log
+                $consoleUrl = "https://helix.dot.net/api/2019-06-17/jobs/$HelixJob/workitems/$WorkItem/console"
+                Write-Host "`n  Console Log: $consoleUrl" -ForegroundColor Yellow
+
+                $consoleLog = Get-HelixConsoleLog -Url $consoleUrl
+                if ($consoleLog) {


$consoleUrl interpolates $WorkItem without URL-encoding. Helix work item names can contain characters that need escaping, which would make the console log fetch fail in -HelixJob -WorkItem mode. Use the same [uri]::EscapeDataString($WorkItem) approach used elsewhere when building the console URL.

Copilot · 2026-03-10T23:03:13Z

.github/CODEOWNERS

+# Default owners for everything in the repo
+* @lewing @danmoseley @viktorhofer
+


PR description says .github/CODEOWNERS is "blank owners (to be filled in)", but this change adds concrete owners (@lewing @danmoseley @viktorhofer). Either update the PR description/action checklist or adjust the CODEOWNERS file to match the stated intent.

Copilot · 2026-03-10T23:03:13Z

README.md

+| *(no plugins yet)* | Contribute one! See [CONTRIBUTING.md](CONTRIBUTING.md). |
+


The "What's Included" table claims there are no plugins yet, but this PR adds the dotnet-dnceng plugin (and marketplace entries). Update the table to list the new plugin so the README matches the repo contents.

Suggested change

| *(no plugins yet)* | Contribute one! See [CONTRIBUTING.md](CONTRIBUTING.md). |

| `dotnet-dnceng` | CI infrastructure and engineering system tools for .NET repos. |

| *(more coming soon)* | Contribute one! See [CONTRIBUTING.md](CONTRIBUTING.md). |

Copilot · 2026-03-10T23:03:13Z

plugins/dotnet-dnceng/skills/ci-analysis/scripts/Get-CIStatus.ps1

+    # Handle direct Helix job query
+    if ($PSCmdlet.ParameterSetName -eq 'HelixJob') {
+        Write-Host "`n=== Helix Job $HelixJob ===" -ForegroundColor Yellow
+        Write-Host "URL: https://helix.dot.net/api/jobs/$HelixJob" -ForegroundColor Gray


In HelixJob mode the script prints https://helix.dot.net/api/jobs/$HelixJob, but the rest of the script uses the versioned Helix API (/api/2019-06-17/...). The printed URL is likely incorrect/misleading; print the versioned API URL (or the Helix portal URL) consistently.

Suggested change

Write-Host "URL: https://helix.dot.net/api/jobs/$HelixJob" -ForegroundColor Gray

Write-Host "URL: https://helix.dot.net/api/2019-06-17/jobs/$HelixJob" -ForegroundColor Gray

- URL-encode WorkItem in Helix console URL to handle special chars - Use versioned Helix API URL consistently (2019-06-17) - Update README plugin table to list dotnet-dnceng - (PR description updated separately to reflect CODEOWNERS state) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

danmoseley and others added 7 commits March 10, 2026 16:10

TEST: introduce deliberate spec violations (will revert)

e408398

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add @ViktorHofer to CODEOWNERS

7f1d982

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

danmoseley requested a review from Copilot March 10, 2026 22:59

Copilot started reviewing on behalf of danmoseley March 10, 2026 22:59 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

ViktorHofer approved these changes Mar 10, 2026

View reviewed changes

lewing approved these changes Mar 10, 2026

View reviewed changes

danmoseley merged commit 022d89a into main Mar 11, 2026
3 checks passed

danmoseley deleted the init-infrastructure branch March 11, 2026 02:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initialize repository with checkin validation and ci-analysis skill#2

Initialize repository with checkin validation and ci-analysis skill#2
danmoseley merged 8 commits intomainfrom
init-infrastructure

danmoseley commented Mar 10, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 10, 2026

Uh oh!

Copilot AI Mar 10, 2026

Uh oh!

Copilot AI Mar 10, 2026

Uh oh!

Copilot AI Mar 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		# Default owners for everything in the repo
		* @lewing @danmoseley @viktorhofer

		\| (no plugins yet) \| Contribute one! See [CONTRIBUTING.md](CONTRIBUTING.md). \|

	\| (no plugins yet) \| Contribute one! See [CONTRIBUTING.md](CONTRIBUTING.md). \|
	\| `dotnet-dnceng` \| CI infrastructure and engineering system tools for .NET repos. \|
	\| (more coming soon) \| Contribute one! See [CONTRIBUTING.md](CONTRIBUTING.md). \|

	Write-Host "URL: https://helix.dot.net/api/jobs/$HelixJob" -ForegroundColor Gray
	Write-Host "URL: https://helix.dot.net/api/2019-06-17/jobs/$HelixJob" -ForegroundColor Gray

Conversation

danmoseley commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's included

ci-analysis skill (from lewing/agent-plugins)

Workflows

Repository structure

Configuration

Documentation

What's NOT included (by design)

Action needed

Uh oh!

github-actions bot commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Warnings (1)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

danmoseley commented Mar 10, 2026 •

edited

Loading

github-actions bot commented Mar 10, 2026 •

edited

Loading