CI Investigation: adopt dotnet/arcade-skills plugin + thin MAUI context skill by PureWeen · Pull Request #34438 · dotnet/maui

PureWeen · 2026-03-11T20:06:48Z

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Summary

This PR adopts the dotnet/arcade-skills plugin system for CI investigation in dotnet/maui, replacing the need for custom in-repo PowerShell scripts.

Two files are added:

.github/copilot/settings.json — repo-level plugin declaration that auto-installs the dotnet-dnceng plugin for all users
.github/skills/azdo-build-investigator/SKILL.md — thin MAUI-specific context supplement (~60 lines, no scripts)

Background & Motivation

The Problem

When investigating CI failures on dotnet/maui PRs, contributors and AI agents need to:

Query Azure DevOps builds across 3 pipelines (maui-pr, maui-pr-devicetests, maui-pr-uitests)
Dig into Helix test logs for device test failures
Analyze MSBuild binlogs for obscure build failures
Detect hidden test failures caused by XHarness exiting with code 0 even when tests fail

PR #34335 (feature/azdo-ci-instructions) addressed this with ~700 lines of custom PowerShell scripts. During review of that PR, we discovered dotnet/arcade-skills — a .NET engineering-maintained plugin that provides native MCP tooling for exactly this problem space, and already lists dotnet/maui as a supported repository.

Why arcade-skills Instead of Custom Scripts

The dotnet-dnceng plugin in arcade-skills provides:

MCP Server	Tool	Replaces
`ado-dnceng-public`	Native ADO queries via `@azure-devops/mcp`	`Get-BuildInfo.ps1`, `Get-BuildErrors.ps1`, `Get-PrBuildIds.ps1`
`hlx`	Helix test infrastructure via `lewing.helix.mcp`	`Get-HelixLogs.ps1`
`mcp-binlog-tool`	MSBuild binlog analysis via `baronfel.binlog.mcp`	`Get-BuildBinlogs.ps1` + manual `binlogtool`

MCP tools are first-class AI primitives — the AI calls them directly with structured parameters rather than running shell scripts and parsing text output. This is more reliable and maintainable.

Auto-Loading (No User Action Required)

The key mechanism that makes this work seamlessly:

.github/copilot/settings.json supports enabledPlugins (introduced in Copilot CLI v0.0.422) — a "declarative plugin auto-install" that runs at session startup when a user opens this repository. Every user who opens dotnet/maui gets the dotnet-dnceng plugin with all its MCP servers automatically. No /plugin install command needed.

{
  "extraKnownMarketplaces": [
    { "url": "https://github.com/dotnet/arcade-skills" }
  ],
  "enabledPlugins": ["dotnet-dnceng@dotnet-arcade-skills"]
}

What the MAUI Context Skill Adds

The arcade-skills ci-analysis skill is excellent but contains outdated MAUI-specific information (it lists maui-public as the pipeline name, which is wrong). The thin azdo-build-investigator/SKILL.md provides corrections and MAUI-specific domain knowledge:

Correct Pipeline Names/IDs

Pipeline	Definition ID	Purpose
`maui-pr`	302	Main build — check first
`maui-pr-devicetests`	314	Helix device tests
`maui-pr-uitests`	313	Appium UI tests

XHarness Exit-0 Blind Spot

XHarness (used in maui-pr-devicetests) exits with code 0 even when tests fail. The ADO job shows ✅ "Succeeded" while actual test failures hide inside Helix work items. The SKILL.md documents how to detect this via the Helix ResultSummaryByBuild API.

This quirk was discovered while investigating PRs with the s/agent-gate-failed label where CI appeared green but tests were actually failing.

Container Artifact Quirk for Binlogs

MAUI build artifacts are Container type (not PipelineArtifact), so standard az pipelines runs artifact download does not work for binlogs. The SKILL.md documents the correct download approach using the ADO File Container API with Bearer auth.

Relationship to PR #34335

PR #34335 (feature/azdo-ci-instructions) adds the same investigation capability via 5 custom PowerShell scripts. This PR supersedes that approach. The knowledge gained building those scripts (XHarness exit-0 discovery, Container artifact API approach, pipeline IDs) is preserved in the SKILL.md here.

We recommend closing #34335 in favor of this approach, which:

Has ~5% of the code to maintain
Uses MCP tooling that will improve over time as arcade-skills evolves
Auto-loads for all contributors without any setup

Files Changed

.github/copilot/settings.json                    (new) — repo-level plugin auto-install
.github/skills/azdo-build-investigator/SKILL.md  (new) — MAUI-specific CI context

Testing

Verified .github/copilot/settings.json schema matches Copilot CLI v0.0.422+ enabledPlugins format
Verified dotnet-dnceng@dotnet-arcade-skills resolves against the marketplace at https://github.com/dotnet/arcade-skills/.github/plugin/marketplace.json
SKILL.md pipeline IDs verified against live ADO builds: maui-pr=302, maui-pr-devicetests=314, maui-pr-uitests=313

/cc @PureWeen

Adopts dotnet/arcade-skills (dotnet-dnceng plugin) for CI investigation instead of maintaining custom PowerShell scripts in-repo. ## What this adds ### .github/copilot/settings.json Declares the dotnet-dnceng plugin as a repo-level dependency via enabledPlugins. This auto-installs the plugin for any user who opens the MAUI repository — no manual /plugin install required. The dotnet-dnceng plugin provides: - ado-dnceng-public MCP server (native ADO queries via @azure-devops/mcp) - hlx MCP server (Helix test infrastructure via lewing.helix.mcp) - mcp-binlog-tool MCP server (MSBuild binlog analysis via baronfel.binlog.mcp) - ci-analysis skill (full CI investigation workflow) ### .github/skills/azdo-build-investigator/SKILL.md Thin MAUI-specific context supplement (no scripts) that corrects and extends the arcade-skills ci-analysis skill with: - Correct MAUI pipeline names/IDs (arcade-skills has outdated 'maui-public') - maui-pr (302) - main build - maui-pr-devicetests (314) - Helix device tests - maui-pr-uitests (313) - Appium UI tests - Investigation priority order - XHarness exit-0 blind spot: XHarness exits 0 even when tests fail, requiring cross-check via Helix ResultSummaryByBuild API - Container artifact quirk for binlog download (not PipelineArtifact) ## Why this approach Previously we maintained 5 PowerShell scripts (~700 lines total) for ADO queries that are now better served by native MCP tooling. The arcade-skills dotnet-dnceng plugin is maintained by the .NET engineering team and supports dotnet/maui explicitly. The in-repo SKILL.md provides MAUI-specific domain knowledge that the generic arcade-skills ci-analysis skill doesn't have, and it auto-loads as a project skill without any user action. Related: dotnet/arcade-skills - https://github.com/dotnet/arcade-skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-11T20:06:58Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34438

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34438"

Copilot

Pull request overview

This PR adopts the dotnet/arcade-skills plugin system for CI investigation in dotnet/maui by enabling the dotnet-dnceng plugin repo-wide and adding a thin MAUI-specific skill that corrects/augments MAUI pipeline and investigation details.

Changes:

Add .github/copilot/settings.json to auto-enable the dotnet-dnceng@dotnet-arcade-skills plugin via the dotnet/arcade-skills marketplace.
Add .github/skills/azdo-build-investigator/SKILL.md to provide MAUI-specific CI investigation context (pipeline IDs/names, XHarness “exit 0” blind spot, container artifacts notes).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
.github/skills/azdo-build-investigator/SKILL.md	Adds MAUI-specific CI investigation guidance to complement the plugin’s `ci-analysis` skill.
.github/copilot/settings.json	Declares the external marketplace and enables auto-install/auto-enable for the `dotnet-dnceng` plugin.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-11T20:10:35Z

.github/skills/azdo-build-investigator/SKILL.md

+**How to detect hidden test failures**: Query the `ResultSummaryByBuild` Helix API endpoint:
+```
+GET https://helix.dot.net/api/2019-06-17/jobs/{correlationId}/aggregated
+```
+Look for `Failed` > 0 in the response even when the ADO build job shows green.
+
+When `ci-analysis` reports a `maui-pr-devicetests` build as passing but the PR has a `s/agent-gate-failed` label or the user suspects device test failures, always cross-check Helix `ResultSummaryByBuild`.


The Helix summary endpoint is shown with a {correlationId} placeholder and referred to as ResultSummaryByBuild, but elsewhere in this repo Helix identifiers are treated as a Helix jobId (GUID) extracted from logs (e.g., jobs/{guid}/workitems). To avoid confusion for users following this doc, rename the placeholder to {jobId} (or {helixJobId}) and clarify where to obtain it (ADO timeline logs / Helix submission output).

Copies the ci-analysis skill from dotnet/arcade-skills plugin into .github/skills/ci-analysis/ so it auto-loads for anyone cloning the repo without requiring '/plugin install'. - Vendored Get-CIStatus.ps1 script + 11 reference docs - Updated azdo-build-investigator to reference local ci-analysis - Script works standalone via gh CLI (no MCP servers required) Source: dotnet/arcade-skills@022d89a (dotnet-dnceng plugin) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace vendored ci-analysis skill files with enabledPlugins reference to dotnet/arcade-skills plugin. This allows the ci-analysis, darc, and skill-creator skills to load automatically from the remote plugin when Copilot CLI starts, with zero local files. - Update settings.json to use enabledPlugins object format - Remove 14 vendored ci-analysis files (SKILL.md, scripts, references) - Update azdo-build-investigator to download script on demand Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ld-investigator The pr-build-status skill (4 scripts, 624 lines) is fully superseded by the ci-analysis skill from dotnet/arcade-skills (2,275 lines, 25+ functions). Every capability (build IDs, stage status, errors, Helix logs) exists in ci-analysis with additional features like known-issue cross-referencing, PR file correlation, and structured JSON output. Changes: - Delete .github/skills/pr-build-status/ (SKILL.md + 4 scripts) - Simplify azdo-build-investigator SKILL.md (remove curl workaround, scripts auto-cached via enabledPlugins) - Update copilot-instructions.md, README-AI.md, pr-finalize references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 11, 2026 20:06

Copilot started reviewing on behalf of PureWeen March 11, 2026 20:07 View session

Copilot AI reviewed Mar 11, 2026

View reviewed changes

PureWeen marked this pull request as draft March 11, 2026 20:38

github-actions bot mentioned this pull request Mar 12, 2026

[repo-status] Daily Repo Status - March 12, 2026 🌟 #34443

Closed

github-actions bot and others added 3 commits March 12, 2026 12:59

PureWeen marked this pull request as ready for review March 13, 2026 16:20

github-actions bot mentioned this pull request Mar 15, 2026

[repo-status] Daily Repo Status - March 15, 2026 🌟 #34479

Closed

kubaflo approved these changes Mar 16, 2026

View reviewed changes

PureWeen merged commit e2f570f into main Mar 16, 2026
4 of 5 checks passed

PureWeen deleted the feature/arcade-skills-adoption branch March 16, 2026 16:45

This was referenced Mar 17, 2026

[repo-status] Daily Repo Status - March 17, 2026 🌟 #34507

Closed

[repo-status] 🌟 Daily Repo Status - March 20, 2026 #34578

Closed

[repo-status] 🌟 Daily Repo Status - March 21, 2026 #34588

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Investigation: adopt dotnet/arcade-skills plugin + thin MAUI context skill#34438

CI Investigation: adopt dotnet/arcade-skills plugin + thin MAUI context skill#34438
PureWeen merged 4 commits intomainfrom
feature/arcade-skills-adoption

PureWeen commented Mar 11, 2026

Uh oh!

github-actions bot commented Mar 11, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

PureWeen commented Mar 11, 2026

Summary

Background & Motivation

The Problem

Why arcade-skills Instead of Custom Scripts

Auto-Loading (No User Action Required)

What the MAUI Context Skill Adds

Correct Pipeline Names/IDs

XHarness Exit-0 Blind Spot

Container Artifact Quirk for Binlogs

Relationship to PR #34335

Files Changed

Testing

Uh oh!

github-actions bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Mar 11, 2026 •

edited

Loading