Skip to content

[skills] Add AZDO build investigator skill#10885

Merged
jonathanpeppers merged 4 commits intomainfrom
dev/peppers/az-devops-skill
Mar 4, 2026
Merged

[skills] Add AZDO build investigator skill#10885
jonathanpeppers merged 4 commits intomainfrom
dev/peppers/az-devops-skill

Conversation

@jonathanpeppers
Copy link
Copy Markdown
Member

Add a Copilot skill for investigating Azure DevOps pipeline build failures. Given a build URL, it fetches run details, identifies failed jobs/tasks via the timeline API, downloads .binlog artifacts, and analyzes them with binlogtool to find root causes.

Example usage:

❯ what are the errors here: https://github.com/dotnet/android/pull/10881

◐ The user is asking about errors in a GitHub PR. This is a CI/build failure investigation task. There's an `azdo-build-investigator` skill available that can investigate Azure DevOps pipeline build failures
and analyze .binlog files. However, this is a GitHub PR URL, not an AZDO URL directly. Let me check if the skill can handle GitHub PR URLs - the skill description says "Use this when users share an AZDO
build URL, a GitHub PR URL, or ask about a failing pipeline, build errors, or CI failures." So yes, this skill can handle GitHub PR URLs.

Let me invoke the `azdo-build-investigator` skill.

● skill(azdo-build-investigator)

◐ Let me follow the skill workflow to investigate the GitHub PR. The URL is https://github.com/dotnet/android/pull/10881.

First, let me check if `gh` CLI and `az` CLI are available, and then get the failing checks from the PR.

● Get failing checks from PR #10881
$ gh pr checks 10881 --repo dotnet/android --json "name,state,link,bucket" 2>&1
└ 39 lines...

◐ There's one failing check: `Xamarin.Android-PR` with a link to `https://devdiv.visualstudio.com/0bdbc590-a062-4c3f-b0f6-9383f67865ee/_build/results?buildId=13452282`. Let me parse this:
- Org: `devdiv` (or `devdiv.visualstudio.com`)
- Project: `0bdbc590-a062-4c3f-b0f6-9383f67865ee`
- Build ID: `13452282`

Let me get the build timeline.

● Get failed build timeline records
$ az devops invoke --area build --resource timeline --route-parameters buildId=13452282 --org https://devdiv.visualstudio.com --project 0bdbc590-a062-4c3f-b0f6-9383f67865ee --query "records[?result=='failed']
| [].{name:name, type:type, issues:issues, errorCount:errorCount}" --output json 2>&1
└ 6 lines...

● Get failed build timeline records with project as route param
$ az devops invoke --area build --resource timeline --route-parameters project=0bdbc590-a062-4c3f-b0f6-9383f67865ee buildId=13452282 --org https://devdiv.visualstudio.com --query "records[?result=='failed'] |
[].{name:name, type:type, issues:issues, errorCount:errorCount}" --output json 2>&1
└ 182 lines...

◐ Now I have a clear picture of the failure. The `Xamarin.Android-PR` check is failing in the `Pack .NET Maui` task with 78 RS0016 errors. These are all Roslyn public API analyzer errors — the new symbols
added by this PR are not registered in the declared public API files.

The affected files are in `src\AI\src\Essentials.AI\Platform\MaciOS\`:
1. `NLEmbeddingExtensions.cs` - `NLEmbeddingExtensions` class and `AsIEmbeddingGenerator` method
2. `NLEmbeddingGenerator.cs` - `NLEmbeddingGenerator` class, constructor, `GenerateAsync`, `Dispose`

3. `AppleIntelligenceChatClient.cs` - `AppleIntelligenceChatClient` class, constructor, `GetResponseAsync`, `GetStreamingResponseAsync`

The root cause is that these new public types and members need to be registered in the `PublicAPI.Shipped.txt` or `PublicAPI.Unshipped.txt` file tracked by the `[MAUIAI0001]` analyzer — this is actually a
MAUI repo issue where the integration build is failing. The fix requires updating the public API declaration file.

● The failing check is Xamarin.Android-PR — specifically the "Pack .NET Maui" task, with 78 RS0016 errors.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Root Cause: Missing Public API Declarations

The new Apple Intelligence / MAUI AI types added by this PR are not registered in the public API surface file. All errors are RS0016 from the Roslyn PublicApiAnalyzers:

"Symbol is not part of the declared public API"

The affected symbols are all in src\AI\src\Essentials.AI\Platform\MaciOS\ (a MAUI repo path, not this repo):

┌──────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ File                             │ New Public Symbols                                                  │
├──────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ AppleIntelligenceChatClient.cs   │ Class, constructor, GetResponseAsync, GetStreamingResponseAsync     │
├──────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ NLEmbeddingGenerator.cs          │ Class, constructor, GenerateAsync, Dispose                          │
├──────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ NLEmbeddingExtensions.cs         │ Class, AsIEmbeddingGenerator extension method                       │
└──────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

Fix

The PublicAPI.Unshipped.txt file for the [MAUIAI0001] project needs to be updated to include all these new symbols. This is done in the MAUI repo — you need to add entries like:

[MAUIAI0001]Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient
[MAUIAI0001]Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.AppleIntelligenceChatClient() -> void
[MAUIAI0001]Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.GetResponseAsync(...) -> ...
...

to the PublicAPI.Unshipped.txt file in that project. The full list of symbols to add can be found in the 78 error messages from the build log.

Add a Copilot skill for investigating Azure DevOps pipeline build
failures. Given a build URL, it fetches run details, identifies failed
jobs/tasks via the timeline API, downloads .binlog artifacts, and
analyzes them with binlogtool to find root causes.

Example usage:

    ❯ what are the errors here: #10881

    ◐ The user is asking about errors in a GitHub PR. This is a CI/build failure investigation task. There's an `azdo-build-investigator` skill available that can investigate Azure DevOps pipeline build failures
    and analyze .binlog files. However, this is a GitHub PR URL, not an AZDO URL directly. Let me check if the skill can handle GitHub PR URLs - the skill description says "Use this when users share an AZDO
    build URL, a GitHub PR URL, or ask about a failing pipeline, build errors, or CI failures." So yes, this skill can handle GitHub PR URLs.

    Let me invoke the `azdo-build-investigator` skill.

    ● skill(azdo-build-investigator)

    ◐ Let me follow the skill workflow to investigate the GitHub PR. The URL is #10881.

    First, let me check if `gh` CLI and `az` CLI are available, and then get the failing checks from the PR.

    ● Get failing checks from PR #10881
    $ gh pr checks 10881 --repo dotnet/android --json "name,state,link,bucket" 2>&1
    └ 39 lines...

    ◐ There's one failing check: `Xamarin.Android-PR` with a link to `https://devdiv.visualstudio.com/0bdbc590-a062-4c3f-b0f6-9383f67865ee/_build/results?buildId=13452282`. Let me parse this:
    - Org: `devdiv` (or `devdiv.visualstudio.com`)
    - Project: `0bdbc590-a062-4c3f-b0f6-9383f67865ee`
    - Build ID: `13452282`

    Let me get the build timeline.

    ● Get failed build timeline records
    $ az devops invoke --area build --resource timeline --route-parameters buildId=13452282 --org https://devdiv.visualstudio.com --project 0bdbc590-a062-4c3f-b0f6-9383f67865ee --query "records[?result=='failed']
    | [].{name:name, type:type, issues:issues, errorCount:errorCount}" --output json 2>&1
    └ 6 lines...

    ● Get failed build timeline records with project as route param
    $ az devops invoke --area build --resource timeline --route-parameters project=0bdbc590-a062-4c3f-b0f6-9383f67865ee buildId=13452282 --org https://devdiv.visualstudio.com --query "records[?result=='failed'] |
    [].{name:name, type:type, issues:issues, errorCount:errorCount}" --output json 2>&1
    └ 182 lines...

    ◐ Now I have a clear picture of the failure. The `Xamarin.Android-PR` check is failing in the `Pack .NET Maui` task with 78 RS0016 errors. These are all Roslyn public API analyzer errors — the new symbols
    added by this PR are not registered in the declared public API files.

    The affected files are in `src\AI\src\Essentials.AI\Platform\MaciOS\`:
    1. `NLEmbeddingExtensions.cs` - `NLEmbeddingExtensions` class and `AsIEmbeddingGenerator` method
    2. `NLEmbeddingGenerator.cs` - `NLEmbeddingGenerator` class, constructor, `GenerateAsync`, `Dispose`

    3. `AppleIntelligenceChatClient.cs` - `AppleIntelligenceChatClient` class, constructor, `GetResponseAsync`, `GetStreamingResponseAsync`

    The root cause is that these new public types and members need to be registered in the `PublicAPI.Shipped.txt` or `PublicAPI.Unshipped.txt` file tracked by the `[MAUIAI0001]` analyzer — this is actually a
    MAUI repo issue where the integration build is failing. The fix requires updating the public API declaration file.

    ● The failing check is Xamarin.Android-PR — specifically the "Pack .NET Maui" task, with 78 RS0016 errors.

    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Root Cause: Missing Public API Declarations

    The new Apple Intelligence / MAUI AI types added by this PR are not registered in the public API surface file. All errors are RS0016 from the Roslyn PublicApiAnalyzers:

    "Symbol is not part of the declared public API"

    The affected symbols are all in src\AI\src\Essentials.AI\Platform\MaciOS\ (a MAUI repo path, not this repo):

    ┌──────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
    │ File                             │ New Public Symbols                                                  │
    ├──────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
    │ AppleIntelligenceChatClient.cs   │ Class, constructor, GetResponseAsync, GetStreamingResponseAsync     │
    ├──────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
    │ NLEmbeddingGenerator.cs          │ Class, constructor, GenerateAsync, Dispose                          │
    ├──────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
    │ NLEmbeddingExtensions.cs         │ Class, AsIEmbeddingGenerator extension method                       │
    └──────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

    Fix

    The PublicAPI.Unshipped.txt file for the [MAUIAI0001] project needs to be updated to include all these new symbols. This is done in the MAUI repo — you need to add entries like:

    [MAUIAI0001]Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient
    [MAUIAI0001]Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.AppleIntelligenceChatClient() -> void
    [MAUIAI0001]Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.GetResponseAsync(...) -> ...
    ...

    to the PublicAPI.Unshipped.txt file in that project. The full list of symbols to add can be found in the 78 error messages from the build log.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 4, 2026 17:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new GitHub Copilot skill to help investigate Azure DevOps pipeline build failures by resolving PR URLs to failing checks, querying timelines/logs via Azure DevOps APIs, and analyzing .binlog artifacts with binlogtool.

Changes:

  • Introduces a new azdo-build-investigator skill definition and workflow documentation.
  • Documents prerequisites (gh, az + azure-devops extension, binlogtool) and a step-by-step investigation process.
  • Provides common error-pattern heuristics tailored to the dotnet/android repo.

Comment thread .github/skills/azdo-build-investigator/SKILL.md Outdated
Comment thread .github/skills/azdo-build-investigator/SKILL.md Outdated
Comment thread .github/skills/azdo-build-investigator/SKILL.md
jonathanpeppers and others added 2 commits March 4, 2026 11:43
- Add project to --route-parameters for timeline and logs endpoints
- Use {orgUrl} extracted from input instead of hardcoding dev.azure.com
- Condense skill content to reduce token usage

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers jonathanpeppers added the copilot `copilot-cli` or other AIs were used to author this label Mar 4, 2026
@jonathanpeppers jonathanpeppers requested a review from Copilot March 4, 2026 18:08
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Comment thread .github/skills/azdo-build-investigator/SKILL.md Outdated
Comment thread .github/skills/azdo-build-investigator/SKILL.md Outdated
- Clarify prerequisite requirements (az always, gh/binlogtool conditional)
- Fix logs endpoint to save raw text to file instead of requesting JSON

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

@jonathanpeppers jonathanpeppers merged commit fe5138d into main Mar 4, 2026
5 of 9 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Apr 4, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

copilot `copilot-cli` or other AIs were used to author this

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants