fix(audit): surface activation job errors when no agent artifacts were uploaded#19678
fix(audit): surface activation job errors when no agent artifacts were uploaded#19678
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…tivation job failure diagnosis Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Fixes missing audit diagnostics for workflow runs that fail before agent artifacts are uploaded by downloading workflow run logs even when gh run download reports no artifacts.
Changes:
- Update
downloadRunArtifactsto attemptdownloadWorkflowRunLogson theErrNoArtifactspath and only delete the output directory if it’s still empty. - Simplify the
actions/setup-cli/install.shgh extension installflow and adjust the GitHub Actions output value. - Bump pinned versions in
constraint-solving-potd.lock.yml(agent version, mcpg image) and add aconcurrencyblock for theconclusionjob.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| pkg/cli/logs_download.go | Falls back to downloading workflow logs when no artifacts are present, enabling pre-agent error extraction. |
| actions/setup-cli/install.sh | Changes how gh-aw is installed via gh extension install and what version is reported to Actions outputs. |
| .github/workflows/constraint-solving-potd.lock.yml | Updates pinned tool/image versions and adds conclusion-job concurrency. |
Comments suppressed due to low confidence (1)
actions/setup-cli/install.sh:253
installed_versionis being set to the requested${VERSION}rather than the detected${INSTALLED_VERSION}fromgh aw version. This can report an incorrect value (e.g., when${VERSION}islatestor whengh extension installinstalls a different version than requested). Prefer writing${INSTALLED_VERSION}here.
# Set output for GitHub Actions
if [ -n "${GITHUB_OUTPUT}" ]; then
echo "installed_version=${VERSION}" >> "${GITHUB_OUTPUT}"
fi
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
|
|
||
| if [ $install_result -eq 0 ]; then | ||
| # Try to install using gh | ||
| if gh extension install "$REPO" --force 2>&1 | tee /tmp/gh-install.log; then |
There was a problem hiding this comment.
The gh extension install ... | tee ... pipeline will report success based on tee’s exit code (since this script only uses set -e, not set -o pipefail). This can mask install failures and incorrectly skip the manual-install fallback (especially if an older gh-aw is already installed). Consider enabling set -o pipefail for this block or capturing the PIPESTATUS[0] from the gh extension install command explicitly.
| if gh extension install "$REPO" --force 2>&1 | tee /tmp/gh-install.log; then | |
| gh extension install "$REPO" --force 2>&1 | tee /tmp/gh-install.log | |
| gh_install_status=${PIPESTATUS[0]} | |
| if [ "$gh_install_status" -eq 0 ]; then |
| # Try to install using gh | ||
| if gh extension install "$REPO" --force 2>&1 | tee /tmp/gh-install.log; then |
There was a problem hiding this comment.
This gh extension install path no longer respects the requested $VERSION (it always installs the latest) even though the script supports ./install-gh-aw.sh vX.Y.Z. If version pinning is required, restore the --pin "$VERSION" behavior (when $VERSION != latest) or otherwise ensure the installed version matches the requested one.
This issue also appears on line 250 of the same file.
When a workflow fails at
activation(before the agent runs), no artifacts are uploaded, sogh run downloadreturnsErrNoArtifacts. The original code deleted the output directory and returned early, leavingextractPreAgentStepErrorswith noworkflow-logs/to scan — producing an audit report with zero error details.Changes
pkg/cli/logs_download.go— In theErrNoArtifactspath ofdownloadRunArtifacts, calldownloadWorkflowRunLogsbefore returning. Workflow run logs are always available via the GitHub API regardless of artifact upload status. Only remove the output directory if log download also fails and leaves the directory empty.With
workflow-logs/{job}/populated,extractPreAgentStepErrorscan scan##[error]annotations from activation step logs and populateAuditData.Errorswith the actual failure reason.Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
https://api.github.com/graphql/usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GO111MODULE x_amd64/link git rev-�� --show-toplevel x_amd64/link /usr/bin/git -json GO111MODULE 64/bin/go git(http block)https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1/usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq .object.sha --get remote.origin.url /usr/bin/git -json GO111MODULE 64/bin/go git log --oneline -1 /usr/bin/git -json GO111MODULE 64/bin/go git(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v3/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq .object.sha runs/20260305-032257-28002/test-463592499/.github/workflows GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go 2742�� -json 27421/b402/_testmain.go om/testorg/testrepo.git GOINSECURE GOMOD GOMODCACHE /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/link(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v5/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha user.name Test User /usr/bin/git che/go-build/d9/git GOPROXY 64/bin/go git rev-�� --show-toplevel /opt/hostedtoolcache/go/1.25.0/xGO111MODULE /usr/bin/git /tmp/go-build133git -trimpath 64/bin/go git(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha --show-toplevel go /usr/bin/git 42/001/test-emptgit GO111MODULE ache/go/1.25.0/x--show-toplevel git rev-�� --show-toplevel go /usr/bin/git -json GO111MODULE ache/go/1.25.0/x--show-toplevel git(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v6/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha --show-toplevel ortcfg fg g/styles/theme.ggit g/styles/theme_trev-parse 64/bin/go git rev-�� --show-toplevel ache/go/1.25.0/xGO111MODULE /usr/bin/git 4370605/b411/_pkgit GO111MODULE 64/bin/go git(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha --get remote.origin.url /usr/bin/git go1.25.0 -c=4 -nolocalimports git conf�� --get remote.origin.url /usr/bin/git 4370605/b389/_pkgit GO111MODULE 64/bin/go git(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha --show-toplevel go /usr/bin/git -json GO111MODULE 64/bin/go git rev-�� --show-toplevel go /usr/bin/infocmp -json GO111MODULE 64/pkg/tool/linu--show-toplevel infocmp(http block)https://api.github.com/repos/actions/download-artifact/git/ref/tags/v8/usr/bin/gh gh api /repos/actions/download-artifact/git/ref/tags/v8 --jq .object.sha -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 2257-28002/test-493784196/.github/workflows GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/download-artifact/git/ref/tags/v8 --jq .object.sha -json GO111MODULE .a GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet(http block)/usr/bin/gh gh api /repos/actions/download-artifact/git/ref/tags/v8 --jq .object.sha -json main.go /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/actions/github-script/git/ref/tags/v8/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE go k/gh�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD erignore go(http block)/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha -c=4 -nolocalimports -importcfg /tmp/go-build57527421/b374/importcfg -pack /home/REDACTED/work/gh-aw/gh-aw/pkg/cli/access_log.go /home/REDACTED/work/gh-aw/gh-aw/pkg/cli/actionlint.go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha b7505c45f80cd646GOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE ache/go/1.25.0/xGO111MODULE env 4370605/b393/_pkGOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/actions/setup-go/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq .object.sha /tmp/gh-aw-test-runs/20260305-032257-28002/test-2759002707 rev-parse .a g/timeutil/formagit g/timeutil/formarev-parse 64/bin/go /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/vet -ato�� -bool -buildtags ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet -errorsas -ifaceassert -nilfunc ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet(http block)https://api.github.com/repos/actions/setup-node/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha /tmp/gh-aw-test-runs/20260305-032257-28002/test-2759002707 rev-parse /usr/bin/git -c=4 -nolocalimports -importcfg git init�� /home/REDACTED/work/gh-aw/gh-aw/pkg/fileutil/fileutil.go _UliyopLjxfi ache/go/1.25.0/x64/pkg/tool/linux_amd64/compile 4370605/b413/_pkgit GO111MODULE 64/bin/go ache/go/1.25.0/x64/pkg/tool/linux_amd64/compile(http block)https://api.github.com/repos/actions/upload-artifact/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v4 --jq .object.sha SameOutput268600355/001/stability-test.md GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/actions/upload-artifact/git/ref/tags/v7/usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v7 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v7 --jq .object.sha -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 355/001/stability-test.md GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v7 --jq .object.sha -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts/usr/bin/gh gh run download 1 --dir test-logs/run-1(http block)/usr/bin/gh gh run download 1 --dir test-logs/run-1 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go tlog�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts/usr/bin/gh gh run download 12345 --dir test-logs/run-12345(http block)/usr/bin/gh gh run download 12345 --dir test-logs/run-12345 GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE 4X/C061xVfYOKCjcDjyPxD-/1pjaMAEPwhrn9NjtCOgd(http block)https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts/usr/bin/gh gh run download 12346 --dir test-logs/run-12346(http block)/usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts/usr/bin/gh gh run download 2 --dir test-logs/run-2(http block)/usr/bin/gh gh run download 2 --dir test-logs/run-2 GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE qP/zDIGdj0VU87CwGYky17F/6TtT44wK-test.v=true(http block)https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts/usr/bin/gh gh run download 3 --dir test-logs/run-3(http block)/usr/bin/gh gh run download 3 --dir test-logs/run-3 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go tlog�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts/usr/bin/gh gh run download 4 --dir test-logs/run-4(http block)/usr/bin/gh gh run download 4 --dir test-logs/run-4 GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts/usr/bin/gh gh run download 5 --dir test-logs/run-5(http block)/usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go tlog�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/workflows/usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100(http block)/usr/bin/gh gh workflow list --json name,state,path GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE go env ck 'scripts/**/*GOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE sh(http block)https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0/usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq .object.sha y-test.md GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.2.3/usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE sh(http block)https://api.github.com/repos/github/gh-aw/git/ref/tags/v2.0.0/usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE node(http block)/usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq .object.sha e=false GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE node(http block)https://api.github.com/repos/github/gh-aw/git/ref/tags/v3.0.0/usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq .object.sha ript formatting GOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE node(http block)https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999/usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env 4068364924/.github/workflows GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/nonexistent/repo/actions/runs/12345/usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/owner/repo/actions/workflows/usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go GOINSECURE GOMOD GOMODCACHE go env ck 'scripts/**/*GOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/owner/repo/contents/file.md/tmp/go-build57527421/b383/cli.test /tmp/go-build57527421/b383/cli.test -test.testlogfile=/tmp/go-build57527421/b383/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true GOINSECURE GOMOD GOMODCACHE ortcfg env g/stringutil/ansGOINSECURE g/stringutil/ideGOMOD 64/bin/go GOINSECURE GOMOD ode-gyp-bin/sh ache/go/1.25.0/xGO111MODULE(http block)https://api.github.com/repos/test-owner/test-repo/actions/secrets/usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE go ache�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)If you need me to access, download, or install something from one of these locations, you can either:
Original prompt
This section details on the original issue you should resolve
<issue_title>[cli-tools-test] audit: no diagnostic details shown for workflows that fail at the activation job</issue_title>
<issue_description>## Problem Description
When auditing a workflow run that fails before the agent job runs (e.g., at the
activationorpre_activationjob), theaudittool returns no downloaded files and no error details, making it impossible to debug the failure.Affected Tool
audit(MCPagenticworkflows-audit)22693958004,22690865467,22691884770,22689634900,22694877580(all "Issue Monster" failures)Steps to Reproduce
activationjob (not theagentjob)gh aw audit (run_id)or use theagenticworkflows-auditMCP toolExpected Behavior
The audit report should include:
activationjob stepactivationjob)Actual Behavior
The audit returns:
downloaded_files: null— no files are downloaded"Workflow 'Issue Monster' failed with 1 error(s)"— no specifics"Check the Errors section below for specific error messages"— but there is no Errors section{ "metrics": { "error_count": 1 }, "key_findings": [{ "title": "Workflow Failed", "description": "Workflow 'Issue Monster' failed with 1 error(s)", "severity": "critical" }], "recommendations": [{ "action": "Review error logs to identify root cause of failure", "example": "Check the Errors section below for specific error messages and file locations" }], "downloaded_files": null }Root Cause
The
auditcommand usesgh run downloadto fetch uploaded GitHub Actions artifacts. When a workflow fails before the agent job (e.g., activation fails), no agent artifacts are uploaded — sogh run downloadreturnsErrNoArtifacts. The audit then proceeds with an emptyrunOutputDir, soextractDownloadedFiles()returns nothing.The existing code in
audit_report.go(line 251–255) attempts to callextractPreAgentStepErrors(run.LogsPath)for failed runs, but this also fails becauserun.LogsPathis empty.The underlying GitHub Actions job logs are always available via the GitHub API (e.g.,
gh run view --log), but the audit command does not fall back to fetching them when artifact download finds nothing.Impact
activation(e.g., due to missing/invalid configuration, checkout failures, or environment errors)Reproduction Data
Consistently reproducible with the "Issue Monster" workflow which has been failing at
activationon every run for the past 24+ hours:22693958004activation22690865467activation22691884770activation22689634900activationAll runs show
jobs[activation].conclusion = "failure"andjobs[agent].conclusion = "skipped".Suggested Fix
When
ErrNoArtifactsis returned and the run hasconclusion = "failure":gh run view --log --job (activation_job_id)errorsfield anddownloaded_fileswith these job logsAlternatively, always attempt to download workflow job logs in addition to artifacts for failed runs.
Environment
Comments on the Issue (you are @copilot in this section)
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.