Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Fixes a flaky/invalid ordering assertion in TestStepSummaryIncludesWorkflowOverview by ensuring the step-order comparison is made between steps in the same workflow job, rather than across different jobs in the compiled lock YAML.
Changes:
- Updated
TestStepSummaryIncludesWorkflowOverviewto compare “Generate workflow overview” ordering against “Download prompt artifact” (same agent job) instead of “Create prompt” (activation job). - Regenerated the auto-generated
specs/artifacts.mdartifact-location reference.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
pkg/workflow/step_summary_test.go |
Adjusts the step-order assertion to avoid cross-job index comparisons in compiled YAML. |
specs/artifacts.md |
Updates the generated artifact reference content (jobs/artifacts/paths/usages). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Verify step runs before "Download prompt artifact" (in the same agent job) | ||
| // Note: "Create prompt" is in the activation job, so we compare against | ||
| // "Download prompt artifact" which is in the same agent job | ||
| promptIdx := strings.Index(lockContent, "- name: Download prompt artifact") |
There was a problem hiding this comment.
promptIdx is computed via strings.Index(...) but the test never asserts the step exists first. If the generated YAML changes and the step is missing/renamed, promptIdx will be -1 and the subsequent ordering assertion will fail with a misleading message. Add an explicit strings.Contains/promptIdx == -1 check and fail with a clear error before comparing indexes.
| promptIdx := strings.Index(lockContent, "- name: Download prompt artifact") | |
| promptIdx := strings.Index(lockContent, "- name: Download prompt artifact") | |
| if promptIdx == -1 { | |
| t.Fatalf("Expected 'Download prompt artifact' step to be present in compiled workflow, but it was not found") | |
| } |
| ### Job: `activation` | ||
|
|
||
| **Artifacts Uploaded:** | ||
|
|
||
| - `prompt` | ||
| - **Paths**: `/tmp/gh-aw/aw-prompts/prompt.txt` | ||
| - **Used in**: 84 workflow(s) - agent-performance-analyzer.md, agent-persona-explorer.md, ai-moderator.md, archie.md, bot-detection.md, brave.md, breaking-change-checker.md, changeset.md, chroma-issue-indexer.md, ci-coach.md, ci-doctor.md, cli-consistency-checker.md, cloclo.md, code-scanning-fixer.md, codex-github-remote-mcp-test.md, commit-changes-analyzer.md, copilot-pr-merged-report.md, copilot-pr-nlp-analysis.md, craft.md, daily-choice-test.md, daily-copilot-token-report.md, daily-fact.md, daily-file-diet.md, daily-issues-report.md, daily-mcp-concurrency-analysis.md, daily-news.md, daily-observability-report.md, daily-repo-chronicle.md, daily-team-status.md, deep-report.md, dependabot-burner.md, dependabot-go-checker.md, dev-hawk.md, dev.md, dictation-prompt.md, example-custom-error-patterns.md, example-permissions-warning.md, firewall.md, functional-pragmatist.md, github-mcp-structural-analysis.md, glossary-maintainer.md, go-fan.md, go-pattern-detector.md, grumpy-reviewer.md, hourly-ci-cleaner.md, issue-classifier.md, issue-monster.md, issue-triage-agent.md, layout-spec-maintainer.md, mergefest.md, metrics-collector.md, notion-issue-summary.md, pdf-summary.md, plan.md, poem-bot.md, pr-nitpick-reviewer.md, pr-triage-agent.md, python-data-charts.md, q.md, refiner.md, release.md, repo-audit-analyzer.md, repository-quality-improver.md, research.md, scout.md, security-compliance.md, security-review.md, slide-deck-maintainer.md, smoke-copilot-sdk.md, smoke-project.md, smoke-temporary-id.md, stale-repo-identifier.md, super-linter.md, technical-doc-writer.md, test-create-pr-error-handling.md, test-dispatcher.md, test-project-url-default.md, test-workflow.md, tidy.md, typist.md, video-analyzer.md, weekly-issue-summary.md, workflow-generator.md, workflow-health-manager.md |
There was a problem hiding this comment.
This PR is described as a targeted test fix, but it also includes a very large regeneration of the auto-generated specs/artifacts.md content (new jobs/paths/usages). If this regeneration is intentional, it should be called out in the PR description; if not, consider dropping it from this PR to keep the change focused.
TestStepSummaryIncludesWorkflowOverviewwas failing because it compared string positions of steps from different jobs. "Create prompt" (activation job) always appears before "Generate workflow overview" (agent job) in the compiled YAML, making the index comparison invalid.Changes
The test now correctly validates that workflow overview generation happens before prompt download in the agent job execution sequence.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
https://api.github.com/graphql/usr/bin/gh gh auth status ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet -30(http block)/usr/bin/gh gh auth status ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet --no�� --noprofile Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/vet(http block)/usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw(http block)https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1/usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq .object.sha runs/20260217-000049-14171/test-4184702056/.github/workflows stmain.go ache/node/24.13.0/x64/bin/node low-c5ec5066-267git /tmp/go-build352rev-parse tnet/tools/bash bash t-11�� sistency_GoAndJavaScript3417362662/001/test-empty-frontmatter.md r ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet -unreachable=falgit /tmp/go-build352rev-parse ache/node/24.13.--show-toplevel ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet(http block)/usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq .object.sha "prettier" --write '**/*.cjs' '**/*.ts' '**/*.json' --ignore-path ../../../.pret.prettierignore gh /opt/hostedtoolcache/node/24.13.0/x64/bin/npx view owner/repo /usr/bin/git npx pret�� --write ../../../**/*.json /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/vet --ignore-path ../../../.prettirev-parse /usr/bin/git /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/vet(http block)https://api.github.com/repos/actions/ai-inference/git/ref/tags/v2/usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v2 --jq .object.sha(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v3/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq .object.sha 3239145438 64/pkg/tool/linu--jq g_.a --local credential.usernrev-parse ache/go/1.25.0/x--show-toplevel node --ch�� /tmp/validate-js-426382582.cjs ache/go/1.25.0/x64/pkg/tool/linu-buildtags ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet se 7734060/b051/vetrev-parse 9-44e5-baf6-dcab--show-toplevel ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq .object.sha --show-toplevel git k/node_modules/.bin/sh --show-toplevel /tmp/go-build176rev-parse /usr/bin/git git rev-�� rite '**/*.cjs' '**/*.ts' '**/*.json' --ignore-path ../../../.pr**/*.json git k/node_modules/.bin/node WorkflowFiles_Trgit head /usr/bin/git git(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq .object.sha --get-regexp 64/src/internal/fuzz/counters_supported.go ache/go/1.25.0/x64/pkg/tool/linux_amd64/cgo(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq .object.sha get --local .cfg credential.helpegit(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq .object.sha --show-toplevel ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet /usr/bin/git Generate workflogit /home/REDACTED/worrev-parse ache/go/1.25.0/x--show-toplevel git rev-�� --show-toplevel l /usr/bin/git -unreachable=falgit /tmp/go-build352rev-parse $name) { has--show-toplevel git(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v5/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha -1679075352.cjs 64/pkg/tool/linux_amd64/vet /usr/bin/git 006b85bc .cfg 64/pkg/tool/linu--get-regexp git rev-�� --show-toplevel 64/pkg/tool/linux_amd64/vet 64/pkg/tool/linux_amd64/link(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha 7790249/b418/timeutil.test --jq ache/node/24.13.0/x64/bin/node -unreachable=falgit /tmp/go-build352rev-parse ache/node/24.13.--show-toplevel iiEtxJ64HjgrP/g5FpiJ9ENbZatjTay5oJ/BhqTCoRMGewfss9ZXZGY/X4XoDkfiiEtxJ64HjgrP t-42�� sistency_KeyOrdering1825432062/001/test1.md tail 7790249/b418/_pkg_.a(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha user.email test@example.com /usr/bin/git --noprofile(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v6/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v6.0.2/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6.0.2 --jq .object.sha fflib@v1.0.0/difflib/difflib.go(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6.0.2 --jq .object.sha(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6.0.2 --jq .object.sha ify@v1.11.1/assert/assertion_compare.go ify@v1.11.1/assert/assertion_format.go x_amd64/cgo r(http block)https://api.github.com/repos/actions/complex/git/ref/tags/v1/usr/bin/gh gh api /repos/actions/complex/git/ref/tags/v1 --jq .object.sha 006b85bc .cfg 64/pkg/tool/linux_amd64/vet(http block)https://api.github.com/repos/actions/github-script/git/ref/tags/v7/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v7 --jq .object.sha(http block)https://api.github.com/repos/actions/github-script/git/ref/tags/v8/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha -bool -buildtags sh -errorsas -ifaceassert -nilfunc bash --no�� celain --ignore-submodules | head -n 10 -tests rgo/bin/bash(http block)/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha --noprofile -tests rgo/bin/bash(http block)/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha f/tags/v6 -tests ache/go/1.25.0/x64/bin/bash(http block)https://api.github.com/repos/actions/setup-go/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq .object.sha --show-toplevel ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet /usr/bin/git --noprofile(http block)/usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq .object.sha i/install.sh node ache/node/24.13.0/x64/bin/node ./../.prettieriggit /opt/hostedtoolcrev-parse /usr/bin/gh sh t-19�� /v1.0.0 gh /opt/hostedtoolcache/go/1.25.0/x64/bin/node x_amd64/vet x_amd64/vet /usr/bin/git node(http block)https://api.github.com/repos/actions/setup-go/git/ref/tags/v6/usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v6 --jq .object.sha(http block)https://api.github.com/repos/actions/setup-node/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha g/stringutil/ide-s g/stringutil/pat-w ache/go/1.25.0/x-buildmode=exe(http block)/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha --show-toplevel ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet /usr/bin/git --noprofile(http block)/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha 4 node /opt/pipx_bin/sh /home/REDACTED/worgit bash /usr/bin/git sh -c runs/20260217-000152-18728/test-222739931/.github/workflows git ache/node/24.13.0/x64/bin/node /tmp/TestCollectgit config run-script/lib/n--show-toplevel npm(http block)https://api.github.com/repos/actions/setup-node/git/ref/tags/v6/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq .object.sha(http block)https://api.github.com/repos/actions/setup-node/git/ref/tags/v6.2.0/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6.2.0 --jq .object.sha ify@v1.11.1/assert/yaml/yaml_default.go(http block)/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6.2.0 --jq .object.sha(http block)https://api.github.com/repos/anchore/sbom-action/git/ref/tags/v0/usr/bin/gh gh api /repos/anchore/sbom-action/git/ref/tags/v0 --jq .object.sha(http block)https://api.github.com/repos/docker/build-push-action/git/ref/tags/v6/usr/bin/gh gh api /repos/docker/build-push-action/git/ref/tags/v6 --jq .object.sha(http block)https://api.github.com/repos/docker/login-action/git/ref/tags/v3/usr/bin/gh gh api /repos/docker/login-action/git/ref/tags/v3 --jq .object.sha(http block)https://api.github.com/repos/docker/metadata-action/git/ref/tags/v5/usr/bin/gh gh api /repos/docker/metadata-action/git/ref/tags/v5 --jq .object.sha(http block)https://api.github.com/repos/docker/setup-buildx-action/git/ref/tags/v3/usr/bin/gh gh api /repos/docker/setup-buildx-action/git/ref/tags/v3 --jq .object.sha(http block)https://api.github.com/repos/github/gh-aw/tmp/go-build3541743673/b001/workflow.test /tmp/go-build3541743673/b001/workflow.test -test.testlogfile=/tmp/go-build3541743673/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true ew@v1.1.1/spew/bypass.go ew@v1.1.1/spew/common.go 64/pkg/tool/linux_amd64/compile(http block)https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts/usr/bin/gh gh run download 1 --dir test-logs/run-1 x_amd64/vet ache/node/24.13.0/x64/bin/node(http block)/usr/bin/gh gh run download 1 --dir test-logs/run-1 git /usr/bin/git 8WsF/avxLPGO5v5Zgit config /usr/bin/git git rev-�� */*.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path ../../../.prettierignore git 64/pkg/tool/linux_amd64/asm --show-toplevel 7790249/b414/_terev-parse ache/node/24.13.--show-toplevel 64/pkg/tool/linux_amd64/asm(http block)https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts/usr/bin/gh gh run download 12345 --dir test-logs/run-12345 x_amd64/vet ache/node/24.13.0/x64/bin/node(http block)/usr/bin/gh gh run download 12345 --dir test-logs/run-12345 git /usr/bin/git s/test.md -goversion /usr/bin/git git rev-�� */*.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path git 64/pkg/tool/linux_amd64/compile --show-toplevel e/git-remote-httrev-parse ache/node/24.13.--show-toplevel 64/pkg/tool/linux_amd64/compile(http block)https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts/usr/bin/gh gh run download 12346 --dir test-logs/run-12346 x_amd64/vet ache/node/24.13.0/x64/bin/node(http block)/usr/bin/gh gh run download 12346 --dir test-logs/run-12346 git 64/bin/sh s/test.md -goversion /usr/bin/git git rev-�� */*.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path git Name,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle /home/REDACTED/worgit /home/REDACTED/worrev-parse /usr/bin/git 64/pkg/tool/linux_amd64/asm(http block)https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts/usr/bin/gh gh run download 2 --dir test-logs/run-2 x_amd64/vet x_amd64/compile output-qiybvo.txgit(http block)/usr/bin/gh gh run download 2 --dir test-logs/run-2 git /usr/bin/git y5oJ/g5FpiJ9ENbZgit rev-parse /usr/bin/git git rev-�� */*.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path ../../../.prettierignore git 64/pkg/tool/linux_amd64/compile --show-toplevel l(http block)https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts/usr/bin/gh gh run download 3 --dir test-logs/run-3 x_amd64/vet x_amd64/compile(http block)/usr/bin/gh gh run download 3 --dir test-logs/run-3 4qBSgoBRlrikz/av--package-lock-only /usr/bin/git ry=1 -tests(http block)https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts/usr/bin/gh gh run download 4 --dir test-logs/run-4 x_amd64/vet x_amd64/compile(http block)/usr/bin/gh gh run download 4 --dir test-logs/run-4 git /usr/bin/git --show-toplevel bash 7790249/b416/vet--show-toplevel git rev-�� */*.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path ../../../.prettierignore git 64/pkg/tool/linux_amd64/cgo --show-toplevel l /opt/hostedtoolc--show-toplevel 64/pkg/tool/linux_amd64/cgo(http block)https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts/usr/bin/gh gh run download 5 --dir test-logs/run-5 x_amd64/vet x_amd64/vet(http block)/usr/bin/gh gh run download 5 --dir test-logs/run-5 iiEtxJ64HjgrP/g5FpiJ9ENbZatjTay5oJ/BhqTCoRMGewfss9ZXZGY/X4XoDkfiiEtxJ64HjgrP /usr/bin/git sistency_KeyOrdegit tail 7790249/b418/_pk--show-toplevel git rev-�� */*.json' '!../../../pkg/workflow/js/**/*.json' --ignore-path ../../../.prettierignore git 64/pkg/tool/linux_amd64/vet --show-toplevel cp ache/node/24.13.--show-toplevel 64/pkg/tool/linux_amd64/vet(http block)https://api.github.com/repos/github/gh-aw/actions/workflowsCo-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> t -errorsas -ifaceassert -nilfunc bash --no�� n/git/ref/tags/v3 -tests /home/REDACTED/.dotnet/tools/bash` (http block)
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> ache/go/1.25.0/x64/bin/git -errorsas -ifaceassert -nilfunc bash --no�� rd -tests /home/REDACTED/.local/bin/bash` (http block)
Original prompt
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.