Improve fix-flaky-test skill and reproduce workflow#14710
Merged
Conversation
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 14710Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 14710" |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the repository’s flaky-test fixing workflow/documentation and improves the repro tooling (workflow + local scripts) to better handle common failure modes seen in quarantined/flaky test investigations.
Changes:
- Expands the fix-flaky-test skill with a two-branch investigation/fix workflow, PR templating/attribution, and session-state tracking guidance.
- Enhances reproduce-flaky-tests automation with zero-test-run detection and improved quarantine-filter handling.
- Updates local repetition scripts to support optional Docker cleanup and zero-test detection; adds test review guidelines + wiring in
AGENTS.md.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
run-test-repeatedly.sh |
Adds CI/local detection, optional Docker cleanup, zero-test detection, and adjusts DCP cleanup matching. |
run-test-repeatedly.ps1 |
Mirrors shell script improvements for Windows/PowerShell. |
.github/workflows/reproduce-flaky-tests.yml |
Captures test output for zero-test detection and clears quarantine exclusion during dotnet test. |
.github/skills/fix-flaky-test/SKILL.md |
Documents a two-branch workflow, adds PR templating, and updates quarantine/zero-test guidance. |
.github/instructions/test-review-guidelines.instructions.md |
New guidelines/checklist for reviewing tests for common flakiness patterns. |
AGENTS.md |
Applies the new test review guidelines instructions to tests/**/*.cs. |
Contributor
🎬 CLI E2E Test RecordingsThe following terminal recordings are available for commit
📹 Recordings uploaded automatically from CI run #22463511913 |
ee2b6bc to
b840081
Compare
Move run-test-repeatedly.ps1 and run-test-repeatedly.sh from .github/workflows/fix-flaky-test/ to .github/skills/fix-flaky-test/ next to the SKILL.md file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add .github/instructions/test-review-guidelines.instructions.md with common flaky test patterns table and review checklist. Register it in AGENTS.md for tests/**/*.cs files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Override _NonQuarantinedTestRunAdditionalArgs to disable quarantine exclusion filter during dotnet test - Capture test output and detect zero-test runs (exit code 8 masked by --ignore-exit-code 8) - Fix pipefail: wrap dotnet test | tee with set +e / set -e so PIPESTATUS is captured correctly under GitHub Actions defaults - Apply same improvements to both Linux and Windows jobs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add CI vs local detection with Docker cleanup prompt - Fix DCP cleanup regex to match bare 'dcp' processes - Add zero-test detection (exit code 8 masked by Testing.props) - Add Docker container/volume/network cleanup between iterations - Update path references after move to skills directory Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add CI vs local detection with Docker cleanup prompt - Add zero-test detection (exit code 8 masked by Testing.props) - Add Docker container/volume cleanup between iterations - Update path references after move to skills directory Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Restructure around two-branch workflow (investigation + clean fix) - Add automated PR labeling with [automated] prefix - Add PR body template with skill attribution and verification table - Add contention-sensitive test handling (quarantine-project mode) - Add session state tracking for investigation/fix branches - Add local confidence tiers to scale CI verification - Add squash guidance and PR creation failure handling - Add note explaining issue stays open after fix PR - Update quarantine docs to reflect build+test requirements - Update script path references after move to skills directory Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dd7a132 to
011c179
Compare
JamesNK
approved these changes
Feb 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improves the fix-flaky-test skill, reproduce workflow, and local test scripts.
Changes
Move scripts — Move
run-test-repeatedly.ps1andrun-test-repeatedly.shfrom.github/workflows/fix-flaky-test/to.github/skills/fix-flaky-test/next toSKILL.md.Test review guidelines — Add
.github/instructions/test-review-guidelines.instructions.mdwith common flaky test patterns table and review checklist, registered inAGENTS.mdfortests/**/*.cs.Reproduce workflow (
reproduce-flaky-tests.yml)_NonQuarantinedTestRunAdditionalArgsto disable quarantine exclusion filter duringdotnet test--ignore-exit-code 8)dotnet test | teewithset +e/set -esoPIPESTATUSis captured correctly under GitHub Actions defaultsLocal test scripts (
run-test-repeatedly.sh,run-test-repeatedly.ps1)dcpprocessesTesting.props)Skill (
.github/skills/fix-flaky-test/SKILL.md)ci.ymldisabled) for reproduce/verify; clean fix branch for final PR[automated]PR prefix when no user interaction occurredTesting
Used this skill end-to-end to fix #13287 (PR #14708).