Update PR review skill with common patterns#177288
Closed
albanD wants to merge 1 commit intopytorch:mainfrom
Closed
Update PR review skill with common patterns#177288albanD wants to merge 1 commit intopytorch:mainfrom
albanD wants to merge 1 commit intopytorch:mainfrom
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/177288
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 28 PendingAs of commit 876236d with merge base b9d551d ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
drisspg
approved these changes
Mar 12, 2026
Collaborator
Author
|
@pytorchbot merge |
Collaborator
Author
|
@pytorchbot merge -f |
|
❌ 🤖 pytorchbot command failed: Try |
Collaborator
Author
|
@pytorchbot merge -f "claude skill only and lint is already green" |
Collaborator
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
5 tasks
pytorchmergebot
pushed a commit
that referenced
this pull request
Mar 13, 2026
## Summary Adds a `workflow_dispatch` workflow that the autorevert system can trigger when it detects an early failure pattern. Claude Opus 4.6 analyzes the suspect commit's diff, failed job logs, and PyTorch source code to determine whether the commit actually caused the CI failures. Returns a structured JSON verdict as an artifact: - **revert** — causal chain found, proceed to revert immediately - **unsure** — inconclusive, continue with restart-to-confirm (default behavior unchanged) - **not_related** — failures unrelated to the change, ignore this signal - **garbage** — signal is unreliable (infra flake, driver crash), suppress for ~2 hours Design doc: https://docs.google.com/document/d/1BA9B7cIIKiapI37fSFGDD7D0F-VwMyRKJW0PoS0KkbY/edit ## Evaluation Results (13/13 correct verdicts) Prototyped and tested on [pytorch/ciforge](https://github.com/pytorch/ciforge). Results across diverse failure types: ### Round 1 (2026-03-12) — 4/4 correct | Test Case | PR | Failure | Expected | Actual | Job | |-----------|-----|---------|----------|--------|-----| | Doc-only change | #177288 | pca_lowrank stride mismatch | not_related | **not_related @ 0.99** | [job](https://github.com/pytorch/ciforge/actions/runs/23016718498) | | Dynamo einops fix | #177165 | detectron2 graph_breaks + test_is_nonzero_mps | not_related | **not_related @ 0.93** | [job](https://github.com/pytorch/ciforge/actions/runs/23016730498) | | MPS cdouble guard | #176985 | test_is_nonzero_mps + pca_lowrank | revert | **revert @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23016740133) | | Lint missing import | #176613 | Lint / lintrunner-noclang-all | revert | **revert @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23013529685) | ### Round 2 (2026-03-13, automated hourly loop) — 9/9 correct (1 cancelled) | Timestamp | PR | Signal Key | Expected | Actual | Job | |-----------|-----|-----------|----------|--------|-----| | 03:12Z | #176613 | Lint / lintrunner-noclang-all | revert | **revert @ 0.98** | [job](https://github.com/pytorch/ciforge/actions/runs/23034497618) | | 03:12Z | #176613 | fsdp/test_fully_shard_comm (test exec) | revert | **revert @ 0.98** | [job](https://github.com/pytorch/ciforge/actions/runs/23034499988) | | 09:11Z | #177273 | test-timeout-270min (infra) | — | *cancelled* | [job](https://github.com/pytorch/ciforge/actions/runs/23043982417) | | 10:12Z | #176019 | AllenaiLongformerBase fail_to_run (periodic) | garbage | **garbage @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23046142800) | | 10:12Z | #176019 | detectron2_fcos IMPROVED (periodic) | not_related | **not_related @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23046144261) | | 11:10Z | #176019 | functorch_dp_cifar10 fail_accuracy (periodic) | not_related | **not_related @ 0.93** | [job](https://github.com/pytorch/ciforge/actions/runs/23048173319) | | 11:10Z | #176019 | basic_gnn_edgecnn IMPROVED (periodic) | not_related | **not_related @ 0.92** | [job](https://github.com/pytorch/ciforge/actions/runs/23048174698) | | 15:09Z | #177096 | S3 PutObject IAM denied - ROCm gfx950 (infra) | garbage | **garbage @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23057146500) | | 16:09Z | #176019 | vit_base_patch16_siglip_256 fail_to_run (periodic) | not_related | **not_related @ 0.97** | [job](https://github.com/pytorch/ciforge/actions/runs/23059634364) | | 16:09Z | #176019 | shufflenet_v2_x1_0 fail_accuracy (periodic) | not_related | **not_related @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23059635765) | ### Summary by verdict type | Verdict | Count | Correct | Avg Confidence | |---------|-------|---------|----------------| | revert | 4 | 4/4 | 0.97 | | garbage | 2 | 2/2 | 0.95 | | not_related | 7 | 7/7 | 0.94 | ## Test plan - [x] Prototyped and tested on pytorch/ciforge with 13 real trunk failure cases - [x] Verified structured JSON output matches schema - [x] Verified verdict artifact uploads correctly - [ ] Trigger via GitHub UI with `workflow_dispatch` on pytorch/pytorch to validate bedrock environment works - [ ] Integrate dispatch call into autorevert lambda (follow-up) Pull Request resolved: #177404 Approved by: https://github.com/wdvr
AaronWang04
pushed a commit
to AaronWang04/pytorch
that referenced
this pull request
Mar 24, 2026
## Summary Adds a `workflow_dispatch` workflow that the autorevert system can trigger when it detects an early failure pattern. Claude Opus 4.6 analyzes the suspect commit's diff, failed job logs, and PyTorch source code to determine whether the commit actually caused the CI failures. Returns a structured JSON verdict as an artifact: - **revert** — causal chain found, proceed to revert immediately - **unsure** — inconclusive, continue with restart-to-confirm (default behavior unchanged) - **not_related** — failures unrelated to the change, ignore this signal - **garbage** — signal is unreliable (infra flake, driver crash), suppress for ~2 hours Design doc: https://docs.google.com/document/d/1BA9B7cIIKiapI37fSFGDD7D0F-VwMyRKJW0PoS0KkbY/edit ## Evaluation Results (13/13 correct verdicts) Prototyped and tested on [pytorch/ciforge](https://github.com/pytorch/ciforge). Results across diverse failure types: ### Round 1 (2026-03-12) — 4/4 correct | Test Case | PR | Failure | Expected | Actual | Job | |-----------|-----|---------|----------|--------|-----| | Doc-only change | pytorch#177288 | pca_lowrank stride mismatch | not_related | **not_related @ 0.99** | [job](https://github.com/pytorch/ciforge/actions/runs/23016718498) | | Dynamo einops fix | pytorch#177165 | detectron2 graph_breaks + test_is_nonzero_mps | not_related | **not_related @ 0.93** | [job](https://github.com/pytorch/ciforge/actions/runs/23016730498) | | MPS cdouble guard | pytorch#176985 | test_is_nonzero_mps + pca_lowrank | revert | **revert @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23016740133) | | Lint missing import | pytorch#176613 | Lint / lintrunner-noclang-all | revert | **revert @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23013529685) | ### Round 2 (2026-03-13, automated hourly loop) — 9/9 correct (1 cancelled) | Timestamp | PR | Signal Key | Expected | Actual | Job | |-----------|-----|-----------|----------|--------|-----| | 03:12Z | pytorch#176613 | Lint / lintrunner-noclang-all | revert | **revert @ 0.98** | [job](https://github.com/pytorch/ciforge/actions/runs/23034497618) | | 03:12Z | pytorch#176613 | fsdp/test_fully_shard_comm (test exec) | revert | **revert @ 0.98** | [job](https://github.com/pytorch/ciforge/actions/runs/23034499988) | | 09:11Z | pytorch#177273 | test-timeout-270min (infra) | — | *cancelled* | [job](https://github.com/pytorch/ciforge/actions/runs/23043982417) | | 10:12Z | pytorch#176019 | AllenaiLongformerBase fail_to_run (periodic) | garbage | **garbage @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23046142800) | | 10:12Z | pytorch#176019 | detectron2_fcos IMPROVED (periodic) | not_related | **not_related @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23046144261) | | 11:10Z | pytorch#176019 | functorch_dp_cifar10 fail_accuracy (periodic) | not_related | **not_related @ 0.93** | [job](https://github.com/pytorch/ciforge/actions/runs/23048173319) | | 11:10Z | pytorch#176019 | basic_gnn_edgecnn IMPROVED (periodic) | not_related | **not_related @ 0.92** | [job](https://github.com/pytorch/ciforge/actions/runs/23048174698) | | 15:09Z | pytorch#177096 | S3 PutObject IAM denied - ROCm gfx950 (infra) | garbage | **garbage @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23057146500) | | 16:09Z | pytorch#176019 | vit_base_patch16_siglip_256 fail_to_run (periodic) | not_related | **not_related @ 0.97** | [job](https://github.com/pytorch/ciforge/actions/runs/23059634364) | | 16:09Z | pytorch#176019 | shufflenet_v2_x1_0 fail_accuracy (periodic) | not_related | **not_related @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23059635765) | ### Summary by verdict type | Verdict | Count | Correct | Avg Confidence | |---------|-------|---------|----------------| | revert | 4 | 4/4 | 0.97 | | garbage | 2 | 2/2 | 0.95 | | not_related | 7 | 7/7 | 0.94 | ## Test plan - [x] Prototyped and tested on pytorch/ciforge with 13 real trunk failure cases - [x] Verified structured JSON output matches schema - [x] Verified verdict artifact uploads correctly - [ ] Trigger via GitHub UI with `workflow_dispatch` on pytorch/pytorch to validate bedrock environment works - [ ] Integrate dispatch call into autorevert lambda (follow-up) Pull Request resolved: pytorch#177404 Approved by: https://github.com/wdvr
EmanueleCoradin
pushed a commit
to EmanueleCoradin/pytorch
that referenced
this pull request
Mar 30, 2026
Two main change is to push the reviwer to use sub agents. And added claude-suggested and manually reviewed patterns that we want to see. trying to strike a thorough but not too verbose balance. Pull Request resolved: pytorch#177288 Approved by: https://github.com/drisspg
EmanueleCoradin
pushed a commit
to EmanueleCoradin/pytorch
that referenced
this pull request
Mar 30, 2026
## Summary Adds a `workflow_dispatch` workflow that the autorevert system can trigger when it detects an early failure pattern. Claude Opus 4.6 analyzes the suspect commit's diff, failed job logs, and PyTorch source code to determine whether the commit actually caused the CI failures. Returns a structured JSON verdict as an artifact: - **revert** — causal chain found, proceed to revert immediately - **unsure** — inconclusive, continue with restart-to-confirm (default behavior unchanged) - **not_related** — failures unrelated to the change, ignore this signal - **garbage** — signal is unreliable (infra flake, driver crash), suppress for ~2 hours Design doc: https://docs.google.com/document/d/1BA9B7cIIKiapI37fSFGDD7D0F-VwMyRKJW0PoS0KkbY/edit ## Evaluation Results (13/13 correct verdicts) Prototyped and tested on [pytorch/ciforge](https://github.com/pytorch/ciforge). Results across diverse failure types: ### Round 1 (2026-03-12) — 4/4 correct | Test Case | PR | Failure | Expected | Actual | Job | |-----------|-----|---------|----------|--------|-----| | Doc-only change | pytorch#177288 | pca_lowrank stride mismatch | not_related | **not_related @ 0.99** | [job](https://github.com/pytorch/ciforge/actions/runs/23016718498) | | Dynamo einops fix | pytorch#177165 | detectron2 graph_breaks + test_is_nonzero_mps | not_related | **not_related @ 0.93** | [job](https://github.com/pytorch/ciforge/actions/runs/23016730498) | | MPS cdouble guard | pytorch#176985 | test_is_nonzero_mps + pca_lowrank | revert | **revert @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23016740133) | | Lint missing import | pytorch#176613 | Lint / lintrunner-noclang-all | revert | **revert @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23013529685) | ### Round 2 (2026-03-13, automated hourly loop) — 9/9 correct (1 cancelled) | Timestamp | PR | Signal Key | Expected | Actual | Job | |-----------|-----|-----------|----------|--------|-----| | 03:12Z | pytorch#176613 | Lint / lintrunner-noclang-all | revert | **revert @ 0.98** | [job](https://github.com/pytorch/ciforge/actions/runs/23034497618) | | 03:12Z | pytorch#176613 | fsdp/test_fully_shard_comm (test exec) | revert | **revert @ 0.98** | [job](https://github.com/pytorch/ciforge/actions/runs/23034499988) | | 09:11Z | pytorch#177273 | test-timeout-270min (infra) | — | *cancelled* | [job](https://github.com/pytorch/ciforge/actions/runs/23043982417) | | 10:12Z | pytorch#176019 | AllenaiLongformerBase fail_to_run (periodic) | garbage | **garbage @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23046142800) | | 10:12Z | pytorch#176019 | detectron2_fcos IMPROVED (periodic) | not_related | **not_related @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23046144261) | | 11:10Z | pytorch#176019 | functorch_dp_cifar10 fail_accuracy (periodic) | not_related | **not_related @ 0.93** | [job](https://github.com/pytorch/ciforge/actions/runs/23048173319) | | 11:10Z | pytorch#176019 | basic_gnn_edgecnn IMPROVED (periodic) | not_related | **not_related @ 0.92** | [job](https://github.com/pytorch/ciforge/actions/runs/23048174698) | | 15:09Z | pytorch#177096 | S3 PutObject IAM denied - ROCm gfx950 (infra) | garbage | **garbage @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23057146500) | | 16:09Z | pytorch#176019 | vit_base_patch16_siglip_256 fail_to_run (periodic) | not_related | **not_related @ 0.97** | [job](https://github.com/pytorch/ciforge/actions/runs/23059634364) | | 16:09Z | pytorch#176019 | shufflenet_v2_x1_0 fail_accuracy (periodic) | not_related | **not_related @ 0.95** | [job](https://github.com/pytorch/ciforge/actions/runs/23059635765) | ### Summary by verdict type | Verdict | Count | Correct | Avg Confidence | |---------|-------|---------|----------------| | revert | 4 | 4/4 | 0.97 | | garbage | 2 | 2/2 | 0.95 | | not_related | 7 | 7/7 | 0.94 | ## Test plan - [x] Prototyped and tested on pytorch/ciforge with 13 real trunk failure cases - [x] Verified structured JSON output matches schema - [x] Verified verdict artifact uploads correctly - [ ] Trigger via GitHub UI with `workflow_dispatch` on pytorch/pytorch to validate bedrock environment works - [ ] Integrate dispatch call into autorevert lambda (follow-up) Pull Request resolved: pytorch#177404 Approved by: https://github.com/wdvr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two main change is to push the reviwer to use sub agents.
And added claude-suggested and manually reviewed patterns that we want to see. trying to strike a thorough but not too verbose balance.