Display the related broken trunk (under BROKEN TRUNK) and similar failures (under FLAKY) on Dr.CI by huydhn · Pull Request #5149 · pytorch/test-infra

huydhn · 2024-04-29T23:55:02Z

As part of #5139, I'm updating hasSimilarFailures and rename isBrokenTrunk to getTrunkFailure to return the related similar and trunk failures. These related failures can then be displayed on Dr.CI to provide additional insight on how it classifies the results.

This PR covers the first two items, the rest will be on the next PR I'm working on.

Provide additional information about broken trunk failures
Provide additional information about similar flaky failures
Provide additional information about unstable failures
Provide additional information about other flaky failures including jobs suppressed by labels and infra flaky.

Testing

On pytorch/pytorch#124920,

curl --request POST \
--url "http://localhost:3000/api/drci/drci?prNumber=124920" \
--header "Authorization: TOKEN" \
--data 'repo=pytorch'

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124920

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit a6516ea6789e12a1a80a8c8cc7ce63698d443821 with merge base 59a1f1f308545e3ac1d81940a51f8dc0db3d82d4 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-focal-py3_8-clang9-xla / test (xla, 1, 1, linux.12xlarge) (gh)(similar failure)
test_all_cpu_tensor

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
inductor/test_cudagraph_trees.py::CudaGraphTreeTests::test_mutation_cudagraph_managed_tensors_config_backend_cudagraphs

This comment was automatically generated by Dr. CI and updates every 15 minutes.

vercel · 2024-04-29T23:55:07Z

@huydhn is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

vercel · 2024-04-29T23:57:51Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
torchci	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 30, 2024 6:17pm

clee2000

A couple of minor things but other wise looks fine

clee2000 · 2024-04-30T17:43:09Z

torchci/pages/api/drci/drci.ts

+      if (header === "BROKEN TRUNK") {
+        output += ` ([trunk failure](${relatedJobLink}))`;
+      } else if (header === "FLAKY") {
+        output += `([similar failure](${relatedJobLink}))`;


Suggested change

output += `([similar failure](${relatedJobLink}))`;

output += ` ([similar failure](${relatedJobLink}))`;

clee2000 · 2024-04-30T17:43:17Z

torchci/pages/api/drci/drci.ts

+      } else if (header === "FLAKY") {
+        output += `([similar failure](${relatedJobLink}))`;
+      } else {
+        output += `([related job](${relatedJobLink}))`;


Suggested change

output += `([related job](${relatedJobLink}))`;

output += ` ([related job](${relatedJobLink}))`;

clee2000 · 2024-04-30T17:44:48Z

torchci/pages/api/drci/drci.ts

+
    if (job.failure_captures) {
-      output += `    \`${job.failure_captures[0]}\`\n`;
+      output += `\n    \`${job.failure_captures[0]}\`\n`;


sanity check, i think you might need a \n to divide lines within a category if the failures captures doesnt exist

Oh, yeah, good catch. it would screw up the format otherwise

As the next part of #5149, this PR provides some additional info about unstable and infra flaky jobs. * [x] #5149 * [x] Provide link to the issue that mark the job as unstable * [x] Give the label(s) that suppress the job * [x] Print the rule from https://github.com/pytorch/test-infra/blob/generated-stats/stats/flaky-rules.json that marks the job as flaky * [x] Explain the reason for infra flaky (I couldn't find any recent examples for manual testing) ### Testing 1. From pytorch/pytorch#125264 <details open><summary>NEW FAILURE - The following job has failed:</summary> * [pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu)](https://hud.pytorch.org/pr/pytorch/pytorch/125264#24452065236) ([gh](https://github.com/pytorch/pytorch/actions/runs/8901751864/job/24452065236)) `inductor/test_cudagraph_trees.py::CudaGraphTreeTests::test_aliasing_static_ref` </details> <details ><summary>FLAKY - The following job failed but was likely due to flakiness present on trunk:</summary> * [Lint / lintrunner-noclang / linux-job](https://hud.pytorch.org/pr/pytorch/pytorch/125264#24446808455) ([gh](https://github.com/pytorch/pytorch/actions/runs/8901751878/job/24446808455)) (matched **linux** rule in [flaky-rules.json](https://github.com/pytorch/test-infra/blob/generated-stats/stats/flaky-rules.json)) `The process '/usr/bin/git' failed with exit code 1` </details> 2. From pytorch/executorch#3318 <details ><summary>UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:</summary> * [Android / test-llama-app / mobile-job (android)](https://hud.pytorch.org/pr/pytorch/executorch/3318#24282434625) ([gh](https://github.com/pytorch/executorch/actions/runs/8842776071/job/24282434625)) ([#3344](pytorch/executorch#3344)) `Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers` </details> 3. From pytorch/pytorch#125143 * [Lint / lintrunner-noclang / linux-job](https://hud.pytorch.org/pr/pytorch/pytorch/125143#24373801771) ([gh](https://github.com/pytorch/pytorch/actions/runs/8878104746/job/24373801771)) `>>> Lint for torch/onnx/_internal/onnx_proto_utils.py:` </details> <details ><summary>FLAKY - The following jobs failed but were likely due to flakiness present on trunk:</summary> * [BC Lint / bc_linter](https://hud.pytorch.org/pr/pytorch/pytorch/125143#24450453134) ([gh](https://github.com/pytorch/pytorch/actions/runs/8878104658/job/24450453134)) (suppressed by suppress-bc-linter) `Process completed with exit code 1.` * [pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu)](https://hud.pytorch.org/pr/pytorch/pytorch/125143#24374207841) ([gh](https://github.com/pytorch/pytorch/actions/runs/8878104713/job/24374207841)) ([similar failure](https://hud.pytorch.org/pytorch/pytorch/commit/1a0b24776212b383d025010e935f33f58a96e276#24348608242)) `test_foreach.py::TestForeachCUDA::test_binary_op_list_slow_path__foreach_div_cuda_bool` </details>

huydhn added 2 commits April 29, 2024 16:44

Display the related trunk and similar failures on Dr.CI

faea73f

Minor tweak

4fe9f4c

huydhn requested review from clee2000 and malfet April 29, 2024 23:55

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 29, 2024

huydhn requested a review from a team April 29, 2024 23:55

vercel bot deployed to Preview April 29, 2024 23:59 View deployment

clee2000 approved these changes Apr 30, 2024

View reviewed changes

Fix review comments

c83bf33

vercel bot deployed to Preview April 30, 2024 18:17 View deployment

huydhn merged commit 4a5f0bf into pytorch:main Apr 30, 2024

huydhn mentioned this pull request May 1, 2024

Provide additional info about unstable and infra flaky jobs #5156

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Display the related broken trunk (under BROKEN TRUNK) and similar failures (under FLAKY) on Dr.CI#5149

Display the related broken trunk (under BROKEN TRUNK) and similar failures (under FLAKY) on Dr.CI#5149
huydhn merged 3 commits intopytorch:mainfrom
huydhn:drci-display-broken-trunk-similar-failures

huydhn commented Apr 29, 2024 •

edited

Loading

Uh oh!

vercel bot commented Apr 29, 2024

Uh oh!

vercel bot commented Apr 29, 2024 •

edited

Loading

Uh oh!

clee2000 left a comment

Uh oh!

clee2000 Apr 30, 2024

Uh oh!

clee2000 Apr 30, 2024

Uh oh!

clee2000 Apr 30, 2024

Uh oh!

huydhn Apr 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	output += `([similar failure](${relatedJobLink}))`;
	output += ` ([similar failure](${relatedJobLink}))`;

	output += `([related job](${relatedJobLink}))`;
	output += ` ([related job](${relatedJobLink}))`;

Conversation

huydhn commented Apr 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124920

✅ You can merge normally! (2 Unrelated Failures)

Uh oh!

vercel bot commented Apr 29, 2024

Uh oh!

vercel bot commented Apr 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clee2000 left a comment

Choose a reason for hiding this comment

Uh oh!

clee2000 Apr 30, 2024

Choose a reason for hiding this comment

Uh oh!

clee2000 Apr 30, 2024

Choose a reason for hiding this comment

Uh oh!

clee2000 Apr 30, 2024

Choose a reason for hiding this comment

Uh oh!

huydhn Apr 30, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

huydhn commented Apr 29, 2024 •

edited

Loading

vercel bot commented Apr 29, 2024 •

edited

Loading