🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154448

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Cancelled Jobs, 1 Unrelated Failure

As of commit 2f4cadb with merge base 16d05e1 ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Looks like this test would still hang. i.e. we cannot add pg._allgather API call either.
https://github.com/pytorch/pytorch/actions/runs/15284040734/job/42991439449

Closing based on observations from #154448 (comment)

Re-opening to propose the removal of the API between backend._set_enable_nan_check(False) and backend._set_enable_nan_check(True) in this test_nan_assert function.

Due to the removal of the first all_gather, the NaN check runs early. The main thread has yet to launch the ncclAllGather kernel. At this point, the CUDA launch call senses a DSA, thus throws an exception (Cuda failure 'unspecified launch failure'). This behavior is different from the previous behavior of SIGABRT.

In that case, we can use with self.assertRaises to assert the throw of that exception (instead of SIGABRT testing). But it is still not reliable enough -- too sensitive to timing.

I wonder if we can funnel both behavior into SIGABRT, by:

try:
    pg.all_gather(...)
except Exception:
    sys.exit(signal.abort)

@pytorchbot rebase -b main

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

Successfully rebased main-reenable-test-nan-assert onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout main-reenable-test-nan-assert && git pull --rebase)

So before the last change the error was:

Expected -6 but got 6.

Now I added a "-" sign to 6, then the error becomes:
Expected -6 but got 250.

What is 250? -6 in two's complement would be 11111010 (binary) or 250 (decimal)

So our error code comparison is in decimal, perhaps we need some post processing to change 11111010 back to -6.

@pytorchbot merge -f "docker tag missing? does not seem related"

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team