🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86677

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures, 3 Pending

As of commit 489c914:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorchbot rebase

@pytorchbot successfully started a rebase job. Check the current status here

Tried to rebase and push PR #86677, but it was already up to date

Created pytorch-linux-focal-rocm5.2-py3.8 and pytorch-linux-focal-rocm5.1-py3.8 ecr containers

@pytorchbot merge

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Merge failed

Reason: The following mandatory check(s) failed (Rule OSS CI):

• pull

Dig deeper by viewing the failures on hud

@pytorchbot rebase

@pytorchbot successfully started a rebase job. Check the current status here

Successfully rebased rocm_py38_upgrade onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm_py38_upgrade && git pull --rebase)

@seemethere @malfet ,

See the below failures in few CUDA jobs

ERROR [0.003s]: test_nn_functional_group_norm (main.TestFunctionalTracing)

Traceback (most recent call last):
File "/var/lib/jenkins/workspace/test/test_fx.py", line 4075, in functional_test
symbolic_trace(fn)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 1070, in symbolic_trace
graph = tracer.trace(root, concrete_args)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 739, in trace
(self.create_arg(fn(*args)),),
File "/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py", line 2523, in group_norm
_verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:]))
File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 274, in iter
return self.tracer.iter(self)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 183, in iter
raise TraceError('Proxy object cannot be iterated. This can be '
torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when the Proxy is used in a loop or as a *args or **kwargs function argument. See the torch.fx docs on pytorch.org for a more detailed explanation of what types of control flow can be traced, and check out the Proxy docstring for help troubleshooting Proxy iteration errors

As I understand this PR triggers a rebuild of all CI images for ROCm but not sure how CUDA image is changing, can you please help on why the CUDA images are failing. Checking on the error it is related to python version (#60069).

See the below failures in few CUDA jobs

ERROR [0.003s]: test_nn_functional_group_norm (main.TestFunctionalTracing)

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_fx.py", line 4075, in functional_test symbolic_trace(fn) File "/opt/conda/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 1070, in symbolic_trace graph = tracer.trace(root, concrete_args) File "/opt/conda/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 739, in trace (self.create_arg(fn(*args)),), File "/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py", line 2523, in group_norm _verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:])) File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 274, in iter return self.tracer.iter(self) File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 183, in iter raise TraceError('Proxy object cannot be iterated. This can be ' torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when the Proxy is used in a loop or as a *args or **kwargs function argument. See the torch.fx docs on pytorch.org for a more detailed explanation of what types of control flow can be traced, and check out the Proxy docstring for help troubleshooting Proxy iteration errors

As I understand this PR triggers a rebuild of all CI images for ROCm but not sure how CUDA image is changing, can you please help on why the CUDA images are failing. Checking on the error it is related to python version (#60069).

Surprisingly, I'm also seeing exactly the same failure on my "benign" PR #86993 to update docker image. So it's not a coincidence, let me try to dig a bit deeper to understand what's going on here

Rebase the PR works for me #86993. So I suspect some unrelated test failures (either flaky or they have been fixed)

@pytorchbot rebase

@pytorchbot successfully started a rebase job. Check the current status here

Successfully rebased rocm_py38_upgrade onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm_py38_upgrade && git pull --rebase)

@pytorchbot merge

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team