🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138922

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 30 New Failures, 1 Unrelated Failure

As of commit 794f6a4 with merge base c3fbec7 ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorchbot rebase -b main

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

Successfully rebased gh/bobrenjc93/90/orig onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/138922)

@pytorchbot revert -m 'Sorry for reverting your change but there is some slow tests failing after this land' -c nosignal

inductor/test_efficient_conv_bn_eval.py::EfficientConvBNEvalCpuTests::test_basic_cpu GH job link HUD commit link

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

@bobrenjc93 your PR has been successfully reverted.

Opposite. We will recompile less. But because all float inputs are now treated as arguments, previously where we could inline constant into compiled code, now it's another argument load. This matters more on CPU I think.

Thanks for the reply.

• To better understand the changes of behavior:
  • Is the previous behavior that we wrap the float scalar value into a float tensor (based on the description of this flag:

    pytorch/torch/_dynamo/config.py

    Lines 65 to 67 in 2e7ba0b

    	# Whether or not to specialize on float inputs. Dynamo will always promote
    	# float inputs into Tensor inputs, but at the moment, backends inconsistently
    	# support codegen on float (this is to be fixed).

    ) and inline the float tensor as a constant into the compiled code?
  • But for now, the behavior is the float will be a input argument of compiled function.
• If the float value is input argument, shouldn't we add some guard to its value? I remember the case of input integer as argument, when the integer values changed, it will causes recompilation like this one: Avoid recompilation for inputs integer number #132849

@pytorchbot merge

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

@leslie-fang-intel I'd recommend reading this design doc https://docs.google.com/document/d/1HswUSp9H6mg8Vg27mhRk8YzC9q_uf63b6wz-gwx65BQ/edit?pli=1&tab=t.0#heading=h.xvyiqp8tuje6

inline the float tensor as a constant into the compiled code?

Previously we just inline the float. There was no tensorfication.

shouldn't we add some guard to its value

If for whatever reason we need to guard, we'll actually fall back specialization. See #140346

@bobrenjc93 I believe this might be causing perf regression on torchbench: https://hud.pytorch.org/benchmark/torchbench/inductor_no_cudagraphs?dashboard=torchinductor&startTime=Mon,%2018%20Nov%202024%2022:00:43%20GMT&stopTime=Mon,%2025%20Nov%202024%2022:00:43%20GMT&granularity=hour&mode=training&dtype=amp&deviceName=cuda%20(a100)&lBranch=gh/bobrenjc93/90/head&lCommit=e67d26992e8a473aec6ede13b8cb928208c3df34&rBranch=main&rCommit=1bdb92cbff6f8f6fd1842e8fd9969bca688eaea3

I also bisected on torchbench llama model (on A100 devvm):
python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inductor --disable-cudagraphs --device cuda --inference --bfloat16 --print-memory --output output.csv --only llama --batch-size 32
5268754 (11/22 nightly): 2.177902x
995e307: 2.196x
c513f01: 2.198x
11c786d: 2.160x
ba5253d: 1.748x <- this PR
51b6126 (11/23 nightly): 1.781900x

Discussed in Inductor group chat and the group agrees to revert this PR

@pytorchbot revert -m "perf regression on torchbench"

❌ 🤖 pytorchbot command failed:

@pytorchbot revert: error: the following arguments are required: -c/--classification

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Try @pytorchbot --help for more info.

@pytorchbot revert -m "perf regression on torchbench" -c nosignal

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

@bobrenjc93 your PR has been successfully reverted.

new benchmark runs as of 11/25

base: https://github.com/pytorch/pytorch/actions/runs/12023899968
head: https://github.com/pytorch/pytorch/actions/runs/12023903827

Endorsing the NN module spec changes, although we will have to carefully check if this actually fixes all the benchmarks