[NOT FOR MERGE] Exploratory work on AOTInductor training by benjaminglass1 · Pull Request #155877 · pytorch/pytorch

benjaminglass1 · 2025-06-13T01:28:48Z

Stack from ghstack (oldest at bottom):

TODO for MVP:

Select between training and eval mode within the export model loaders.
Improve UI around parameter updates and gradients (i.e. users should not have to manually apply gradients and maintain a separate copy of the model to apply them to for this to work).
Determine why some tensor constants get dropped when doing AOTInductor training.
Ensure optimizer runtime is integrated into export performance stats. :( Maybe we can export the full forward_and_backward_pass function?
Break out unrelated code improvements into a separate PR.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov

[ghstack-poisoned]

pytorch-bot · 2025-06-13T01:28:52Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155877

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 2 Unrelated Failures

As of commit 67d6fc3 with merge base 1abff80 ():

NEW FAILURES - The following jobs have failed:

inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (inductor_torchbench_cpu_smoketest_perf, 1, 1, linux.24xl.spr-metal) (gh)
Process completed with exit code 1.
pull / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 4, 5, linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_reconstruct.py::ReconstructTest::test_graph_break_in_wrapped_skipped_function
pull / linux-jammy-cuda12.8-py3.10-gcc11-sm89 / test (default, 2, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
dynamo/test_reconstruct.py::ReconstructTest::test_graph_break_in_wrapped_skipped_function
pull / linux-jammy-py3.10-clang18-asan / test (default, 5, 6, linux.4xlarge) (gh)
dynamo/test_reconstruct.py::ReconstructTest::test_graph_break_in_wrapped_skipped_function
pull / linux-jammy-py3.13-clang12 / test (default, 2, 5, linux.4xlarge) (gh)
dynamo/test_reconstruct.py::ReconstructTest::test_graph_break_in_wrapped_skipped_function
pull / linux-jammy-py3.9-clang12 / test (default, 2, 5, linux.4xlarge) (gh)
dynamo/test_reconstruct.py::ReconstructTest::test_graph_break_in_wrapped_skipped_function
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, linux.2xlarge) (gh)
dynamo/test_reconstruct.py::ReconstructTest::test_graph_break_in_wrapped_skipped_function

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-jammy-py3.9-clang12 / test (dynamo_wrapped, 3, 3, linux.2xlarge) (gh) (detected as infra flaky with no log or failing log classifier)

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / linux-jammy-py3_9-clang9-xla / test (xla, 1, 1, linux.12xlarge, unstable) (gh) (#158876)
sccache: error: couldn't connect to server

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 5137652 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 0019580 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 4e13b57 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 7b4dc6c Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 06a6b56 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 65834eb Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 85861d3 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 7c55723 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: a6c8cd2 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 621ad1b Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: ead6d63 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 59a57ff Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 31060c5 Pull Request resolved: #155877

[ghstack-poisoned]

ghstack-source-id: 69a2b61 Pull Request resolved: #155877

benjaminglass1 · 2025-08-25T20:17:56Z

Closing this, as we've decided not to pursue the AOTInductor training work at this time.

Update

d215f31

[ghstack-poisoned]

This was referenced Jun 13, 2025

[Inductor] Delay codegen for fallback arguments and improve typing #154371

Closed

[AOTInductor] Call most runtime fallback ops without calling into Python #154142

Closed

benjaminglass1 mentioned this pull request Jun 13, 2025

[cpp_wrapper] Build main and kernel code in separate threads #154551

Closed

pytorch-bot bot added ciflow/inductor module: dynamo module: inductor release notes: AO frontend release notes: inductor (aoti) labels Jun 13, 2025

benjaminglass1 added a commit that referenced this pull request Jun 13, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

326095c

ghstack-source-id: 5137652 Pull Request resolved: #155877

pytorchbot added the open source label Jun 13, 2025

Update

0fddae5

[ghstack-poisoned]

Update

ee9567a

[ghstack-poisoned]

Update

9dbc58a

[ghstack-poisoned]

Update

f3c6ade

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jun 13, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

28e6b89

ghstack-source-id: 0019580 Pull Request resolved: #155877

Update

1c30ba1

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jun 16, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

581e717

ghstack-source-id: 4e13b57 Pull Request resolved: #155877

Update

67fade2

[ghstack-poisoned]

Update

bf78a53

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jun 18, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

8b8ef68

ghstack-source-id: 7b4dc6c Pull Request resolved: #155877

Update

5d41215

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jun 19, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

af9c4e0

ghstack-source-id: 06a6b56 Pull Request resolved: #155877

Update

cc64f64

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jun 24, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

bbdd5e2

ghstack-source-id: 65834eb Pull Request resolved: #155877

Update

c5a3f2c

[ghstack-poisoned]

This was referenced Jul 10, 2025

[AOTI] Add missing ops to set of C-shim ops which can have nullptr returns #158073

Closed

[benchmarks] Add scalar loss as model output when training #158074

Closed

Grab bag of (mostly) typing improvements #158075

Closed

benjaminglass1 added a commit that referenced this pull request Jul 10, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

3372ed4

ghstack-source-id: 85861d3 Pull Request resolved: #155877

Update

12752a6

[ghstack-poisoned]

Update

b625da2

[ghstack-poisoned]

Update

707c519

[ghstack-poisoned]

Update

d1fc93b

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jul 15, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

f019b16

ghstack-source-id: 7c55723 Pull Request resolved: #155877

benjaminglass1 mentioned this pull request Jul 15, 2025

[RFC] AOTInductor training support #158392

Closed

4 tasks

Update

cd968b6

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jul 16, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

d5ad0f3

ghstack-source-id: a6c8cd2 Pull Request resolved: #155877

Update

456385f

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jul 17, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

01d1a67

ghstack-source-id: 621ad1b Pull Request resolved: #155877

Update

6276859

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jul 21, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

cecbce6

ghstack-source-id: ead6d63 Pull Request resolved: #155877

Update

773e031

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jul 23, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

2368134

ghstack-source-id: 59a57ff Pull Request resolved: #155877

Update

091eba0

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jul 24, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

954dbb9

ghstack-source-id: 31060c5 Pull Request resolved: #155877

Update

67d6fc3

[ghstack-poisoned]

benjaminglass1 added a commit that referenced this pull request Jul 28, 2025

[NOT FOR MERGE] Exploratory work on AOTInductor training

d435574

ghstack-source-id: 69a2b61 Pull Request resolved: #155877

benjaminglass1 closed this Aug 25, 2025

github-actions bot deleted the gh/benjaminglass1/89/head branch September 25, 2025 02:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NOT FOR MERGE] Exploratory work on AOTInductor training#155877

[NOT FOR MERGE] Exploratory work on AOTInductor training#155877
benjaminglass1 wants to merge 36 commits intogh/benjaminglass1/89/basefrom
gh/benjaminglass1/89/head

benjaminglass1 commented Jun 13, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jun 13, 2025 •

edited

Loading

Uh oh!

benjaminglass1 commented Aug 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

benjaminglass1 commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155877

❌ 7 New Failures, 2 Unrelated Failures

Uh oh!

benjaminglass1 commented Aug 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

benjaminglass1 commented Jun 13, 2025 •

edited

Loading

pytorch-bot bot commented Jun 13, 2025 •

edited

Loading