[inductor] Add an AOT compilation mode for Inductor CPP backend by desertfire · Pull Request #94822 · pytorch/pytorch

desertfire · 2023-02-14T15:01:30Z

Stack from ghstack (oldest at bottom):

-> [inductor] Add an AOT compilation mode for Inductor CPP backend #94822

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail.

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with __aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. [ghstack-poisoned]

pytorch-bot · 2023-02-14T15:01:34Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94822

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit aeb7045:

NEW FAILURES - The following jobs have failed:

linux-focal-cpu-py3.8-gcc7-inductor / test (inductor_timm_cpu_accuracy, 2, 2, linux.4xlarge) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with __aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

desertfire · 2023-02-14T15:16:09Z

Output from running test.sh,

Turning on aten_graph for aot_inductor
[2023-02-14 15:12:29,045] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 0

#include "/tmp/torchinductor_binbao/cd/ccdu23xgmx4kl3rilvo5rfytlffccwsazjbxkv4urqfixqspbwj4.h"
extern "C" void kernel_cpp_0(const float* __restrict__ in_ptr0,
                       float* __restrict__ out_ptr0,
                       float* __restrict__ out_ptr1)
{
    {
        for(long i0=0; i0<512; i0+=1)
        {
            auto tmp0 = at::vec::Vectorized<float>::loadu(in_ptr0 + 16*i0);
            auto tmp1 = tmp0.sin();
            auto tmp2 = decltype(tmp1)(1)/(decltype(tmp1)(1) + tmp1.neg().exp());
            auto tmp3 = tmp0.cos();
            auto tmp4 = decltype(tmp3)(1)/(decltype(tmp3)(1) + tmp3.neg().exp());
            tmp2.store(out_ptr0 + 16*i0);
            tmp4.store(out_ptr1 + 16*i0);
        }
        #pragma omp simd simdlen(8) 
        for(long i0=8192; i0<8192; i0+=1)
        {
            auto tmp0 = in_ptr0[i0];
            auto tmp1 = std::sin(tmp0);
            auto tmp2 = std::exp(-tmp1);
            auto tmp3 = 1 / (1 + tmp2);
            auto tmp4 = std::cos(tmp0);
            auto tmp5 = std::exp(-tmp4);
            auto tmp6 = 1 / (1 + tmp5);
            out_ptr0[i0] = tmp3;
            out_ptr1[i0] = tmp6;
        }
    }
}
std::vector<at::Tensor> __aot_inductor_entry(std::vector<at::Tensor> args) {
    at::Tensor arg0_1;
    arg0_1 = args[0];
    auto buf0 = at::empty_strided({8, 4, 16, 16}, {1024, 256, 16, 1}, at::ScalarType::Float); 
    auto buf1 = at::empty_strided({8, 4, 16, 16}, {1024, 256, 16, 1}, at::ScalarType::Float); 
    kernel_cpp_0((float*)(arg0_1.data_ptr()), (float*)(buf0.data_ptr()), (float*)(buf1.data_ptr()));
    arg0_1.reset();
    return std::vector<at::Tensor>({buf0, buf1});
}

[2023-02-14 15:12:35,669] torch._inductor.codecache: [INFO] AOT-Inductor compiles code into: /scratch/binbao/work/pytorch/test/inductor/aot/build/aot_inductor_output.so

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with __aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with __aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. ghstack-source-id: 52d0af8 Pull Request resolved: #94822

jansel

How are model weights handled by this?

torch/_dynamo/eval_frame.py

torch/_inductor/codecache.py

torch/_inductor/config.py

voznesenskym · 2023-02-17T03:06:16Z

Discussed offline a bit - but you don't need to make it part of export. You can make AOTInductor a purely additive thing.

One thing we could do that would be minimal changes is to not change how we produce a module or wrapper today under cpp_wrapper but instead make this aot thing purely additive - produce a side artifact of the .so (as we do), and a .h (As we do not yet do), if a flag is set. This would allow us to button up an impl in very flew lines, at the cost of a little bit of redundant work.

Check out how we do

if self._can_use_cpp_wrapper:
    self.wrapper_code = CppWrapperCodeGen()

You can do something like

if self.aot

And provide your own AOTCodeGen alongside the other CodeGen (either composition, or add support to emit multiple codegen?). You can then use the flag to produce a pure compiled artifact as a side effect - no need to change any of the mainline flow of compilation.

Once you have that - you can just call compile_fx after export, with the aot flag enabled.

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with __aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with __aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. ghstack-source-id: fb078c9 Pull Request resolved: #94822

jansel

What is here looks good to me.

Still need to solve the issue of weight handling.

Should __aot_inductor_entry be just aot_inductor_entry, or perhaps a function name passed in by the user to avoid name conflicts? If the user is intended to call it we shouldn't name it __*.

desertfire · 2023-02-28T18:29:23Z

I will update with refactoring after #95594 lands. Weight handling will come as the next PR.

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with __aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. ghstack-source-id: 226b18f Pull Request resolved: #94822

…ckend" Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

clee2000 · 2023-03-03T18:49:33Z

Also, this PR seems to almost quadruple the time it takes to run inductor/test_torchinductor_opinfo (prev ~1hr, after ~3.75 hr)

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. Pull Request resolved: pytorch/pytorch#94822 Approved by: https://github.com/jansel

…nd (#94822)" This reverts commit 73b6609. Reverted pytorch/pytorch#94822 on behalf of https://github.com/clee2000 due to broke inductor_tmm_cpu_accuracy, https://hud.pytorch.org/pytorch/pytorch/commit/73b66098b2f43be508e1975fd6a425ed6308b993#11745396725

Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. Pull Request resolved: pytorch/pytorch#94822 Approved by: https://github.com/jansel

…nd (#94822)" This reverts commit 73b6609. Reverted pytorch/pytorch#94822 on behalf of https://github.com/clee2000 due to broke inductor_tmm_cpu_accuracy, https://hud.pytorch.org/pytorch/pytorch/commit/73b66098b2f43be508e1975fd6a425ed6308b993#11745396725

Summary: This is a reland of #94822 ghstack-source-id: 1aa9136 Pull Request resolved: #95985

…mode for Inductor CPP backend" Summary: This is a reland of #94822 cc soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

…r CPP backend" Summary: This is a reland of #94822 cc soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

…mode for Inductor CPP backend" Summary: This is a reland of #94822 cc soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

…r CPP backend" Summary: This is a reland of #94822 cc soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

Summary: This is a reland of #94822 ghstack-source-id: dc88f33 Pull Request resolved: #95985

…nd (#95985) Summary: This is a reland of #94822 Pull Request resolved: #95985 Approved by: https://github.com/jansel

Summary: This is a reland of #94822. Solved the long compilation issue for inductor cpp tests. [ghstack-poisoned]

Summary: This is a reland of #94822. Solved the long compilation issue for inductor cpp tests. ghstack-source-id: 9f696b4 Pull Request resolved: #96520

…nd (#95985) Summary: This is a reland of pytorch/pytorch#94822 Pull Request resolved: pytorch/pytorch#95985 Approved by: https://github.com/jansel

… mode for Inductor CPP backend" Summary: This is a reland of #94822. Solved the long compilation issue for inductor cpp tests. cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

…or CPP backend" Summary: This is a reland of #94822. Solved the long compilation issue for inductor cpp tests. cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 [ghstack-poisoned]

…rch#94822) Summary: The AOT mode currently works for the CPP backend. When turned on, Inductor compiles the model code into a .so file with aot_inductor_entry as the entry function. If the AOT compilation fails, Inductor will explicitly fail. Pull Request resolved: pytorch#94822 Approved by: https://github.com/jansel

…nd (pytorch#94822)" This reverts commit 73b6609. Reverted pytorch#94822 on behalf of https://github.com/clee2000 due to broke inductor_tmm_cpu_accuracy, https://hud.pytorch.org/pytorch/pytorch/commit/73b66098b2f43be508e1975fd6a425ed6308b993#11745396725

…nd (pytorch#95985) Summary: This is a reland of pytorch#94822 Pull Request resolved: pytorch#95985 Approved by: https://github.com/jansel

…end (#96520) Summary: This is a reland of #94822. Solved the long compilation issue for inductor cpp tests. Pull Request resolved: #96520 Approved by: https://github.com/huydhn, https://github.com/malfet

…end (#96520) Summary: This is a reland of pytorch/pytorch#94822. Solved the long compilation issue for inductor cpp tests. Pull Request resolved: pytorch/pytorch#96520 Approved by: https://github.com/huydhn, https://github.com/malfet

github-actions bot added ciflow/inductor module: dynamo module: inductor labels Feb 14, 2023

desertfire added the topic: not user facing topic category label Feb 14, 2023

desertfire requested review from Chillee, SherlockNoMad, gmagogsfm, jansel, ngimel, suo, voznesenskym and yinghai February 14, 2023 19:45

jansel requested changes Feb 15, 2023

View reviewed changes

torch/_dynamo/eval_frame.py Outdated Show resolved Hide resolved

torch/_dynamo/eval_frame.py Outdated Show resolved Hide resolved

torch/_inductor/codecache.py Outdated Show resolved Hide resolved

torch/_inductor/config.py Outdated Show resolved Hide resolved

jansel approved these changes Feb 28, 2023

View reviewed changes

desertfire changed the title ~~[WIP][inductor] Add an AOT compilation mode for Inductor~~ [inductor] Add an AOT compilation mode for Inductor CPP backend Mar 1, 2023

desertfire closed this Mar 3, 2023

desertfire added a commit that referenced this pull request Mar 5, 2023

[reland][inductor] Add an AOT compilation mode for Inductor CPP backend

b882e5b

Summary: This is a reland of #94822 ghstack-source-id: 1aa9136 Pull Request resolved: #95985

pytorchmergebot pushed a commit that referenced this pull request Mar 7, 2023

[reland][inductor] Add an AOT compilation mode for Inductor CPP backend

f11dc91

Summary: This is a reland of #94822 ghstack-source-id: dc88f33 Pull Request resolved: #95985

pytorchmergebot pushed a commit that referenced this pull request Mar 8, 2023

[reland][inductor] Add an AOT compilation mode for Inductor CPP backe…

deaf9e5

…nd (#95985) Summary: This is a reland of #94822 Pull Request resolved: #95985 Approved by: https://github.com/jansel

desertfire mentioned this pull request Mar 10, 2023

[reland2][inductor] Add an AOT compilation mode for Inductor CPP backend #96520

Closed

desertfire added a commit that referenced this pull request Mar 10, 2023

[reland2][inductor] Add an AOT compilation mode for Inductor CPP backend

b2f0e58

Summary: This is a reland of #94822. Solved the long compilation issue for inductor cpp tests. [ghstack-poisoned]

facebook-github-bot deleted the gh/desertfire/69/head branch June 8, 2023 16:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[inductor] Add an AOT compilation mode for Inductor CPP backend#94822

[inductor] Add an AOT compilation mode for Inductor CPP backend#94822
desertfire wants to merge 8 commits intogh/desertfire/69/basefrom
gh/desertfire/69/head

desertfire commented Feb 14, 2023 •

edited

Loading

Uh oh!

pytorch-bot bot commented Feb 14, 2023 •

edited

Loading

Uh oh!

desertfire commented Feb 14, 2023

Uh oh!

jansel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

voznesenskym commented Feb 17, 2023

Uh oh!

jansel left a comment

Uh oh!

desertfire commented Feb 28, 2023

Uh oh!

clee2000 commented Mar 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

desertfire commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94822

❌ 1 Failures

Uh oh!

desertfire commented Feb 14, 2023

Uh oh!

jansel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

voznesenskym commented Feb 17, 2023

Uh oh!

jansel left a comment

Choose a reason for hiding this comment

Uh oh!

desertfire commented Feb 28, 2023

Uh oh!

clee2000 commented Mar 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

desertfire commented Feb 14, 2023 •

edited

Loading

pytorch-bot bot commented Feb 14, 2023 •

edited

Loading