Adding limited support for aten::Int by narendasan · Pull Request #870 · pytorch/TensorRT

narendasan · 2022-02-14T17:12:40Z

Description

This PR adds support for aten::Int / prim::NumToTensor in a few limited cases.

a direct prim::NumToTensor -> aten::Int
prim::NumToTensor -> X -> aten::Int in cases where the tensors used are single use and can safely be fused

Fixes #513, Fixes #707
Partially: #867, #829, #785, #711, #660

Type of change

Please delete options that are not relevant and/or add your own.

New feature (non-breaking change which adds functionality)

Checklist:

My code follows the style guidelines of this project (You can use the linters)
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas and hacks
I have made corresponding changes to the documentation
I have added tests to verify my fix or my feature
New and existing unit tests pass locally with my changes

github-actions

Code conforms to Python style guidelines

github-actions

There are some changes that do not conform to C++ style guidelines:

diff --git a/workspace/core/lowering/passes/remove_unnecessary_casts.cpp b/tmp/changes.txt
index 7f6cc85..064a2cb 100644
--- a/workspace/core/lowering/passes/remove_unnecessary_casts.cpp
+++ b/tmp/changes.txt
@@ -1,5 +1,5 @@
-#include "torch/csrc/jit/passes/subgraph_rewrite.h"
#include "torch/csrc/jit/ir/constants.h"
+#include "torch/csrc/jit/passes/subgraph_rewrite.h"

#include "core/util/prelude.h"

@@ -10,7 +10,6 @@ namespace core {
namespace lowering {
namespace passes {

-
// Presumably this is safe since torch::jit::EraseNumberTypesOnBlock exists which just
// removes prim::TensorToNum, aten::Float, aten::Int and prim::NumToTensor nodes outright
void RemoveUnnecessaryCasts(std::shared_ptr<torch::jit::Graph>& graph) {
@@ -77,8 +76,8 @@ void RemoveSingleUse0DTensors(std::shared_ptr<torch::jit::Graph>& g) {
              if (user->output()->uses().size() == 1) {
                auto potential_cast = user->output()->uses()[0].user;
                // The downstream user is aten::Int
-                if (potential_cast->kind() == c10::Symbol::fromQualString("aten::Int")
-                    || potential_cast->kind() == c10::Symbol::fromQualString("aten::Float")) {
+                if (potential_cast->kind() == c10::Symbol::fromQualString("aten::Int") ||
+                    potential_cast->kind() == c10::Symbol::fromQualString("aten::Float")) {
                  LOG_GRAPH("Downstream user is aten::Int/aten::Float");
                  auto arg = use.offset;

@@ -88,13 +87,11 @@ void RemoveSingleUse0DTensors(std::shared_ptr<torch::jit::Graph>& g) {
                        LOG_GRAPH("Input " << k << " is a Tensor");
                        if (user->inputs()[k]->node()->kind() == c10::Symbol::fromQualString("prim::NumToTensor")) {
                          auto num_to_tensor = user->inputs()[k]->node();
-                          
-                          LOG_GRAPH("Found a prim::NumToTensor / aten::[Int/Float] pair with an intermediate operation:\n    " 
-                                      << *(*it)
-                                      << *num_to_tensor
-                                      << *user 
-                                      << *potential_cast);
-                          
+
+                          LOG_GRAPH(
+                              "Found a prim::NumToTensor / aten::[Int/Float] pair with an intermediate operation:\n    "
+                              << *(*it) << *num_to_tensor << *user << *potential_cast);
+
                          // Replace the Tensor Constant with a scalar constant
                          LOG_GRAPH("Deleting 0-dim Tensor: " << **it);
                          torch::jit::WithInsertPoint gaurd(*it);
@@ -126,19 +123,16 @@ void RemoveSingleUse0DTensors(std::shared_ptr<torch::jit::Graph>& g) {
                            // has a different schema than the original
                            case c10::aten::add:
                              new_node = g->create(
-                                user->kind(),
-                                torch::jit::ArrayRef<torch::jit::Value*>({user->inputs()[0], user->inputs()[1]}),
-                                1);
+                                  user->kind(),
+                                  torch::jit::ArrayRef<torch::jit::Value*>({user->inputs()[0], user->inputs()[1]}),
+                                  1);
                              new_node->insertAfter(user);
                              new_node->outputs()[0]->setType(c10::IntType::get());
                              user->outputs()[0]->replaceAllUsesWith(new_node->outputs()[0]);
                              user->destroy();
                              break;
                            default:
-                              new_node = g->create(
-                                user->kind(),
-                                user->inputs(),
-                                1);
+                              new_node = g->create(user->kind(), user->inputs(), 1);
                              new_node->insertAfter(user);
                              new_node->outputs()[0]->setType(c10::IntType::get());
                              user->outputs()[0]->replaceAllUsesWith(new_node->outputs()[0]);
@@ -148,7 +142,7 @@ void RemoveSingleUse0DTensors(std::shared_ptr<torch::jit::Graph>& g) {

                          LOG_GRAPH("New intermediate operation: " << *new_node);
                          LOG_GRAPH(new_node->schema());
-                          
+
                          // Delete aten::Int
                          LOG_GRAPH("Deleting aten::[Int/Float]: " << *potential_cast);
                          potential_cast->output()->replaceAllUsesWith(potential_cast->inputs()[0]);
@@ -163,12 +157,11 @@ void RemoveSingleUse0DTensors(std::shared_ptr<torch::jit::Graph>& g) {
          }
        }
      }
-    } 
+    }
  }
  LOG_ERROR("Post removing single use 0-dim Tensor operations: " << *g);
}

-
} // namespace passes
} // namespace lowering
} // namespace core
diff --git a/workspace/tests/core/lowering/test_remove_unnecessary_casts.cpp b/tmp/changes.txt
index ef370a8..62f913e 100644
--- a/workspace/tests/core/lowering/test_remove_unnecessary_casts.cpp
+++ b/tmp/changes.txt
@@ -102,8 +102,7 @@ TEST(LoweringPasses, RemoveSingleUse0DTensorsIntCorrectly) {

  auto first_op = *(sg->block()->nodes().begin());
  torch::jit::WithInsertPoint guard(first_op);
-  torch::jit::Value* r = sg->insertConstant(
-      c10::scalar_to_tensor(8), c10::nullopt, first_op->scope());
+  torch::jit::Value* r = sg->insertConstant(c10::scalar_to_tensor(8), c10::nullopt, first_op->scope());
  r->copyMetadata(first_op->output());
  r->setType(c10::TensorType::get());
  first_op->output()->replaceAllUsesWith(r);
@@ -141,8 +140,7 @@ TEST(LoweringPasses, RemoveSingleUse0DTensorsFloatCorrectly) {

  auto first_op = *(sg->block()->nodes().begin());
  torch::jit::WithInsertPoint guard(first_op);
-  torch::jit::Value* r = sg->insertConstant(
-      c10::scalar_to_tensor(8.0), c10::nullopt, first_op->scope());
+  torch::jit::Value* r = sg->insertConstant(c10::scalar_to_tensor(8.0), c10::nullopt, first_op->scope());
  r->copyMetadata(first_op->output());
  r->setType(c10::TensorType::get());
  first_op->output()->replaceAllUsesWith(r);
ERROR: Some files do not conform to style guidelines

ncomly-nvidia · 2022-02-22T16:48:10Z

Likely fixes #732 as well.

This commit adds a pass to lower out aten::[Int/Float/Bool], aten::NumToTensor pairs w.o. exception. We are assumming this is safe as there are similar passes in PyTorch for ONNX lowering however the scope of this rule is intentionally limited to avoid possible cases where it is not safe. Therefore it should not be expected that all aten::Int issues will be solved with this change and the operator itself remains a limitation of TorchTRT Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

0D Tensors Now we remove select more complex aten::Int cases found in models such as BERT, like the following: ``` graph(%0: int): %1: Tensor = prim::Constant[value={8}]() %2: int = prim::Constant[value=1]() %3: Tensor = prim::NumToTensor(%0) %4: Tensor = aten::add(%1, %3, %2) %5: int = aten::Int(%4) %6: int = aten::add(%5, %5) return (%6)"; graph(%0: int): %1: int = prim::Constant[value=8]() %4: int = aten::add(%1, %0) %6: int = aten::add(%4, %4) return (%6)"; ``` Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

Lower logging level on debug info Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

github-actions

There are some changes that do not conform to Python style guidelines:

--- /workspace/tests/modules/hub.py	(original)
+++ /workspace/tests/modules/hub.py	(reformatted)
@@ -191,7 +191,6 @@
conditional_script_model = torch.jit.script(conditional_model)
torch.jit.save(conditional_script_model, "conditional_scripted.jit.pt")

-
enc = BertTokenizer.from_pretrained("bert-base-uncased")
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)
Reformatting /workspace/tests/modules/hub.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_trt_intercompatibility.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_to_backend_api.py
ERROR: Some files do not conform to style guidelines

github-actions

There are some changes that do not conform to C++ style guidelines:

diff --git a/workspace/tests/cpp/test_default_input_types.cpp b/tmp/changes.txt
index 752f51e..a79ddaf 100644
--- a/workspace/tests/cpp/test_default_input_types.cpp
+++ b/tmp/changes.txt
@@ -116,4 +116,5 @@ TEST_P(CppAPITests, InputsRespectUserSettingFP32WeightsFP16In) {
INSTANTIATE_TEST_SUITE_P(
    CompiledModuleForwardIsCloseSuite,
    CppAPITests,
-    testing::Values(PathAndInput({"tests/modules/resnet18_traced.jit.pt", {{1, 3, 224, 224}}, {at::kFloat} /*unused*/, 2e-5})));
+    testing::Values(
+        PathAndInput({"tests/modules/resnet18_traced.jit.pt", {{1, 3, 224, 224}}, {at::kFloat} /*unused*/, 2e-5})));
ERROR: Some files do not conform to style guidelines

github-actions

There are some changes that do not conform to C++ style guidelines:

diff --git a/workspace/tests/cpp/test_default_input_types.cpp b/tmp/changes.txt
index 752f51e..a79ddaf 100644
--- a/workspace/tests/cpp/test_default_input_types.cpp
+++ b/tmp/changes.txt
@@ -116,4 +116,5 @@ TEST_P(CppAPITests, InputsRespectUserSettingFP32WeightsFP16In) {
INSTANTIATE_TEST_SUITE_P(
    CompiledModuleForwardIsCloseSuite,
    CppAPITests,
-    testing::Values(PathAndInput({"tests/modules/resnet18_traced.jit.pt", {{1, 3, 224, 224}}, {at::kFloat} /*unused*/, 2e-5})));
+    testing::Values(
+        PathAndInput({"tests/modules/resnet18_traced.jit.pt", {{1, 3, 224, 224}}, {at::kFloat} /*unused*/, 2e-5})));
ERROR: Some files do not conform to style guidelines

github-actions

There are some changes that do not conform to Python style guidelines:

--- /workspace/tests/modules/hub.py	(original)
+++ /workspace/tests/modules/hub.py	(reformatted)
@@ -191,7 +191,6 @@
conditional_script_model = torch.jit.script(conditional_model)
torch.jit.save(conditional_script_model, "conditional_scripted.jit.pt")

-
enc = BertTokenizer.from_pretrained("bert-base-uncased")
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)
Reformatting /workspace/tests/modules/hub.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_trt_intercompatibility.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_to_backend_api.py
ERROR: Some files do not conform to style guidelines

github-actions

Code conforms to C++ style guidelines

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

peri044 · 2022-04-05T04:53:48Z

core/lowering/passes/remove_set_attrs.cpp

+
+  std::string set_attr_pattern = R"IR(
+        graph(%self, %0):
+            None = prim::SetAttr[name="_has_warned"](%self, %0)


Can you mention in a comment about this specific attribute _has_warned ? Where is this used ? etc ..

peri044 · 2022-04-05T04:54:12Z

core/lowering/passes/remove_set_attrs.cpp

+namespace lowering {
+namespace passes {
+
+void RemoveSetAttrs(const torch::jit::Module& mod, std::string method_name) {


Where is this lowering pass used ?

There was a version of transformers that had this and it was breaking the conversion process since setattr does not have a schema. But later versions dont use this so I removed it from the set of active passes

core/lowering/passes/remove_unnecessary_casts.cpp

narendasan added the WIP Work is in progress, pull request should not be merged yet label Feb 14, 2022

github-actions bot added component: core Issues re: The core compiler component: lowering Issues re: The lowering / preprocessing passes component: tests Issues re: Tests labels Feb 14, 2022

narendasan linked an issue Feb 14, 2022 that may be closed by this pull request

🐛 [Bug] Torch-TensorRT do not support gpt2 #867

Closed

github-actions bot approved these changes Feb 14, 2022

View reviewed changes

github-actions bot requested changes Feb 14, 2022

View reviewed changes

narendasan added 4 commits April 4, 2022 15:28

refactor: Apply linting

e63908b

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

chore: Add BERT to the model set

8139da9

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

narendasan force-pushed the aten_int_num_tensor branch from 30ee238 to 8139da9 Compare April 4, 2022 22:49

narendasan added 2 commits April 4, 2022 18:14

refactor(//core/lowering/passes/remove_unnecessary_casts):

72c7b76

Lower logging level on debug info Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

feat(//tests): Adding BERT to the test suite

7996a10

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

narendasan removed the WIP Work is in progress, pull request should not be merged yet label Apr 5, 2022

narendasan requested a review from peri044 April 5, 2022 01:23

github-actions bot requested changes Apr 5, 2022

View reviewed changes

github-actions bot approved these changes Apr 5, 2022

View reviewed changes

refactor: apply linting

83ae991

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

narendasan force-pushed the aten_int_num_tensor branch from af8d22d to 83ae991 Compare April 5, 2022 02:17

github-actions bot approved these changes Apr 5, 2022

View reviewed changes

peri044 reviewed Apr 5, 2022

View reviewed changes

ruoqianguo mentioned this pull request Apr 8, 2022

✨[Feature] Some operators need to be supported in Torch-TensorRT #766

Closed

narendasan merged commit b35f69a into master Apr 9, 2022

narendasan deleted the aten_int_num_tensor branch April 9, 2022 03:43

Conversation

narendasan commented Feb 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist:

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

ncomly-nvidia commented Feb 22, 2022

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

peri044 Apr 5, 2022

Choose a reason for hiding this comment

Uh oh!

peri044 Apr 5, 2022

Choose a reason for hiding this comment

Uh oh!

narendasan Apr 5, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

narendasan commented Feb 14, 2022 •

edited

Loading