Skip to content

fix: prelu perf gap on Unet#3717

Merged
zewenli98 merged 8 commits intomainfrom
decompose_prelu
Aug 20, 2025
Merged

fix: prelu perf gap on Unet#3717
zewenli98 merged 8 commits intomainfrom
decompose_prelu

Conversation

@zewenli98
Copy link
Copy Markdown
Collaborator

@zewenli98 zewenli98 commented Jul 24, 2025 •
edited
Loading

Description

In Monai/Unet, I observed that Torch-TRT's prelu is slower than ONNX-TRT, and what is worse is that using prelu converter is even slower than decomposing it.

Update: The perf issue is because to_copy forces to insert Cast Layer.

Fixes #3715 #3723

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@zewenli98 zewenli98 requested review from narendasan and peri044 July 24, 2025 21:58
@zewenli98 zewenli98 self-assigned this Jul 24, 2025
@github-actions github-actions bot added component: lowering Issues re: The lowering / preprocessing passes component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Jul 24, 2025
@zewenli98 zewenli98 marked this pull request as draft July 26, 2025 00:07
@github-actions github-actions bot added component: conversion Issues re: Conversion stage and removed component: lowering Issues re: The lowering / preprocessing passes labels Jul 28, 2025
@zewenli98 zewenli98 changed the title Decompose prelu Fix prelu perf gap Jul 28, 2025
@zewenli98 zewenli98 changed the title Fix prelu perf gap fix: prelu perf gap on Unet Jul 28, 2025
@zewenli98 zewenli98 marked this pull request as ready for review July 28, 2025 21:03
@github-actions github-actions bot added the component: converters Issues re: Specific op converters label Jul 28, 2025
narendasan
narendasan approved these changes Jul 28, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@narendasan narendasan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@narendasan
Copy link
Copy Markdown
Collaborator

narendasan commented Jul 28, 2025

If possible consider adding a test, we talked a bit about a new INetwork verification test based on expected numbers of ops. Might be a simple case to build the harness for

@zewenli98
Copy link
Copy Markdown
Collaborator Author

zewenli98 commented Jul 28, 2025

@narendasan I'll merge this PR now and cherrypick to the release 2.8, and then work on the number of ops check in another PR for the next release. sounds like a plan?

@zewenli98
Copy link
Copy Markdown
Collaborator Author

zewenli98 commented Jul 28, 2025

there's an error. will merge after fixing

HolyWu
HolyWu reviewed Jul 29, 2025
View reviewed changes
@zewenli98
Copy link
Copy Markdown
Collaborator Author

zewenli98 commented Jul 29, 2025 

def forward(self, x):
    x_1 = x + 1
    y = torch.ops.aten._to_copy.default(x_1, dtype=torch.float)
    z = torch.ops.aten._to_copy.default(x_1, dtype=torch.float)
    return y, z

this case fails because self.engine.num_io_tensors=2, len(self.input_names)=1, len(self.output_names)=2, which doesn't meet the assertion:

assert self.engine.num_io_tensors == len(self.input_names) + len(self.output_names)

self.input_names: ['x']
self.output_names: ['output0', 'output1']

Since output0 is invalid, it gets self.engine.num_io_tensors=2.

@github-actions github-actions bot removed the component: converters Issues re: Specific op converters label Jul 29, 2025
@zewenli98 zewenli98 requested a review from narendasan August 8, 2025 23:05
narendasan
narendasan reviewed Aug 12, 2025
View reviewed changes
py/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py
for i, output in enumerate(outputs):
# In some cases, the same output tensor may be marked multiple times, such as _to_oppy,
# so we skip marking if the output is already marked
if id(output) in marked_outputs_ids:
Copy link
Copy Markdown
Collaborator

@narendasan narendasan Aug 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this id function come from?

Copy link
Copy Markdown
Collaborator Author

@zewenli98 zewenli98 Aug 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

id() is a built-in function that returns the "identity" of an object.

peri044
peri044 reviewed Aug 12, 2025
View reviewed changes
py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py Outdated
args[0],
kwargs.get("dtype", args[0].dtype),
force_layer=True,
force_layer=False,
Copy link
Copy Markdown
Collaborator

@peri044 peri044 Aug 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there might be cases where we might need to actually force_layer=True. Do you know when that would be useful?. Also consider adding a comment here conveying that force_layer=False results in better performance.

Copy link
Copy Markdown
Collaborator Author

@zewenli98 zewenli98 Aug 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know when we need force_layer=True? My understanding is that 1) Since cast_trt_tensor would be called even if the force_layer=False, it only adds Cast Layer if necessary. 2) If an operator satisfies not is_only_operator_on_placeholder, that means there's other operator on the placeholder or the lowering pass repair_input_as_output to promise "input is not output", so we don't have to explicitly add Cast Layer.

@zewenli98 zewenli98 merged commit 33a53cf into main Aug 20, 2025
84 of 85 checks passed
@zewenli98 zewenli98 mentioned this pull request Aug 21, 2025
Closed
@zewenli98 zewenli98 deleted the decompose_prelu branch January 12, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@narendasan narendasan narendasan approved these changes

@peri044 peri044 peri044 left review comments

+1 more reviewer

@HolyWu HolyWu HolyWu left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@zewenli98 zewenli98

Labels

cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

🐛 [Bug] Torch-TRT's prelu is slower than ONNX-TRT

5 participants

@zewenli98 @narendasan @HolyWu @peri044 @facebook-github-bot