Skip to content

Fix profiling with record_shapes=True and nested tensor#82854

Closed
ngimel wants to merge 15 commits intomasterfrom
ngimel/profiler_shapes
Closed

Fix profiling with record_shapes=True and nested tensor#82854
ngimel wants to merge 15 commits intomasterfrom
ngimel/profiler_shapes

Conversation

@ngimel
Copy link
Collaborator

@ngimel ngimel commented Aug 4, 2022

We probably want more systematic fix for other tensorImpls that don't support sizes (and also there are other places in profiler that call sizes unchecked, which should be fixed), but this is to unblock Transformer profiling.

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 4, 2022

🔗 Helpful links

✅ No Failures (1 Pending)

As of commit 4ccd3b8 (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚


This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@robieta
Copy link
Contributor

robieta commented Aug 5, 2022

Is the goal to actually get shapes out, or just to not break? Because if it's the latter I think we should just change

if (t.defined()) {

to

if (t.defined() && !t.is_nested()) {  // TODO: Handle this better

The reason is that we're looking at better support in #81824 (currently being refactored to use the underlying data for better overhead) and then we could handle SizesStridesPolicy in a fully robust way. WDYT?

@ngimel
Copy link
Collaborator Author

ngimel commented Aug 5, 2022

Yeah the goal is to just not break, looking forward to proper support.

Natalia Gimelshein added 2 commits August 5, 2022 00:19
Copy link
Contributor

@robieta robieta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. (You'll need to rip out the unit test?)

@ngimel
Copy link
Collaborator Author

ngimel commented Aug 5, 2022

Unit test passes as it should (nested tensors call regular mm underneath, and regular mm has correct sizes), and I'd rather not rip it out because otherwise #81824 will break nested tensor profiling again.

@robieta
Copy link
Contributor

robieta commented Aug 5, 2022

Unit test passes as it should (nested tensors call regular mm underneath, and regular mm has correct sizes), and I'd rather not rip it out because otherwise #81824 will break nested tensor profiling again.

Gotcha. Thanks for the explanation.

@ngimel
Copy link
Collaborator Author

ngimel commented Aug 5, 2022

@pytorchbot merge -g

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a merge job. Check the current status here

@github-actions
Copy link
Contributor

github-actions bot commented Aug 5, 2022

Hey @ngimel.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

facebook-github-bot pushed a commit that referenced this pull request Aug 7, 2022
)

Summary:
We probably want more systematic fix for other tensorImpls that don't support sizes (and also there are other places in profiler that call sizes unchecked, which should be fixed), but this is to unblock Transformer profiling.

Pull Request resolved: #82854
Approved by: https://github.com/robieta

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/bdb0abb234e4914d68a99abf06184b3926b0abf0

Reviewed By: kit1980

Differential Revision: D38478641

Pulled By: ngimel

fbshipit-source-id: e4e774042029d206ebf7fb6a92c84182fae8640e
@github-actions github-actions bot deleted the ngimel/profiler_shapes branch February 18, 2024 01:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants