Skip to content

[Profiler] add Preliminary queue depth computation#79874

Closed
davidchencsl wants to merge 6 commits intogh/davidchencsl/3/basefrom
gh/davidchencsl/3/head
Closed

[Profiler] add Preliminary queue depth computation#79874
davidchencsl wants to merge 6 commits intogh/davidchencsl/3/basefrom
gh/davidchencsl/3/head

Conversation

@davidchencsl
Copy link
Copy Markdown
Contributor

@davidchencsl davidchencsl commented Jun 20, 2022

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Jun 20, 2022

🔗 Helpful links

✅ No Failures (0 Pending)

As of commit 96aa3fb (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚


This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@davidchencsl davidchencsl requested a review from robieta June 20, 2022 16:59
davidchencsl added a commit that referenced this pull request Jun 20, 2022
ghstack-source-id: 4c060d6
Pull Request resolved: #79874
@davidchencsl davidchencsl changed the title add heuristic and idle time computation [Profiler] add heuristic and idle time computation Jun 20, 2022
davidchencsl added a commit that referenced this pull request Jun 21, 2022
ghstack-source-id: 5306353
Pull Request resolved: #79874
Test Plan:
Add test to check the queue depth

[ghstack-poisoned]
davidchencsl added a commit that referenced this pull request Jun 21, 2022
ghstack-source-id: c4c8da2
Pull Request resolved: #79874
Copy link
Copy Markdown
Contributor

@robieta robieta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High level looks good. Just a couple of minor tweaks needed.

Comment thread test/test_profiler.py Outdated
time.sleep(0.1)
basic_evaluation = _utils.BasicEvaluation(prof.profiler)
for entry in basic_evaluation.compute_queue_depth():
self.assertTrue(entry.queue_depth > 0)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering the fact that we force the second half to be overhead bound, why is queue_depth > 0?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment thread torch/profiler/_utils.py
key=lambda x: x.start_us())

kernel_mapping: Dict[_KinetoEvent, int] = {}
for cuda_launch_event in cuda_launch_events:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an N**2 algorithm, but it can be done in O(N)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've change it to keep track of the last mapped kernel index, and start searching from that since list is sorted.

Comment thread test/test_profiler.py
Test Plan:
Add test to check the queue depth

[ghstack-poisoned]
davidchencsl added a commit that referenced this pull request Jun 21, 2022
ghstack-source-id: 11e4d04
Pull Request resolved: #79874
@davidchencsl
Copy link
Copy Markdown
Contributor Author

Added commit to fix the nits.

Test Plan:
Add test to check the queue depth

[ghstack-poisoned]
davidchencsl added a commit that referenced this pull request Jun 21, 2022
ghstack-source-id: 73b0e5d
Pull Request resolved: #79874
Copy link
Copy Markdown
Contributor

@robieta robieta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@davidchencsl
Copy link
Copy Markdown
Contributor Author

@pytorchmergebot merge -g

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

@pytorchbot successfully started a merge job. Check the current status here

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

@davidchencsl your PR has been successfully merged.

@github-actions
Copy link
Copy Markdown
Contributor

Hey @davidchencsl.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

facebook-github-bot pushed a commit that referenced this pull request Jun 22, 2022
Summary:
Pull Request resolved: #79874

Approved by: https://github.com/robieta

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/45e3afdef266b3776019f4e921ceff156e67a776

Reviewed By: atalman

Differential Revision: D37327457

Pulled By: davidchencsl

fbshipit-source-id: 2427ae32d87614e8a530e5b01d7cf593fe7e51e0
@davidchencsl davidchencsl changed the title [Profiler] add heuristic and idle time computation [Profiler] add Preliminary queue depth computation Jun 22, 2022
@facebook-github-bot facebook-github-bot deleted the gh/davidchencsl/3/head branch June 25, 2022 14:16
miladm pushed a commit to miladm/pytorch that referenced this pull request Jun 27, 2022
laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants