Skip to content

futher scheduler changes for invoke_quant: prologue low prec, (slightly) more aggressive fusion#145104

Closed
eellison wants to merge 13 commits intogh/eellison/752/basefrom
gh/eellison/752/head
Closed

futher scheduler changes for invoke_quant: prologue low prec, (slightly) more aggressive fusion#145104
eellison wants to merge 13 commits intogh/eellison/752/basefrom
gh/eellison/752/head

Conversation

@eellison
Copy link
Contributor

@eellison eellison commented Jan 17, 2025

[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Jan 17, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/145104

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 1 Pending

As of commit cf18367 with merge base 49082f9 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]
[ghstack-poisoned]
eellison added a commit that referenced this pull request Jan 17, 2025
…ly) more aggressive fusion

ghstack-source-id: fbbbda4
Pull Request resolved: #145104
@pytorch-bot pytorch-bot bot temporarily deployed to upload-benchmark-results January 17, 2025 20:55 Inactive
@pytorch-bot pytorch-bot bot temporarily deployed to upload-benchmark-results January 17, 2025 20:55 Inactive
@eellison eellison requested review from Chillee and drisspg January 17, 2025 21:00
@eellison eellison added the topic: not user facing topic category label Jan 17, 2025
@@ -3477,36 +3521,7 @@ def can_fuse(self, node1: BaseSchedulerNode, node2: BaseSchedulerNode) -> bool:
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe move the whole if block regarding prologue fusion to can_fuse_prologue to make can_fuse smaller.

"""
Heuristics to avoid benchmarking predictably slow prologue fusions
"""
# user opt into more aggressive prologue fusion, dont use heuristics
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

User opt in by using invoke_quant?

Copy link
Contributor

@jansel jansel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failing tests?

[ghstack-poisoned]
eellison added a commit that referenced this pull request Jan 28, 2025
…ly) more aggressive fusion

ghstack-source-id: 74e7765
Pull Request resolved: #145104
@pytorch-bot pytorch-bot bot temporarily deployed to upload-benchmark-results January 28, 2025 01:30 Inactive
@pytorch-bot pytorch-bot bot temporarily deployed to upload-benchmark-results January 28, 2025 01:30 Inactive
[ghstack-poisoned]
eellison added a commit that referenced this pull request Jan 29, 2025
…ly) more aggressive fusion

ghstack-source-id: 676617b
Pull Request resolved: #145104
@pytorch-bot pytorch-bot bot temporarily deployed to upload-benchmark-results January 29, 2025 01:23 Inactive
@pytorch-bot pytorch-bot bot had a problem deploying to upload-benchmark-results January 29, 2025 01:23 Failure
Comment on lines +495 to +497
self.low_precision_codegen_ops = OrderedSet[str]()
# more aggressive prologue fusion
self.invoke_quant_ops = OrderedSet[str]()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.low_precision_codegen_ops = OrderedSet[str]()
# more aggressive prologue fusion
self.invoke_quant_ops = OrderedSet[str]()
self.low_precision_codegen_ops : OrderedSet[str] = OrderedSet()
# more aggressive prologue fusion
self.invoke_quant_ops : OrderedSet[str] = OrderedSet()

to avoid runtime calls to __getitem__.

@pytorch-bot pytorch-bot bot temporarily deployed to upload-benchmark-results January 30, 2025 00:51 Inactive
@pytorch-bot pytorch-bot bot temporarily deployed to upload-benchmark-results January 30, 2025 00:51 Inactive
…ec, (slightly) more aggressive fusion"


Respect invoke_quant low precision options, also, be more aggressive in attepmting fusion.

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov

[ghstack-poisoned]
…ec, (slightly) more aggressive fusion"


Respect invoke_quant low precision options, also, be more aggressive in attepmting fusion.

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov

[ghstack-poisoned]
eellison added a commit that referenced this pull request Jan 30, 2025
…ly) more aggressive fusion

ghstack-source-id: 601adc8
Pull Request resolved: #145104
…ec, (slightly) more aggressive fusion"


Respect invoke_quant low precision options, also, be more aggressive in attepmting fusion.

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov

[ghstack-poisoned]
eellison added a commit that referenced this pull request Jan 31, 2025
…ly) more aggressive fusion

ghstack-source-id: 7247fdf
Pull Request resolved: #145104
[ghstack-poisoned]
eellison added a commit that referenced this pull request Feb 6, 2025
…ly) more aggressive fusion

ghstack-source-id: 7ca092b
Pull Request resolved: #145104
[ghstack-poisoned]
[ghstack-poisoned]
eellison added a commit that referenced this pull request Feb 8, 2025
…ly) more aggressive fusion

ghstack-source-id: e5af8b5
Pull Request resolved: #145104
[ghstack-poisoned]
eellison added a commit that referenced this pull request Feb 9, 2025
…ly) more aggressive fusion

ghstack-source-id: e04f409
Pull Request resolved: #145104
@eellison
Copy link
Contributor Author

eellison commented Feb 9, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

@eellison
Copy link
Contributor Author

@pytorchbot merge -f "rocm test taking a while"

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the gh/eellison/752/head branch March 13, 2025 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants