Add new pattern matchers for SDPA by Valentine233 · Pull Request #113004 · pytorch/pytorch

Valentine233 · 2023-11-06T03:06:46Z

Add two new pattern matchers to enable SDPA in more models.

Pattern 14: BertLarge
Pattern 15: DistilBert

Perf on SPR:

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

pytorch-bot · 2023-11-06T03:06:49Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/113004

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 7591c0d with merge base 4115354 ():

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

pull / linux-focal-py3_8-clang9-xla / test (xla, 1, 1, linux.12xlarge, unstable) (gh)
Process completed with exit code 128.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

torch/_inductor/fx_passes/post_grad.py

torch/_inductor/fx_passes/fuse_attention.py

test/inductor/test_fused_attention.py

Valentine233 · 2024-01-10T05:51:02Z

Please help review this PR, thank you! @peterbell10 @bdhirsh @drisspg

eellison

Can you serialize the new patterns:

https://github.com/pytorch/pytorch/blob/main/torchgen/fuse_attention_patterns/gen_attention_patterns.py

test/inductor/test_fused_attention.py

Valentine233 · 2024-01-18T05:51:58Z

torch/_inductor/fx_passes/fuse_attention.py

Hi @eellison @drisspg,
The pattern 16 makes Bert CUDA go into efficient attention instead of sdpa math, causing accuracy issue. I disable the pattern for CUDA and add a comment.

Reproduce:
python test/distributed/test_dynamo_distributed.py -k test_hf_bert_ddp_inductor

Traceback log:

Error: :[2024-01-18 02:26:46,462] torch._dynamo.utils: [ERROR] Accuracy failed: allclose not within tol=0.0001 1159Error: :[2024-01-18 02:26:46,463] torch._dynamo.utils: [ERROR] Accuracy failed: allclose not within tol=0.0001 1160Error: :[2024-01-18 02:26:46,463] torch._dynamo.utils: [ERROR] Accuracy failed: allclose not within tol=0.0001 1161Error: :[2024-01-18 02:26:46,463] torch._dynamo.utils: [ERROR] Accuracy failed: allclose not within tol=0.0001 1162Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] Caught exception: 1163Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 1164Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 650, in run_test 1165Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 1166Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 1167Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] fn() 1168Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2683, in wrapper 1169Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 1170Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 175, in wrapper 1171Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 1172Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 161, in wrapper 1173Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 1174Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/mock.py", line 1379, in patched 1175Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*newargs, **newkeywargs) 1176Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 326, in test_hf_bert_ddp_inductor 1177Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] self._test_hf_bert_ddp_inductor(static_graph=False) 1178Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 318, in _test_hf_bert_ddp_inductor 1179Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] run_hf_bert_ddp(self, model, inputs, "inductor") 1180Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 242, in run_hf_bert_ddp 1181Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] self.assertTrue(same(correct_results, opt_results)) 1182Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 687, in assertTrue 1183Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] raise self.failureException(msg) 1184Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] AssertionError: False is not true 1185Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] 1186Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 1187Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_dynamo_distributed.py -k test_hf_bert_ddp_inductor 1188Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] 1189Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 1190Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] exiting process 3 with exit code: 10 1191Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] Caught exception: 1192Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 1193Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 650, in run_test 1194Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 1195Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 1196Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] fn() 1197Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2683, in wrapper 1198Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 1199Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 175, in wrapper 1200Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 1201Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 161, in wrapper 1202Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 1203Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/mock.py", line 1379, in patched 1204Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*newargs, **newkeywargs) 1205Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 326, in test_hf_bert_ddp_inductor 1206Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] self._test_hf_bert_ddp_inductor(static_graph=False) 1207Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 318, in _test_hf_bert_ddp_inductor 1208Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] run_hf_bert_ddp(self, model, inputs, "inductor") 1209Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 242, in run_hf_bert_ddp 1210Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] self.assertTrue(same(correct_results, opt_results)) 1211Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 687, in assertTrue 1212Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] raise self.failureException(msg) 1213Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] AssertionError: False is not true 1214Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] 1215Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 1216Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_dynamo_distributed.py -k test_hf_bert_ddp_inductor 1217Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] 1218Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 1219Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] exiting process 2 with exit code: 10 1220Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] Caught exception: 1221Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 1222Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 650, in run_test 1223Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 1224Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 1225Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] fn() 1226Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2683, in wrapper 1227Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 1228Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 175, in wrapper 1229Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 1230Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 161, in wrapper 1231Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 1232Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/mock.py", line 1379, in patched 1233Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*newargs, **newkeywargs) 1234Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 326, in test_hf_bert_ddp_inductor 1235Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] self._test_hf_bert_ddp_inductor(static_graph=False) 1236Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 318, in _test_hf_bert_ddp_inductor 1237Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] run_hf_bert_ddp(self, model, inputs, "inductor") 1238Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 242, in run_hf_bert_ddp 1239Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] self.assertTrue(same(correct_results, opt_results)) 1240Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 687, in assertTrue 1241Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] raise self.failureException(msg) 1242Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] AssertionError: False is not true 1243Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] 1244Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 1245Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_dynamo_distributed.py -k test_hf_bert_ddp_inductor 1246Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] 1247Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 1248Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 1249Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] Caught exception: 1250Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 1251Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 650, in run_test 1252Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 1253Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 1254Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] fn() 1255Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2683, in wrapper 1256Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 1257Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 175, in wrapper 1258Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 1259Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 161, in wrapper 1260Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 1261Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/mock.py", line 1379, in patched 1262Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] return func(*newargs, **newkeywargs) 1263Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 326, in test_hf_bert_ddp_inductor 1264Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] self._test_hf_bert_ddp_inductor(static_graph=False) 1265Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 318, in _test_hf_bert_ddp_inductor 1266Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] run_hf_bert_ddp(self, model, inputs, "inductor") 1267Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 242, in run_hf_bert_ddp 1268Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] self.assertTrue(same(correct_results, opt_results)) 1269Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 687, in assertTrue 1270Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] raise self.failureException(msg) 1271Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] AssertionError: False is not true 1272Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] 1273Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 1274Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_dynamo_distributed.py -k test_hf_bert_ddp_inductor 1275Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] 1276Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 1277Error: 1-18 02:26:46,464] torch.testing._internal.common_distributed: [ERROR] exiting process 0 with exit code: 10

@eellison @drisspg Please help review, thanks~

Is the error still occurring? it doesnt look like you have any failures. What did you fix ?

I think @Valentine233 makes this pattern only hit on CPU in extra pattern check to avoid this accuracy issue.

CC @drisspg on efficient attention numerics mismatch: https://github.com/pytorch/pytorch/pull/113004/files#diff-d2f5fd73265c51de38cecb3b0309a2e555e8260feb7bd16a963fd9b89321a537R664

Can you make an issue to track this? I am not sure what the tolerances were for the test but depending on the deviation they could be within the regime of the fused kernels using iterative softmax?

Sure, I would create an issue and assign to you.

Please see #117957.

Valentine233 · 2024-01-20T00:43:31Z

@pytorchbot merge

pytorchmergebot · 2024-01-20T00:46:07Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Valentine233 marked this pull request as draft November 6, 2023 03:06

Valentine233 added the topic: not user facing topic category label Nov 6, 2023

github-actions bot added module: inductor ciflow/inductor labels Nov 6, 2023

pytorchbot added the open source label Nov 6, 2023

Valentine233 commented Nov 6, 2023

View reviewed changes

torch/_inductor/fx_passes/post_grad.py Outdated Show resolved Hide resolved

Valentine233 marked this pull request as ready for review November 6, 2023 07:04

peterbell10 reviewed Nov 6, 2023

View reviewed changes

torch/_inductor/fx_passes/fuse_attention.py Outdated Show resolved Hide resolved

colesbury added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 6, 2023

Valentine233 force-pushed the new_sdpa_pattern branch 2 times, most recently from def7da7 to 1fd3a5b Compare December 14, 2023 06:21

Valentine233 requested review from Chillee, bdhirsh, jgong5 and peterbell10 December 14, 2023 06:42

Valentine233 force-pushed the new_sdpa_pattern branch from 1fd3a5b to 689f28d Compare December 14, 2023 07:12

Valentine233 force-pushed the new_sdpa_pattern branch from 689f28d to 2d1ff41 Compare January 5, 2024 02:22

Valentine233 added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 5, 2024

Valentine233 marked this pull request as draft January 5, 2024 02:43

Valentine233 force-pushed the new_sdpa_pattern branch 2 times, most recently from 2f87aa1 to 3af17c7 Compare January 7, 2024 14:10

Valentine233 marked this pull request as ready for review January 8, 2024 01:24

jgong5 reviewed Jan 8, 2024

View reviewed changes

test/inductor/test_fused_attention.py Show resolved Hide resolved

Valentine233 requested review from drisspg and jgong5 January 9, 2024 07:32

jgong5 approved these changes Jan 10, 2024

View reviewed changes

drisspg requested a review from eellison January 10, 2024 21:05

eellison reviewed Jan 10, 2024

View reviewed changes

test/inductor/test_fused_attention.py Outdated Show resolved Hide resolved

Valentine233 added 7 commits January 17, 2024 19:37

Update Add new pattern matchers for SDPA

8b4093d

add comments for sdpa patterns

d482632

add comment for sdpa 15 cuda

fdb3c71

add serializations for new patterns

9e7a0dc

Add UT for test_sdpa_rewriter_15_cuda

c8dd2a4

fix cuda pattern match bug

3637d21

add patterns with dropout

511ba22

Valentine233 force-pushed the new_sdpa_pattern branch 4 times, most recently from 540a61f to e1c9ff6 Compare January 18, 2024 05:51

Valentine233 commented Jan 18, 2024

View reviewed changes

Valentine233 force-pushed the new_sdpa_pattern branch from e1c9ff6 to ceaf0e1 Compare January 18, 2024 06:53

Valentine233 mentioned this pull request Jan 18, 2024

[Inductor] Request to support get_attr in pattern matcher #117735

Open

disable sfdp_pattern_16 CUDA

7591c0d

Valentine233 force-pushed the new_sdpa_pattern branch from ceaf0e1 to 7591c0d Compare January 18, 2024 07:55

Valentine233 requested a review from drisspg January 19, 2024 02:46

leslie-fang-intel approved these changes Jan 19, 2024

View reviewed changes

eellison approved these changes Jan 19, 2024

View reviewed changes

pytorchmergebot added the merging label Jan 20, 2024

pytorchmergebot closed this in d4cc1c5 Jan 20, 2024

pytorchmergebot added Merged and removed merging labels Jan 20, 2024

Valentine233 mentioned this pull request Jan 22, 2024

SDPA accuracy issue for CUDA (Disable temporarily) #117957

Open

sanchitintel mentioned this pull request Feb 14, 2024

Inductor pattern-matching seems to be sensitive to the presence/absence of clone nodes #119911

Closed

github-actions bot deleted the new_sdpa_pattern branch February 22, 2024 01:49

BoyuanFeng mentioned this pull request Apr 15, 2025

improve noop elimination for view #151095

Closed

Conversation

Valentine233 commented Nov 6, 2023 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/113004

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Valentine233 commented Jan 10, 2024

Uh oh!

eellison left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Valentine233 Jan 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Valentine233 Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

eellison Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

leslie-fang-intel Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

eellison Jan 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drisspg Jan 20, 2024

Choose a reason for hiding this comment

Uh oh!

Valentine233 Jan 20, 2024

Choose a reason for hiding this comment

Uh oh!

Valentine233 Jan 22, 2024

Choose a reason for hiding this comment

Uh oh!

Valentine233 commented Jan 20, 2024

Uh oh!

pytorchmergebot commented Jan 20, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Valentine233 commented Nov 6, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Nov 6, 2023 •

edited

Loading

Valentine233 Jan 18, 2024 •

edited

Loading

eellison Jan 19, 2024 •

edited

Loading