Skip to content

Added benchmark file for FA3 SDPA#173026

Closed
howardzhang-cv wants to merge 4 commits intogh/howardzhang-cv/9/basefrom
gh/howardzhang-cv/9/head
Closed

Added benchmark file for FA3 SDPA#173026
howardzhang-cv wants to merge 4 commits intogh/howardzhang-cv/9/basefrom
gh/howardzhang-cv/9/head

Conversation

@howardzhang-cv
Copy link
Copy Markdown
Contributor

@howardzhang-cv howardzhang-cv commented Jan 22, 2026 •
edited
Loading

Stack from ghstack (oldest at bottom):

Summary:
Added benchmark file for FA3 SDPA
Compares FA3 to FA2 fp16, bf16
Compares FA3 bf16 to FA3 fp8

Test Plan: python benchmarks/transformer/sdpa_fa3.py

@howardzhang-cv
Update
88dd236 
[ghstack-poisoned]
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot Bot commented Jan 22, 2026 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/173026

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 2216e3a with merge base cee5acf (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot Bot added the topic: not user facing topic category label Jan 22, 2026
howardzhang-cv added a commit that referenced this pull request Jan 22, 2026
@howardzhang-cv
Added benchmark file for FA3 SDPA
cdd9d19 
Summary:
Added benchmark file for FA3 SDPA
Compares FA3 to FA2 fp16, bf16
Compares FA3 bf16 to FA3 fp8

Test Plan: python benchmarks/transformer/sdpa_fa3.py


ghstack-source-id: d23f288
Pull-Request: #173026
This was referenced Jan 22, 2026
Closed
Closed
Closed
@howardzhang-cv howardzhang-cv marked this pull request as draft January 22, 2026 01:11
@howardzhang-cv
Update
b123e0a 
[ghstack-poisoned]
howardzhang-cv added a commit that referenced this pull request Jan 22, 2026
@howardzhang-cv
Added benchmark file for FA3 SDPA
ef0783e 
Summary:
Added benchmark file for FA3 SDPA
Compares FA3 to FA2 fp16, bf16
Compares FA3 bf16 to FA3 fp8

Test Plan: python benchmarks/transformer/sdpa_fa3.py


ghstack-source-id: 7f6e9ac
Pull-Request: #173026
@howardzhang-cv howardzhang-cv marked this pull request as ready for review January 26, 2026 22:04
@howardzhang-cv
Update
3a78748 
[ghstack-poisoned]
howardzhang-cv added a commit that referenced this pull request Jan 26, 2026
@howardzhang-cv
Added benchmark file for FA3 SDPA
d3cb13d 
Summary:
Added benchmark file for FA3 SDPA
Compares FA3 to FA2 fp16, bf16
Compares FA3 bf16 to FA3 fp8

Test Plan: python benchmarks/transformer/sdpa_fa3.py


ghstack-source-id: 6588eaa
Pull-Request: #173026
drisspg
drisspg reviewed Jan 26, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@drisspg drisspg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just update this one: https://github.com/pytorch/pytorch/blob/main/benchmarks/transformer/sdpa.py to support Impls for fa e.g. fa2 fa3 and fa4

@howardzhang-cv
Update
2216e3a 
[ghstack-poisoned]
howardzhang-cv added a commit that referenced this pull request Jan 27, 2026
@howardzhang-cv
Added benchmarking for FA3 SDPA
3650916 
Summary:
Added benchmarking for FA3 SDPA in benchmarks/transformer/sdpa.py
Compares FA3 to FA2 for bf16

Test Plan: python benchmarks/transformer/sdpa.py

ghstack-source-id: 098c956
Pull-Request: #173026
@howardzhang-cv
Copy link
Copy Markdown
Contributor Author

howardzhang-cv commented Jan 27, 2026

can we just update this one: https://github.com/pytorch/pytorch/blob/main/benchmarks/transformer/sdpa.py to support Impls for fa e.g. fa2 fa3 and fa4

Yeah I originally made a new file because I didn't want to modify existing benchmarks, but this makes more sense for DRY. I just added flash_impl as a config to sdpa.py. Could not add FA4 as an option because there seems to be some backwards pass issues, not sure if it's just my build.

@howardzhang-cv
Copy link
Copy Markdown
Contributor Author

howardzhang-cv commented Jan 27, 2026

@pytorchbot merge

@pytorch-bot pytorch-bot Bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 27, 2026
@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Jan 27, 2026

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions Bot deleted the gh/howardzhang-cv/9/head branch February 27, 2026 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@drisspg drisspg drisspg approved these changes

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@howardzhang-cv @pytorchmergebot @drisspg