Skip to content

Enable torch.compile for triton backend#1422

Merged
merrymercy merged 10 commits intomainfrom
triton-torch-compile
Sep 14, 2024
Merged

Enable torch.compile for triton backend#1422
merrymercy merged 10 commits intomainfrom
triton-torch-compile

Conversation

@merrymercy
Copy link
Copy Markdown
Contributor

@merrymercy merrymercy commented Sep 14, 2024 •
edited
Loading

w/ torch.compile

python3 -m sglang.bench_latency --model meta-llama/Meta-Llama-3-8B --batch-size 1 --input 128 --output 8 --attention-backend triton --enable-torch-compile
Decode.  median latency: 0.00616 s, median throughput:    162.40 token/s

w/o torch.compile

python3 -m sglang.bench_latency --model meta-llama/Meta-Llama-3-8B --batch-size 1 --input 128 --output 8 --attention-backend triton
Decode.  median latency: 0.00704 s, median throughput:    142.05 token/s

@hnyls2002
Copy link
Copy Markdown
Collaborator

hnyls2002 commented Sep 14, 2024

w/ torch.compile

python3 -m sglang.bench_latency --model meta-llama/Meta-Llama-3-8B --batch-size 1 --input 128 --output 8 --attention-backend triton --enable-torch-compile
Decode.  median latency: 0.00616 s, median throughput:    162.40 token/s

w/o torch.compile

python3 -m sglang.bench_latency --model meta-llama/Meta-Llama-3-8B --batch-size 1 --input 128 --output 8 --attention-backend triton
Decode.  median latency: 0.00704 s, median throughput:    142.05 token/s

The result is so impressive

@zhyncs zhyncs mentioned this pull request Sep 14, 2024
Closed
5 tasks
ispobock
ispobock reviewed Sep 14, 2024
View reviewed changes
Comment thread python/sglang/srt/layers/attention_backend.py
@merrymercy merrymercy merged commit 9463bc1 into main Sep 14, 2024
@merrymercy merrymercy deleted the triton-torch-compile branch September 14, 2024 22:38
@merrymercy merrymercy restored the triton-torch-compile branch September 15, 2024 03:07
@merrymercy merrymercy deleted the triton-torch-compile branch September 15, 2024 14:28
@merrymercy merrymercy mentioned this pull request Sep 19, 2024
Merged
3 tasks
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
@merrymercy
Enable torch.compile for triton backend (sgl-project#1422)
19e7e83
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ispobock ispobock ispobock left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@merrymercy @hnyls2002 @ispobock