Skip to content

[FEAT] Improved PagedAttention FP8 (faster kvcache dequant v2) #347

Closed
tjtanaa wants to merge 5 commits intoROCm:llama_fp8_12062024from
EmbeddedLLM:paged-attn-updated
Closed

[FEAT] Improved PagedAttention FP8 (faster kvcache dequant v2) #347
tjtanaa wants to merge 5 commits intoROCm:llama_fp8_12062024from
EmbeddedLLM:paged-attn-updated

Conversation

@tjtanaa
Copy link

@tjtanaa tjtanaa commented Dec 27, 2024 •
edited by github-actions bot
Loading

Description

This is a PR to merge https://github.com/ROCm/vllm/blob/shsanyal_develop_cpa_fp8 optimized attention.cu kernel into llama_fp8_12062024 branch.

CAVEAT

Currently the attention.cu kernel does not support block size of 32 and head size of 64.
The vLLM model unittests are failing as it uses small models e.g. Gemma, Llama which has head size of 64.

Performance over this Feature PR (#346) which is another implementation of faster kvcache dequant

The following is a benchmark_throughput results of Llama-3.1-70B with fp8 dynamic quantization and kv-cache-dtype of fp8_e4m3. For sequence input token length 2048 and output token length 2048:

Branch of vll-rocmfork Req/s Total Tokens/s Output Tokens/s
main 0.29 1196.2 598.1
llama-fp8-12062024 0.28 1152.46 576.23
paged-attn-fp8 #346 0.47 1932.74 966.37
this PR 0.62 2537.03 1268.51

@tjtanaa tjtanaa changed the title [FEAT] Improved PagedAttention FP8 (faster kvcache dequant) [FEAT] Improved PagedAttention FP8 (faster kvcache dequant v2) Dec 27, 2024
@tjtanaa tjtanaa marked this pull request as draft December 27, 2024 16:34
@tjtanaa tjtanaa force-pushed the paged-attn-updated branch from 3b816e7 to 73f257d Compare January 2, 2025 08:09
@sanyalington sanyalington mentioned this pull request Jan 21, 2025
Merged
@tjtanaa
Copy link
Author

tjtanaa commented Jan 24, 2025

This PR has been dropped for #385 and #372

@tjtanaa tjtanaa closed this Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Alexei-V-Ivanov-AMD Alexei-V-Ivanov-AMD Awaiting requested review from Alexei-V-Ivanov-AMD Alexei-V-Ivanov-AMD will be requested when the pull request is marked ready for review Alexei-V-Ivanov-AMD is a code owner

@gshtras gshtras Awaiting requested review from gshtras gshtras will be requested when the pull request is marked ready for review gshtras is a code owner

@maleksan85 maleksan85 Awaiting requested review from maleksan85 maleksan85 will be requested when the pull request is marked ready for review maleksan85 is a code owner

@sunway513 sunway513 Awaiting requested review from sunway513 sunway513 will be requested when the pull request is marked ready for review sunway513 is a code owner

@charlifu charlifu Awaiting requested review from charlifu charlifu will be requested when the pull request is marked ready for review charlifu is a code owner

@mawong-amd mawong-amd Awaiting requested review from mawong-amd mawong-amd will be requested when the pull request is marked ready for review mawong-amd is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tjtanaa @vllmellm