Skip to content

Fix issue with corrupted output on long generation for GPT #2344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
andrewchernyh wants to merge 2 commits into from
Closed

Fix issue with corrupted output on long generation for GPT #2344

andrewchernyh wants to merge 2 commits into from

Conversation

@andrewchernyh
Copy link
Contributor

@andrewchernyh andrewchernyh commented Sep 22, 2022

Fix issue described in #2300

Only updating MAX_OUT_TOKES doesn't helps and main issue is in temp_buf that was allocated just after output tensor

Then in attention_unfused
T* workspace = (T*)output + bsz * seq_len * heads * k; where output is temp_buf from ds_softmax_context
In the result temp_buf overlaps query_cont allocated in ds_softmax_context

I've put temp_buf just after kv_cache and it helps

Minimal steps to reproduce:

python3 benchmarks/inference/gpt-bench.py -m EleutherAI/gpt-neo-125M --kernel-inject --deepspeed --dtype=fp32 --max-tokens=1020 --trials 1

@ghost
Copy link

ghost commented Sep 22, 2022 •
edited by ghost
Loading

CLA assistant check
All CLA requirements met.

@andrewchernyh andrewchernyh mentioned this pull request Sep 22, 2022
Closed
@andrewchernyh
Copy link
Contributor Author

andrewchernyh commented Sep 23, 2022

Not actual more, see #2300 (comment)

@andrewchernyh andrewchernyh mentioned this pull request Sep 26, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jeffra jeffra Awaiting requested review from jeffra

@samyam samyam Awaiting requested review from samyam

@tjruwase tjruwase Awaiting requested review from tjruwase tjruwase is a code owner

@ShadenSmith ShadenSmith Awaiting requested review from ShadenSmith

@conglongli conglongli Awaiting requested review from conglongli

@awan-10 awan-10 Awaiting requested review from awan-10

@cli99 cli99 Awaiting requested review from cli99

@eltonzheng eltonzheng Awaiting requested review from eltonzheng

@minjiaz minjiaz Awaiting requested review from minjiaz

@RezaYazdaniAminabadi RezaYazdaniAminabadi Awaiting requested review from RezaYazdaniAminabadi

@duli2012 duli2012 Awaiting requested review from duli2012

@mrwyattii mrwyattii Awaiting requested review from mrwyattii

@yaozhewei yaozhewei Awaiting requested review from yaozhewei

@arashb arashb Awaiting requested review from arashb

@xiaoxiawu-microsoft xiaoxiawu-microsoft Awaiting requested review from xiaoxiawu-microsoft

@samadejacobs samadejacobs Awaiting requested review from samadejacobs

@cmikeh2 cmikeh2 Awaiting requested review from cmikeh2

@GuanhuaWang GuanhuaWang Awaiting requested review from GuanhuaWang

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andrewchernyh