Skip to content

Dont overwrite hook handles in flop profiler #2106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
tjruwase merged 4 commits into from
Jul 19, 2022
Merged

Dont overwrite hook handles in flop profiler #2106

tjruwase merged 4 commits into from
Jul 19, 2022

Conversation

@Sanger2000
Copy link
Contributor

@Sanger2000 Sanger2000 commented Jul 18, 2022

This is a solution to #2046.

As explained there:

I ran into the same issue. The problem here is that huggingface instantiates activation function modules like NewGELUActivation at the python global scope. So, when deepspeed recursively registers hooks to model, several forward hooks are added for the same NewGELUActivation object. But, they all overwrite the same removable handle attributes saved to the object. Then, when deepspeed tries to remove these hooks, it can only remove 1, leaving several stale hooks on the module.

@ghost
Copy link

ghost commented Jul 18, 2022 •
edited by ghost
Loading

CLA assistant check
All CLA requirements met.

cli99
cli99 approved these changes Jul 18, 2022
View reviewed changes
Copy link
Contributor

@cli99 cli99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.

@tjruwase tjruwase merged commit 9027f86 into deepspeedai:master Jul 19, 2022
@insujang insujang mentioned this pull request Sep 11, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jeffra jeffra Awaiting requested review from jeffra

@samyam samyam Awaiting requested review from samyam

@tjruwase tjruwase Awaiting requested review from tjruwase

@ShadenSmith ShadenSmith Awaiting requested review from ShadenSmith

@conglongli conglongli Awaiting requested review from conglongli

@awan-10 awan-10 Awaiting requested review from awan-10

@eltonzheng eltonzheng Awaiting requested review from eltonzheng

@minjiaz minjiaz Awaiting requested review from minjiaz

@RezaYazdaniAminabadi RezaYazdaniAminabadi Awaiting requested review from RezaYazdaniAminabadi

@duli2012 duli2012 Awaiting requested review from duli2012

@mrwyattii mrwyattii Awaiting requested review from mrwyattii

@yaozhewei yaozhewei Awaiting requested review from yaozhewei

@arashb arashb Awaiting requested review from arashb

@xiaoxiawu-microsoft xiaoxiawu-microsoft Awaiting requested review from xiaoxiawu-microsoft

@samadejacobs samadejacobs Awaiting requested review from samadejacobs

1 more reviewer

@cli99 cli99 cli99 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Sanger2000 @cli99 @tjruwase