Skip to content

Optimizer state scaling#44

Merged
andersonic merged 16 commits intomasterfrom
optimizer-state-scaling
Aug 22, 2020
Merged

Optimizer state scaling#44
andersonic merged 16 commits intomasterfrom
optimizer-state-scaling

Conversation

@andersonic
Copy link
Copy Markdown
Contributor

@andersonic andersonic commented Aug 17, 2020

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • Did you read the contributor guideline?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Allows scaling of optimizer state.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 17, 2020
@codecov
Copy link
Copy Markdown

codecov bot commented Aug 17, 2020 •
edited
Loading

Codecov Report

Merging #44 into master will increase coverage by 0.09%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #44      +/-   ##
==========================================
+ Coverage   94.18%   94.28%   +0.09%     
==========================================
  Files          35       35              
  Lines        2065     2100      +35     
==========================================
+ Hits         1945     1980      +35     
  Misses        120      120
Flag Coverage Δ
#Python 94.28% <100.00%> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
fairscale/optim/adam.py 94.66% <100.00%> (+1.62%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e8c2a9...d926510. Read the comment docs.

@andersonic andersonic changed the base branch from add-fp16-optimizer-state to master August 19, 2020 00:01
@andersonic andersonic force-pushed the optimizer-state-scaling branch 2 times, most recently from 744713f to 0eb5ad3 Compare August 19, 2020 22:33
@andersonic andersonic marked this pull request as ready for review August 19, 2020 22:49
@andersonic andersonic marked this pull request as draft August 19, 2020 22:51
@andersonic andersonic marked this pull request as ready for review August 20, 2020 16:21
@andersonic andersonic force-pushed the optimizer-state-scaling branch from 8295557 to fce8747 Compare August 21, 2020 17:30
sidgoyal78
sidgoyal78 reviewed Aug 21, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@sidgoyal78 sidgoyal78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. I have a few inline comments/questions.

@andersonic andersonic requested a review from sidgoyal78 August 21, 2020 22:41
sidgoyal78
sidgoyal78 approved these changes Aug 22, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@sidgoyal78 sidgoyal78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes. Looks good to me!

shruti-bh
shruti-bh approved these changes Aug 22, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@shruti-bh shruti-bh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall!

benchmarks/transformer.py
# Assert that memory usage on each GPU is within 10% of golden run
# Right-hand-side is golden run bytes * 110%
assert torch.cuda.memory_stats(0)["allocated_bytes.all.peak"] < 210479616 * 1.1
assert torch.cuda.memory_stats(0)["allocated_bytes.all.peak"] < 193206272 * 1.1
Copy link
Copy Markdown
Contributor

@shruti-bh shruti-bh Aug 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how did you come up with these number of bytes?

Copy link
Copy Markdown
Contributor

@sidgoyal78 sidgoyal78 Aug 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the same question in one of the previous PRs :D
I guess Jun-Ru printed out the value of torch.cuda.memory_stats(0)["allocated_bytes.all.peak"] and put that number in the check!

@andersonic andersonic merged commit 5251a69 into master Aug 22, 2020
@andersonic andersonic deleted the optimizer-state-scaling branch August 22, 2020 00:14
myleott pushed a commit that referenced this pull request Feb 22, 2021
@sshleifer
move_grads_to_cpu defaults to same value as cpu_offload (#44)
bc7e337
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@msbaines msbaines Awaiting requested review from msbaines

@blefaudeux blefaudeux Awaiting requested review from blefaudeux

2 more reviewers

@sidgoyal78 sidgoyal78 sidgoyal78 approved these changes

@shruti-bh shruti-bh shruti-bh approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@andersonic @sidgoyal78 @shruti-bh @facebook-github-bot