Warmup schedulers in References#4411
Merged
datumbox merged 9 commits intopytorch:mainfrom Sep 17, 2021
Merged
Conversation
16 tasks
fmassa
approved these changes
Sep 17, 2021
Member
fmassa
left a comment
There was a problem hiding this comment.
LGTM, thanks!
Just double-checking, did you run the new schedulers in a loop to compare before / after results?
| main_lr_scheduler = torch.optim.lr_scheduler.LambdaLR( | ||
| optimizer, | ||
| lambda x: (1 - x / (len(data_loader) * args.epochs)) ** 0.9) | ||
| lambda x: (1 - x / (iters_per_epoch * (args.epochs - args.lr_warmup_epochs))) ** 0.9) |
Member
There was a problem hiding this comment.
Created an issue to track if we can now use a stock PyTorch scheduler for this #4438
Contributor
Author
|
@fmassa Thanks! Yes I did. They match really close (to the 5th decimal) most of the times. For reference here are the scripts used to test this: |
facebook-github-bot
pushed a commit
that referenced
this pull request
Sep 30, 2021
Summary: * Warmup on Classficiation references. * Adjust epochs for cosine. * Warmup on Segmentation references. * Warmup on Video classification references. * Adding support of both types of warmup in segmentation. * Use LinearLR in detection. * Fix deprecation warning. Reviewed By: datumbox Differential Revision: D31268039 fbshipit-source-id: d0fe7e334c01201c2413bac8b911d740b9a69bba
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #4281
Adds warmup on the following recipes:
This PR maintains the location where we call
scheduler.step()in each recipe. Segmentation and Video classification do it on the iteration level, Classification does it on the epoch level and Detection does it in a hybrid.Though doing it on the iteration level provides more flexibility, doing so will have slight effects on the reproducibility of existing models. These effects should be minor and largely overshadowed by other differences across runs (such as the randomness of the initialization scheme). The only reason I'm not doing the switch here is because it requires extra work which I'm deferring for when we will start retraining the models using the new utils of Batteries Included.