To add SequentialLR to PyTorch Core Schedulers by iramazanli · Pull Request #64037 · pytorch/pytorch

iramazanli · 2021-08-26T17:33:46Z

In this PR we are proposing a new scheduler --SequentialLR-- which enables list of different schedulers called in different periods of the training process.

The main motivation of this scheduler is recently gained popularity of warming up phase in the training time. It has been shown that having a small steps in initial stages of training can help convergence procedure get faster.

With the help of SequentialLR we mainly enable to call a small constant (or linearly increasing) learning rate followed by actual target learning rate scheduler.

scheduler1 = ConstantLR(optimizer, factor=0.1, total_iters=2)
scheduler2 = ExponentialLR(optimizer, gamma=0.9)
scheduler = SequentialLR(optimizer, schedulers=[scheduler1, scheduler2], milestones=[5])

for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()

which this code snippet will call ConstantLR in the first 5 epochs and will follow up with ExponentialLR in the following epochs.

This scheduler could be used to provide call of any group of schedulers next to each other. The main consideration we should make is every time we switch to a new scheduler we assume that new scheduler starts from the beginning- zeroth epoch.

We also add Chained Scheduler to optim.rst and lr_scheduler.pyi files here.

facebook-github-bot · 2021-08-26T17:33:52Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/64037

💊 CI failures summary and remediations

As of commit b42bb36 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

codecov · 2021-09-08T20:32:47Z

Codecov Report

Merging #64037 (c8563bf) into master (9cc44aa) will increase coverage by 4.86%.
The diff coverage is 92.85%.

❗ Current head c8563bf differs from pull request most recent head b42bb36. Consider uploading reports for the commit b42bb36 to get more accurate results

@@            Coverage Diff             @@
##           master   #64037      +/-   ##
==========================================
+ Coverage   61.79%   66.65%   +4.86%     
==========================================
  Files         710      710              
  Lines       92394    92420      +26     
==========================================
+ Hits        57092    61605    +4513     
+ Misses      35302    30815    -4487

datumbox

LGTM. I left some nits on the comments for your consideration but it's up to you.

I only have a question about your example on the description of this PR. Here is what you have at the time of writing:

scheduler1 = ConstantLR(optimizer, factor=0.1, total_iters=2)
scheduler2 = ExponentialLR(optimizer, gamma=0.9)
scheduler = SequentialLR(optimizer, schedulers=[scheduler1, scheduler2], milestones=[5])

for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()

The example has a funny warmup total_iters=2 and milestones=[5] value mismatch (I believe it's just a typo) which raises an interesting question on how this scheduler works. My understanding is that for the first 2 epochs the warmup will use 10% of the LR, for 3 additional epochs it will be 100% of its value and then from 10 to 100 epoch the ExponentialLR scheduler will pick up. Is my understanding correct? If yes, I think that's the expected behaviour if someone misconfigures the scheduler.

datumbox · 2021-09-09T12:53:16Z

test/test_optim.py

Why only 10 epochs here? I understand that the target below takes 19 possible values.

oh thanks for pointing it out, i indeed changed this test several times, epochs = 10 stayed from the previous version.

datumbox · 2021-09-09T12:53:28Z

test/test_optim.py

Same here. Also perhaps a test with more than 2 schedulers?

test added with 3 schedulers :)

fmassa

Thanks for the PR!

Following @datumbox comment and for my own understanding, what happens if we have milestones = 3 but use it with ConstantLR of size 5? Do we restart from the original LR when we switch schedulers or do we pick the smaller LR?

fmassa · 2021-09-09T13:25:40Z

torch/optim/lr_scheduler.py

Should we add a check that len(milestones) == len(schedulers) - 1?

that's a really good idea ! thanks for pointing it out

facebook-github-bot · 2021-09-09T14:22:11Z

@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

iramazanli · 2021-09-09T15:22:02Z

LGTM. I left some nits on the comments for your consideration but it's up to you.

I only have a question about your example on the description of this PR. Here is what you have at the time of writing:
scheduler1 = ConstantLR(optimizer, factor=0.1, total_iters=2)
scheduler2 = ExponentialLR(optimizer, gamma=0.9)
scheduler = SequentialLR(optimizer, schedulers=[scheduler1, scheduler2], milestones=[5])

for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()
The example has a funny warmup total_iters=2 and milestones=[5] value mismatch (I believe it's just a typo) which raises an interesting question on how this scheduler works. My understanding is that for the first 2 epochs the warmup will use 10% of the LR, for 3 additional epochs it will be 100% of its value and then from 10 to 100 epoch the ExponentialLR scheduler will pick up. Is my understanding correct? If yes, I think that's the expected behaviour if someone misconfigures the scheduler.

Your explanation for the expected behavior is correct, and it's not necessary the typo. because theoretically it could be configured that way without any error.

Given that there can be any kind of schedulers added to SequentialLR i preferred to let all the schedulers behave in the default mode. Please let me know if you have a feedback about it.

iramazanli · 2021-09-09T15:23:29Z

Thanks for the PR!

Following @datumbox comment and for my own understanding, what happens if we have milestones = 3 but use it with ConstantLR of size 5? Do we restart from the original LR when we switch schedulers or do we pick the smaller LR?

The way it behaves is as 3 epoch it behave like ConstantLR, then when we move to new scheduler it just behaves like it started from scratch in the 4-th epoch.

facebook-github-bot · 2021-09-09T15:28:05Z

@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-09-09T15:46:44Z

@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-09-09T15:52:39Z

@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-09-09T16:37:59Z

@iramazanli merged this pull request in 2b41bf4.

facebook-github-bot added the cla signed label Aug 26, 2021

iramazanli changed the title ~~To add Composite Scheduler~~ To add SequentialLR to PyTorch Core Schedulers Sep 2, 2021

iramazanli force-pushed the composite_schedulers branch 20 times, most recently from a5532e4 to c1227cb Compare September 8, 2021 18:00

iramazanli requested review from datumbox and fmassa September 8, 2021 18:03

datumbox mentioned this pull request Sep 8, 2021

Update reference scripts to use the "Batteries Included" utils pytorch/vision#4281

Closed

4 tasks

iramazanli force-pushed the composite_schedulers branch from c1227cb to c8563bf Compare September 8, 2021 18:08

datumbox approved these changes Sep 9, 2021

View reviewed changes

datumbox mentioned this pull request Sep 9, 2021

[RFC] TorchVision with Batteries included - Phase 1 pytorch/vision#3911

Closed

16 tasks

fmassa reviewed Sep 9, 2021

View reviewed changes

iramazanli force-pushed the composite_schedulers branch from c8563bf to 223e2e3 Compare September 9, 2021 15:12

iramazanli force-pushed the composite_schedulers branch from 223e2e3 to 43e72d5 Compare September 9, 2021 15:24

To add SequentiaLR Scheduler

b42bb36

iramazanli force-pushed the composite_schedulers branch from 43e72d5 to b42bb36 Compare September 9, 2021 15:45

facebook-github-bot closed this in 2b41bf4 Sep 9, 2021

facebook-github-bot added the Merged label Sep 9, 2021

Conversation

iramazanli commented Aug 26, 2021 • edited by datumbox Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Aug 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

Uh oh!

codecov bot commented Sep 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datumbox left a comment

Choose a reason for hiding this comment

Uh oh!

datumbox Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

iramazanli Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

datumbox Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

iramazanli Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

fmassa left a comment

Choose a reason for hiding this comment

Uh oh!

fmassa Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

iramazanli Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

iramazanli commented Sep 9, 2021

Uh oh!

iramazanli commented Sep 9, 2021

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iramazanli commented Aug 26, 2021 •

edited by datumbox

Loading

facebook-github-bot commented Aug 26, 2021 •

edited

Loading

codecov bot commented Sep 8, 2021 •

edited

Loading