Skip to content

Disable {save,check}_on_train_epoch_end with check_val_every_n_epoch>1#9156

Merged
carmocca merged 4 commits intomasterfrom
bugfix/check-val-every-n-on-train-epoch-end
Sep 3, 2021
Merged

Disable {save,check}_on_train_epoch_end with check_val_every_n_epoch>1#9156
carmocca merged 4 commits intomasterfrom
bugfix/check-val-every-n-on-train-epoch-end

Conversation

@carmocca
Copy link
Copy Markdown
Contributor

@carmocca carmocca commented Aug 27, 2021 •
edited
Loading

What does this PR do?

Fixes #9151, #9163

Does your PR introduce any breaking changes? If yes, please list them.

None

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • [n/a] Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

@carmocca carmocca added the bug Something isn't working label Aug 27, 2021
@carmocca carmocca added this to the v1.4.x milestone Aug 27, 2021
@carmocca carmocca self-assigned this Aug 27, 2021
@carmocca carmocca changed the title Disable {save,check}_on_train_epoch_end with check_val_every_n_epoch Disable {save,check}_on_train_epoch_end with check_val_every_n_epoch>1 Aug 27, 2021
ananthsub
ananthsub reviewed Aug 27, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@ananthsub ananthsub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carmocca do you think it'd be better to have the user explicitly opt into this without providing a default?

@codecov
Copy link
Copy Markdown

codecov Bot commented Aug 27, 2021 •
edited
Loading

Codecov Report

Merging #9156 (23dbc5a) into master (b497fb8) will increase coverage by 0%.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #9156   +/-   ##
======================================
  Coverage      92%     92%           
======================================
  Files         176     178    +2     
  Lines       14804   14850   +46     
======================================
+ Hits        13651   13701   +50     
+ Misses       1153    1149    -4

@carmocca
Copy link
Copy Markdown
Contributor Author

carmocca commented Aug 27, 2021

No, the problem is that users don't notice something is wrong until the epoch is over and a callback tries to get a non-logged metric key.

We could print a warning before the epoch starts asking to set this flag when trainer.val_check_interval != 1.0 or trainer.check_val_every_n_epoch != 1, but then they'd ask: "Why don't you set it automatically?" which is what we do here.

ananthsub
ananthsub reviewed Aug 27, 2021
View reviewed changes
Comment thread pytorch_lightning/callbacks/early_stopping.py
tchaton
tchaton reviewed Aug 27, 2021
View reviewed changes
Comment thread pytorch_lightning/callbacks/early_stopping.py
@carmocca carmocca requested review from ananthsub and tchaton August 27, 2021 12:18
tchaton
tchaton approved these changes Aug 31, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

Comment thread pytorch_lightning/callbacks/early_stopping.py
@mergify mergify Bot added the ready to be merged PRs ready to be merged label Aug 31, 2021
@tchaton tchaton enabled auto-merge (squash) September 1, 2021 18:30
@carmocca carmocca disabled auto-merge September 3, 2021 13:50
@carmocca carmocca enabled auto-merge (squash) September 3, 2021 13:58
@carmocca carmocca merged commit d5ee8d8 into master Sep 3, 2021
@carmocca carmocca deleted the bugfix/check-val-every-n-on-train-epoch-end branch September 3, 2021 14:27
justusschock added a commit that referenced this pull request Sep 7, 2021
@justusschock
Disable {save,check}_on_train_epoch_end with `check_val_every_n_epo…
0630b3d 
…ch>1` (#9156)
justusschock added a commit that referenced this pull request Sep 7, 2021
@justusschock
Disable {save,check}_on_train_epoch_end with `check_val_every_n_epo…
0bba728 
…ch>1` (#9156)
justusschock added a commit that referenced this pull request Sep 7, 2021
@justusschock
Disable {save,check}_on_train_epoch_end with `check_val_every_n_epo…
579ddce 
…ch>1` (#9156)
awaelchli pushed a commit that referenced this pull request Sep 7, 2021
@justusschock @awaelchli
Disable {save,check}_on_train_epoch_end with `check_val_every_n_epo…
67b1a97 
…ch>1` (#9156)
lexierule pushed a commit that referenced this pull request Sep 10, 2021
@justusschock @lexierule
Disable {save,check}_on_train_epoch_end with `check_val_every_n_epo…
f3e24f0 
…ch>1` (#9156)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tchaton tchaton tchaton approved these changes

@Borda Borda Awaiting requested review from Borda

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@kaushikb11 kaushikb11 Awaiting requested review from kaushikb11

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

@ananthsub ananthsub Awaiting requested review from ananthsub

+2 more reviewers

@awaelchli awaelchli awaelchli approved these changes

@SeanNaren SeanNaren SeanNaren approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@carmocca carmocca

Labels

bug Something isn't working callback ready to be merged PRs ready to be merged

Projects

None yet

Milestone

pl:1.4.x

Development

Successfully merging this pull request may close these issues.

EarlyStopping is based upon callback.on_train_epoch_end, not callback.on_validation_epoch_end

5 participants

@carmocca @ananthsub @awaelchli @SeanNaren @tchaton