Add Trainer.validate(…) method to run one validation epoch by EliaCereda · Pull Request #4948 · Lightning-AI/pytorch-lightning

EliaCereda · 2020-12-02T15:57:08Z

What does this PR do?

author: @carmocca, original author: @EliaCereda

Actually adds the new Trainer.validate(…) method, which performs one evaluation epoch over the validation set.
Adds tests for the new feature
Lots of test cleanups

Fixes #4634
Addresses #4170, although a full solution for that case would be to support running validation before epoch 0 directly inside Trainer.fit(...).
Split from PR #4707, see for full discussion

Before submitting

Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?
Did you verify new and existing tests pass locally with your changes?
If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR

* Replace the `Trainer.testing` attribute with `Trainer.evaluating`, which is currently set to `'test'` if the top-level function called by the user was `Trainer.test(…)` and `None` otherwise. In the next PR, it will be set to `'validation’` when the user calls `validate(…)`. * Update the other components to use the new attribute instead of `Trainer.testing` * Disable the `EarlyStopping` and `ModelCheckpoint` callbacks when `evaluating`. This has no effect when evaluating on the test set, since they were already disabled, but it will be necessary for the validation set * Rename a few other attributes of `Trainer` to clarify that they will be used by both `test(…)` and `validate(…)`

pep8speaks · 2020-12-02T15:57:29Z

Hello @EliaCereda! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-03-10 18:20:57 UTC

…alidate-1

… the validation set

…ed by both test(…) and validate(…)

… hook and callback calls

codecov · 2020-12-02T17:03:30Z

Codecov Report

Merging #4948 (8fab50f) into master (615b2f7) will decrease coverage by 2%.
The diff coverage is 96%.

@@           Coverage Diff           @@
##           master   #4948    +/-   ##
=======================================
- Coverage      94%     92%    -2%     
=======================================
  Files         161     161            
  Lines       11495   11549    +54     
=======================================
- Hits        10751   10588   -163     
- Misses        744     961   +217

tchaton

Great job there ! Still some minors comments to address.

pytorch_lightning/trainer/trainer.py

tests/trainer/test_trainer_validate_loop.py

…the last in the codebase

…alidate-1

pytorch_lightning/trainer/configuration_validator.py

tchaton

LGTM !

tests/accelerators/test_common.py

pytorch_lightning/trainer/trainer.py

SeanNaren

Been lurking on these PRs for a while, nice work guys :)

awaelchli

nice addition
high level review

awaelchli · 2021-03-10T17:39:35Z

tests/plugins/test_sharded_plugin.py

+@pytest.mark.parametrize("trainer_kwargs", (
+    {'num_processes': 2},
+    pytest.param({'gpus': 2}, marks=RunIf(min_gpus=2))
+))
+def test_ddp_sharded_plugin_test_multigpu(tmpdir, trainer_kwargs):


do special tests run with parameterization? I thought that was a limitation.

@Borda or @tchaton do you know?

I could be wrong, I see actually other tests parameterized like this. There was just something in the back of my head

I think yes, but you can always check what test ran in Azure as special tests are GPU based...

Well, looks like this is not in the special_test.sh file so can't check if it does work because it's not being run in CI

tests/core/test_datamodules.py

pytorch_lightning/trainer/trainer.py

pytorch_lightning/trainer/states.py

pytorch_lightning/trainer/trainer.py

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

…ter) to github/third-party/PyTorchLightning/pytorch-lightning Summary: ### New commit log messages ## [UnReleased] - 2021-MM-DD ### Added - Added more explicit exception message when trying to execute `trainer.test()` or `trainer.validate()` with `fast_dev_run=True` ([#6667](Lightning-AI/pytorch-lightning#6667)) - Added `LightningCLI` class to provide simple reproducibility with minimum boilerplate training cli. ([#4492](Lightning-AI/pytorch-lightning#4492)) - Trigger warning when non-metric logged value with multi processes hasn't been reduced ([#6417](Lightning-AI/pytorch-lightning#6417)) - Added `gradient_clip_algorithm` argument to Trainer for gradient clipping by value ([#6123](Lightning-AI/pytorch-lightning#6123)). - Added a way to print to terminal without breaking up the progress bar ([#5470](Lightning-AI/pytorch-lightning#5470)) - Added support to checkpoint after training steps in `ModelCheckpoint` callback ([#6146](Lightning-AI/pytorch-lightning#6146)) - Added `checkpoint` parameter to callback's `on_save_checkpoint` hook ([#6072](Lightning-AI/pytorch-lightning#6072)) - Added `RunningStage.SANITY_CHECKING` ([#4945](Lightning-AI/pytorch-lightning#4945)) - Added `TrainerState.{FITTING,VALIDATING,TESTING,PREDICTING,TUNING}` ([#4945](Lightning-AI/pytorch-lightning#4945)) - Added `Trainer.validate()` method to perform one evaluation epoch over the validation set ([#4948](Lightning-AI/pytorch-lightning#4948)) - Added `LightningEnvironment` for Lightning-specific DDP ([#5915](Lightning-AI/pytorch-lightning#5915)) - Added `teardown()` hook to LightningDataModule ([#4673](Lightning-AI/pytorch-lightning#4673)) - Added `auto_insert_metric_name` parameter to `ModelCheckpoint` ([#6277](Lightning-AI/pytorch-lightning#6277)) - Added arg to `self.log` that enables users to give custom names when dealing with multiple dataloaders ([#6274](Lightning-AI/pytorch-lightning#6274)) - Added `teardown` method to `BaseProfiler` to enable subclasses defining post-profiling steps outside of `__del__` ([#6370](Lightning-AI/pytorch-lightning#6370)) - Added `setup` method to `BaseProfiler` to enable subclasses defining pre-profiling steps for every process ([#6633](Lightning-AI/pytorch-lightning#6633)) - Added no return warning to predict ([#6139](Lightning-AI/pytorch-lightning#6139)) - Added `Trainer.predict` config validation ([#6543](Lightning-AI/pytorch-lightning#6543)) - Added `AbstractProfiler` interface ([#6621](Lightning-AI/pytorch-lightning#6621)) - Added support for including module names for forward in the autograd trace of `PyTorchProfiler` ([#6349](Lightning-AI/pytorch-lightning#6349)) - Added support for the PyTorch 1.8.1 autograd profiler ([#6618](Lightning-AI/pytorch-lightning#6618)) - Added `outputs` parameter to callback's `on_validation_epoch_end` & `on_test_epoch_end` hooks ([#6120](Lightning-AI/pytorch-lightning#6120)) - Added `configure_sharded_model` hook ([#6679](Lightning-AI/pytorch-lightning#6679)) - Added support for `precision=64`, enabling training with double precision ([#6595](Lightning-AI/pytorch-lightning#6595)) - Added support for DDP communication hooks ([#6736](Lightning-AI/pytorch-lightning#6736)) - Added `artifact_location` argument to `MLFlowLogger` which will be passed to the `MlflowClient.create_experiment` call ([#6677](Lightning-AI/pytorch-lightning#6677)) - Added `model` parameter to precision plugins' `clip_gradients` signature ([#6764](Lightning-AI/pytorch-lightning#6764)) ### Changed - Renamed `pytorch_lightning.callbacks.swa` to `pytorch_lightning.callbacks.stochastic_weight_avg` ([#6259](Lightning-AI/pytorch-lightning#6259)) - Refactor `RunningStage` and `TrainerState` usage ([#4945](Lightning-AI/pytorch-lightning#4945)) - Changed `trainer.evaluating` to return `True` if validating or testing ([#4945](Lightning-AI/pytorch-lightning#4945)) - Changed `setup()` and `teardown()` stage argument to take any of `{fit,validate,test,predict}` ([#6386](Lightning-AI/pytorch-lightning#6386)) - Changed profilers to save separate report files per state and rank ([#6621](Lightning-AI/pytorch-lightning#6621)) - Changed `PyTorchProfiler` to use `torch.autograd.profiler.record_function` to record functions ([#6349](Lightning-AI/pytorch-lightning#6349)) ### Deprecated - `period` has been deprecated in favor of `every_n_val_epochs` in the `ModelCheckpoint` callback ([#6146](Lightning-AI/pytorch-lightning#6146)) - Deprecated `trainer.running_sanity_check` in favor of `trainer.sanity_checking` ([#4945](Lightning-AI/pytorch-lightning#4945)) - Deprecated `Profiler(output_filename)` in favor of `dirpath` and `filename` ([#6621](Lightning-AI/pytorch-lightning#6621)) - Deprecated `PytorchProfiler(profiled_functions)` in favor of `record_functions` ([#6349](Lightning-AI/pytorch-lightning#6349)) - Deprecated metrics in favor of `torchmetrics` ([#6505](Lightning-AI/pytorch-lightning#6505), [#6530](Lightning-AI/pytorch-lightning#6530), [#6540](Lightning-AI/pytorch-lightning#6540), [#6547](Lightning-AI/pytorch-lightning#6547), [#6515](Lightning-AI/pytorch-lightning#6515), [#6572](Lightning-AI/pytorch-lightning#6572), [#6573](Lightning-AI/pytorch-lightning#6573), [#6584](Lightning-AI/pytorch-lightning#6584), [#6636](Lightning-AI/pytorch-lightning#6636), [#6637](Lightning-AI/pytorch-lightning#6637), [#6649](Lightning-AI/pytorch-lightning#6649), [#6659](Lightning-AI/pytorch-lightning#6659), ) ### Removed - Removed support for passing a bool value to `profiler` argument of Trainer ([#6164](Lightning-AI/pytorch-lightning#6164)) - Removed no return warning from val/test step ([#6139](Lightning-AI/pytorch-lightning#6139)) - Removed passing a `ModelCheckpoint` instance to `Trainer(checkpoint_callback)` ([#6166](Lightning-AI/pytorch-lightning#6166)) - Removed deprecated Trainer argument `enable_pl_optimizer` and `automatic_optimization` ([#6163](Lightning-AI/pytorch-lightning#6163)) - Removed deprecated metrics ([#6161](Lightning-AI/pytorch-lightning#6161)) * from `pytorch_lightning.metrics.functional.classification` removed `to_onehot`, `to_categorical`, `get_num_classes`, `roc`, `multiclass_roc`, `average_precision`, `precision_recall_curve`, `multiclass_precision_recall_curve` * from `pytorch_lightning.metrics.functional.reduction` removed `reduce`, `class_reduce` - Removed deprecated `ModelCheckpoint` arguments `prefix`, `mode="auto"` ([#6162](Lightning-AI/pytorch-lightning#6162)) - Removed `mode='auto'` from `EarlyStopping` ([#6167](Lightning-AI/pytorch-lightning#6167)) - Removed legacy references for magic keys in the `Result` object ([#6016](Lightning-AI/pytorch-lightning#6016)) - Removed deprecated `LightningModule` `hparams` setter ([#6207](Lightning-AI/pytorch-lightning#6207)) - Removed legacy code to log or include metrics in the progress bar by returning them in a dict with the `"log"/"progress_bar"` magic keys. Use `self.log` instead ([#6734](Lightning-AI/pytorch-lightning#6734)) - Removed `optimizer_idx` argument from `training_step` in manual optimization ([#6093](Lightning-AI/pytorch-lightning#6093)) ### Fixed - Set better defaults for `rank_zero_only.rank` when training is launched with SLURM and torchelastic ([#6802](Lightning-AI/pytorch-lightning#6802)) - Made the `Plugin.reduce` method more consistent across all Plugins to reflect a mean-reduction by default ([#6011](Lightning-AI/pytorch-lightning#6011)) - Move lightning module to correct device type when using LightningDistributedWrapper ([#6070](Lightning-AI/pytorch-lightning#6070)) - Do not print top-k verbose log with `ModelCheckpoint(monitor=None)` ([#6109](Lightning-AI/pytorch-lightning#6109)) - Fixed csv extension check ([#6436](Lightning-AI/pytorch-lightning#6436)) - Fixed `ModelCheckpoint(monitor=None, save_last=True)` not saving checkpoints ([#6136](Lightning-AI/pytorch-lightning#6136)) - Fixed `ModelCheckpoint(save_top_k=0, save_last=True)` not saving the `last` checkpoint ([#6136](Lightning-AI/pytorch-lightning#6136)) - Fixed `.teardown(stage='fit')` getting called during `trainer.test` ([#6386](Lightning-AI/pytorch-lightning#6386)) - Fixed `.on_fit_{start,end}()` getting called during `trainer.test` ([#6386](Lightning-AI/pytorch-lightning#6386)) - Fixed LightningModule `all_gather` on cpu tensors ([#6416](Lightning-AI/pytorch-lightning#6416)) - Fixed torch distributed not available in setup hook for DDP ([#6506](Lightning-AI/pytorch-lightning#6506)) - Fixed `EarlyStopping` logic when `min_epochs` or `min_steps` requirement is not met ([#6705](Lightning-AI/pytorch-lightning#6705)) ## [1.2.7] - 2021-04-06 ### Fixed - Fixed resolve a bug with omegaconf and xm.save ([#6741](Lightning-AI/pytorch-lightning#6741)) - Fixed an issue with IterableDataset when __len__ is not defined ([#6828](Lightning-AI/pytorch-lightning#6828)) - Sanitize None params during pruning ([#6836](Lightning-AI/pytorch-lightning#6836)) - Enforce an epoch scheduler interval when using SWA ([#6588](Lightning-AI/pytorch-lightning#6588)) - Fixed TPU Colab hang issue, post training ([#6816](Lightning-AI/pytorch-lightning#6816)) - Fixed a bug where `TensorBoardLogger` would give a warning and not log correctly to a symbolic link `save_dir` ([#6730](Lightning-AI/pytorch-lightning#6730)) ## [1.2.6] - 2021-03-30 ### Changed - Changed the behavior of `on_epoch_start` to run at the beginning of validation & test epoch ([#6498](Lightning-AI/pytorch-lightning#6498)) ### Removed - Removed legacy code to include `step` dictionary returns in `callback_metrics`. Use `self.log_dict` instead. ([#6682](Lightning-AI/pytorch-lightning#6682)) ### Fixed - Fixed `DummyLogger.log_hyperparams` raising a `TypeError` when running with `fast_dev_run=True` ([#6398](Lightning-AI/pytorch-lightning#6398)) - Fixed error on TPUs when there was no `ModelCheckpoint` ([#6654](Lightning-AI/pytorch-lightning#6654)) - Fixed `trainer.test` freeze on TPUs ([#6654](Lightning-AI/pytorch-lightning#6654)) - Fixed a bug where gradients were disabled after calling `Trainer.predict` ([#6657](Lightning-AI/pytorch-lightning#6657)) - Fixed bug where no TPUs were detected in a TPU pod env ([#6719](Lightning-AI/pytorch-lightning#6719)) ## [1.2.5] - 2021-03-23 ### Changed - Update Gradient Clipping for the TPU Accelerator ([#6576](Lightning-AI/pytorch-lightning#6576)) - Refactored setup for typing friendly ([#6590](Lightning-AI/pytorch-lightning#6590)) ### Fixed - Fixed a bug where `all_gather` would not work correctly with `tpu_cores=8` ([#6587](Lightning-AI/pytorch-lightning#6587)) - Fixed comparing required versions ([#6434](Lightning-AI/pytorch-lightning#6434)) - Fixed duplicate logs appearing in console when using the python logging module ([#6275](Lightning-AI/pytorch-lightning#6275)) - Added Autocast in validation, test and predict modes for Native AMP ([#6565](Lightning-AI/pytorch-lightning#6565)) Reviewed By: shuyingsunshine21 Differential Revision: D27528929 fbshipit-source-id: 311c88f71461c2c79bbf185e28d7a6d683ccc26f

EliaCereda mentioned this pull request Dec 2, 2020

Refactor RunningStage usage in advance of implementing Trainer.validate() #4945

Merged

9 tasks

EliaCereda added 6 commits December 2, 2020 16:59

Merge remote-tracking branch 'upstream/master' into feature/trainer-v…

03d7994

…alidate-1

Add Trainer.validate(...) method to perform one evaluation epoch over…

5a54485

… the validation set

Rename methods in Trainer and Accelerator to reflect that they are us…

e06775c

…ed by both test(…) and validate(…)

Update docs to mention the new Trainer.validate method and associated…

b4e409c

… hook and callback calls

Add tests for Trainer.validate(…)

96e42ba

Update CHANGELOG.md

85b3c9f

EliaCereda force-pushed the feature/trainer-validate-2 branch from 3fa7551 to 85b3c9f Compare December 2, 2020 16:02

EliaCereda mentioned this pull request Dec 2, 2020

Add Trainer.validate(…) method to run one validation epoch #4707

Closed

11 tasks

EliaCereda marked this pull request as ready for review December 2, 2020 16:05

EliaCereda requested review from Borda, SeanNaren, ananyahjha93, awaelchli, justusschock, nateraw, tchaton, teddykoker and williamFalcon as code owners December 2, 2020 16:05

Merge branch 'master' into feature/trainer-validate-2

39113dc

tchaton reviewed Dec 3, 2020

View reviewed changes

EliaCereda added 2 commits December 4, 2020 19:30

Replace usages of Trainer.testing with Trainer.evaluating, should be …

a6be0d8

…the last in the codebase

Merge remote-tracking branch 'upstream/master' into feature/trainer-v…

a922d57

…alidate-1

Borda added this to the 1.2 milestone Dec 4, 2020

Borda added feature Is an improvement or enhancement refactor labels Dec 4, 2020

Clean up calls to LightningDataModule.setup()

595f4e8

Refactor

e582d58

carmocca closed this Mar 8, 2021

carmocca deleted the branch Lightning-AI:master March 8, 2021 14:27

mergify bot added the has conflicts label Mar 8, 2021

carmocca reopened this Mar 8, 2021

carmocca changed the base branch from setup-more-stages to master March 8, 2021 14:33

Merge branch 'master' into feature/trainer-validate-2

1a5b620

mergify bot removed the has conflicts label Mar 8, 2021

carmocca changed the title ~~[Blocked by #6386] Add Trainer.validate(…) method to run one validation epoch~~ Add Trainer.validate(…) method to run one validation epoch Mar 8, 2021

carmocca added 2 commits March 8, 2021 15:39

flake8

5b99ec0

Refactor

9f4dce2

carmocca reviewed Mar 8, 2021

View reviewed changes

pytorch_lightning/trainer/configuration_validator.py Show resolved Hide resolved

carmocca added 3 commits March 8, 2021 16:33

Missing import

088d4bc

Fix test

58fcca4

Same threshold

babb73d

mergify bot added the has conflicts label Mar 8, 2021

tchaton approved these changes Mar 8, 2021

View reviewed changes

tests/accelerators/test_common.py Show resolved Hide resolved

pytorch_lightning/trainer/trainer.py Outdated Show resolved Hide resolved

pytorch_lightning/trainer/trainer.py Show resolved Hide resolved

pytorch_lightning/trainer/trainer.py Show resolved Hide resolved

carmocca added 2 commits March 9, 2021 00:29

Address tchaton's comments

235dc27

Merge branch 'master' into feature/trainer-validate-2

73dd265

mergify bot removed the has conflicts label Mar 8, 2021

carmocca added 2 commits March 10, 2021 02:57

Missing import

e423b98

Merge branch 'master' into feature/trainer-validate-2

cdec83b

carmocca added the ready to be merged PRs ready to be merged label Mar 10, 2021

SeanNaren approved these changes Mar 10, 2021

View reviewed changes

awaelchli approved these changes Mar 10, 2021

View reviewed changes

Apply suggestions from code review

8fab50f

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

carmocca merged commit f4cc745 into Lightning-AI:master Mar 11, 2021

carmocca mentioned this pull request Mar 15, 2021

1.2.x cherries 🍒 #6083

Closed

sahandrez mentioned this pull request Jul 17, 2025

Doing full validation on step 0 #20985

Open

Conversation

EliaCereda commented Dec 2, 2020 • edited by carmocca Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

PR review

Uh oh!

pep8speaks commented Dec 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2021-03-10 18:20:57 UTC

Uh oh!

codecov bot commented Dec 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tchaton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tchaton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SeanNaren left a comment

Choose a reason for hiding this comment

Uh oh!

awaelchli left a comment

Choose a reason for hiding this comment

Uh oh!

awaelchli Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

carmocca Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

awaelchli Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

Borda Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

carmocca Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

EliaCereda commented Dec 2, 2020 •

edited by carmocca

Loading

pep8speaks commented Dec 2, 2020 •

edited

Loading

codecov bot commented Dec 2, 2020 •

edited

Loading