Skip to content

Tuner cleanup on error#21162

Merged
Borda merged 6 commits intoLightning-AI:masterfrom
SkafteNicki:tuner_cleanup_on_error
Sep 4, 2025
Merged

Tuner cleanup on error#21162
Borda merged 6 commits intoLightning-AI:masterfrom
SkafteNicki:tuner_cleanup_on_error

Conversation

@SkafteNicki
Copy link
Copy Markdown
Collaborator

@SkafteNicki SkafteNicki commented Sep 4, 2025 •
edited by github-actions Bot
Loading

What does this PR do?

Fixes #13856
Tuner algorithms will currently leave behind temp checkpoints if the tuning for some reason fails. PR makes sure these are cleaned up using try, except, finally statements even if tuning crashes.

Before submitting
  • Was this discussed/agreed via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist
  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

📚 Documentation preview 📚: https://pytorch-lightning--21162.org.readthedocs.build/en/21162/

@SkafteNicki SkafteNicki self-assigned this Sep 4, 2025
@SkafteNicki SkafteNicki added bug Something isn't working tuner labels Sep 4, 2025
@github-actions github-actions Bot added the pl Generic label for PyTorch Lightning package label Sep 4, 2025
Borda
Borda reviewed Sep 4, 2025
View reviewed changes
Comment thread src/lightning/pytorch/tuner/batch_size_scaling.py Outdated
Borda
Borda reviewed Sep 4, 2025
View reviewed changes
Comment thread src/lightning/pytorch/tuner/lr_finder.py Outdated
SkafteNicki and others added 2 commits September 4, 2025 11:38
@SkafteNicki @Borda
Update src/lightning/pytorch/tuner/batch_size_scaling.py
663e736 
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
@SkafteNicki @Borda
Update src/lightning/pytorch/tuner/lr_finder.py
7837037 
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
@Borda Borda merged commit e1e2534 into Lightning-AI:master Sep 4, 2025
96 of 98 checks passed
Borda pushed a commit that referenced this pull request Sep 4, 2025
@SkafteNicki @Borda
Tuner cleanup on error (#21162)
2e5d022 
* Make sure temp checkpoints are cleaned up on failed tuning
* add testing
* changelog

---------

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

(cherry picked from commit e1e2534)
lantiga pushed a commit that referenced this pull request Sep 5, 2025
@SkafteNicki @lantiga
Tuner cleanup on error (#21162)
17d2d78 
* Make sure temp checkpoints are cleaned up on failed tuning
* add testing
* changelog

---------

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

(cherry picked from commit e1e2534)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Borda Borda Borda approved these changes

@lantiga lantiga Awaiting requested review from lantiga

@tchaton tchaton Awaiting requested review from tchaton tchaton is a code owner

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@ethanwharris ethanwharris Awaiting requested review from ethanwharris ethanwharris is a code owner

Assignees

@SkafteNicki SkafteNicki

Labels

bug Something isn't working pl Generic label for PyTorch Lightning package tuner

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Tuner leaves checkpoint files behind when interrupted

2 participants

@SkafteNicki @Borda