Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Checkpoint saving order #2680

Closed
shtoshni opened this issue Jul 23, 2020 · 2 comments · Fixed by #2881
Closed

Checkpoint saving order #2680

shtoshni opened this issue Jul 23, 2020 · 2 comments · Fixed by #2881
Assignees
@awaelchli
Labels
bug Something isn't working feature Is an improvement or enhancement help wanted Open to be worked on

Comments

@shtoshni
Copy link

shtoshni commented Jul 23, 2020 •
edited
Loading

The last model save action, here, should be after saving the top k model because the best model and best score could have changed. Swapping the order allows resuming the training from the last checkpoint, with the last checkpoint having the latest information about the best model path/score.
@github-actions
Copy link
Contributor

Hi! thanks for your contribution!, great first issue!

@edenlightning edenlightning added feature Is an improvement or enhancement help wanted Open to be worked on labels Jul 29, 2020
@awaelchli awaelchli added the bug Something isn't working label Aug 8, 2020
@awaelchli awaelchli self-assigned this Aug 8, 2020
@awaelchli awaelchli mentioned this issue Aug 8, 2020
Merged
7 tasks
@awaelchli
Copy link
Contributor

awaelchli commented Aug 8, 2020 •
edited
Loading

@shtoshni92 Thanks for the bug report and sorry for the long wait, it was actually an easy fix (see linked PR)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@awaelchli awaelchli

Labels
bug Something isn't working feature Is an improvement or enhancement help wanted Open to be worked on
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

save last model after saving top_k when save_last=True Lightning-AI/pytorch-lightning
3 participants
@awaelchli @shtoshni @edenlightning