Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stopping the code along with a graceful shutdown. #1988

Closed
nsidn98 opened this issue May 28, 2020 · 3 comments
Closed

Stopping the code along with a graceful shutdown. #1988

nsidn98 opened this issue May 28, 2020 · 3 comments
Labels
question Further information is requested

Comments

@nsidn98
Copy link

nsidn98 commented May 28, 2020

Is there a way to stop the training in the model when some criteria are satisfied. Something along the lines:

class myCallback(Callback):
    def __init__(self):
        ...
    def on_epoch_end(self, trainer, pl_module):
        if criteria:
            model.stop_training = True # stops the training; need help here

Note that I also want to have the early stopping feature where the 'val_loss' is monitored but want to stop running the code if some other criteria is satisfied. Also, is my method of having this feature in the callback module correct or should I inherit the early stopping criteria?

@nsidn98 nsidn98 added the question Further information is requested label May 28, 2020
@github-actions
Copy link
Contributor

Hi! thanks for your contribution!, great first issue!

@awaelchli
Copy link
Contributor

awaelchli commented May 28, 2020

You could raise KeyboardInterrupt, this will lead to a graceful shutdown. There is some work in progress in this PR:
#1631
But it should already work if you raise it from within your code

@nsidn98
Copy link
Author

nsidn98 commented May 29, 2020

Got it by using raise KeyboardInterrupt in the code.
Closing the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants