Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support torch.optim.lr_scheduler.ReduceLROnPlateau #320

Merged
merged 18 commits into from
Dec 3, 2019
Merged

Support torch.optim.lr_scheduler.ReduceLROnPlateau #320

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
1a173d8
feat: add reducelronplateau callback
vikmary Oct 6, 2019
acb30c0
feat: use reducelronplateau callback in trainer
vikmary Oct 6, 2019
6827262
feat: only on unsupported lr schedulers
vikmary Oct 6, 2019
b6f827b
feat: merge origin
vikmary Oct 8, 2019
fa4094e
feat: merge master origin (second try)
vikmary Oct 9, 2019
d232a7c
feat: last but not the least merge of master
vikmary Oct 9, 2019
94a285d
Merge branch 'master' into support-reduceonplateau-lr-scheduler
vikmary Oct 17, 2019
26844ac
feat: merge master
vikmary Oct 20, 2019
cae83ca
feat: support only on scheduler in reduceLrOnPlateauScheduler
vikmary Oct 20, 2019
29c5480
feat: merge origin
vikmary Nov 3, 2019
cd946a3
feat: merge origin & refactor
vikmary Nov 7, 2019
78b975b
refactor: code style
vikmary Nov 7, 2019
4f5ae52
Merge branch 'master' into support-reduceonplateau-lr-scheduler
williamFalcon Nov 30, 2019
c9d5618
Update pt_callbacks.py
williamFalcon Dec 1, 2019
75faae5
Update trainer.py
williamFalcon Dec 1, 2019
5465466
Update train_loop_mixin.py
williamFalcon Dec 1, 2019
d16a6c0
Update trainer.py
williamFalcon Dec 1, 2019
7650fa4
Update train_loop_mixin.py
williamFalcon Dec 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pytorch_lightning/callbacks/pt_callbacks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,7 @@ def on_epoch_end(self, epoch, logs=None):
f'\nEpoch {epoch:05d}: {self.monitor} reached',
f'{current:0.5f} (best {self.best:0.5f}), saving model to',
f'{filepath} as top {self.save_top_k}')
self.best = current
self._save_model(filepath)

else:
Expand Down
11 changes: 9 additions & 2 deletions pytorch_lightning/trainer/train_loop_mixin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,6 @@ def training_step(self, batch, batch_nb):
"""

import numpy as np
import tqdm

try:
from apex import amp
Expand Down Expand Up @@ -213,7 +212,15 @@ def train(self):
# update LR schedulers
if self.lr_schedulers is not None:
for lr_scheduler in self.lr_schedulers:
lr_scheduler.step(self.current_epoch)
lr_scheduler.step(epoch=self.current_epoch)
if self.reduce_lr_on_plateau_scheduler is not None:
val_loss = self.callback_metrics.get('val_loss')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems very specific. does it only need to work with val_loss?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it could be any validation metric in theory, but how do we let the user pass it in? Perhaps via a dedicated dict entry in the validation_end output similar to "log" and "progress_bar"? It's one more thing the user needs to remember, but maybe its fine since this lr_scheduler is optional and it should be mentioned in the docs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, good option. let’s do it in a separate PR?

if val_loss is None:
print('ReduceLROnPlateau conditioned on metric `%s` '
'which is not available. Available metrics are: %s' %
('val_loss', ','.join(list(self.callback_metrics.keys()))), RuntimeWarning)
exit(-1)
self.reduce_lr_on_plateau_scheduler.step(val_loss, epoch=self.current_epoch)

# early stopping
met_min_epochs = epoch_nb > self.min_nb_epochs
Expand Down
10 changes: 10 additions & 0 deletions pytorch_lightning/trainer/trainer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,8 @@ def __init__(self,
self.early_stop_callback = None
self.configure_early_stopping(early_stop_callback, logger)

self.reduce_lr_on_plateau_scheduler = None

# configure checkpoint callback
self.checkpoint_callback = checkpoint_callback
self.weights_save_path = weights_save_path
Expand Down Expand Up @@ -376,12 +378,20 @@ def init_optimizers(self, optimizers):
# two lists
elif len(optimizers) == 2 and isinstance(optimizers[0], list):
optimizers, lr_schedulers = optimizers
lr_schedulers, self.reduce_lr_on_plateau_scheduler = self.configure_schedulers(lr_schedulers)
return optimizers, lr_schedulers

# single list or tuple
elif isinstance(optimizers, list) or isinstance(optimizers, tuple):
return optimizers, []

def configure_schedulers(self, schedulers):
for i in range(len(schedulers)):
if isinstance(schedulers[i], torch.optim.lr_scheduler.ReduceLROnPlateau):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

proof here

reduce_lr_on_plateau_scheduler = schedulers.pop(i)
return schedulers, reduce_lr_on_plateau_scheduler
return schedulers, None

def run_pretrain_routine(self, model):
"""
Sanity check a few things before starting actual training
Expand Down