Checkpointing with SLURM

[Taha-Bahadori]

What is your question?

I have a pytorch-lightning code with checkpointing that runs well on my desktop. But when I run it on our cluster with SLURM, the checkpoints do not get saved.

Code

    model = Predictor(args)
    check = ModelCheckpoint(save_top_k=1, verbose=True, monitor='val_acc', mode='max',
            filepath='checks/{epoch})
    trainer = pl.Trainer(checkpoint_callback=check, max_epochs=100, gpus=1)
    trainer.fit(model)

What have you tried?

I run it in the cluster with the following code:

salloc -G 1 srun python main.py

What's your environment?

• OS: Linux
• Packaging conda

[github-actions]

Hi! thanks for your contribution!, great first issue!

[williamFalcon]

is this with 0.8.1?

[Taha-Bahadori]

It was with 0.8.0. I upgraded to 0.8.1 and the problem persists.

Also, my manual checkpointing with torch.save does work.

[ExpectationMax]

Could be related to issue with #2231 experienced the same problems in my setup.

[ExpectationMax]

My hypothesis is that something is going wrong with determining the rank of the process when running on slurm, thus the logger calls are never executed and some outputs for example

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0]

Are never shown, otherwise training executes as regulalry.

This could be due to an incorrect (non-zero) value in rank_zero_only.rank such that the wrapped commands are never executed.

[awaelchli]

You might want to try after this #1504 is merged.

[jeremyjordan]

Can you check to see if the weights are being saved under trainer.weights_save_path?
https://github.com/PyTorchLightning/pytorch-lightning/blob/f278ac42c81fc344fe1cb673877b8a9dfca9c9b5/pytorch_lightning/trainer/training_io.py#L224

My guess is that they're being saved but for some reason hpc_save doesn't save to the normal checkpoint directory.

[Borda]

is this fixed by #2341?

[williamFalcon]

yes