prepare_data called multiple times per node for slurm and elastic training

[tullie]

🐛 Bug

Slurm and elastic training create the training processes per node outside of the lightning context. This means that when the fit function calls prepare_data, the assumption that it's only being called on proc 0 is broken and it gets called for each process.

This is an issue computational reasons (e.g. downloading a whole dataset) and for training stability if the data preparation process isn't deterministic.

See calling code here:
https://github.com/PyTorchLightning/pytorch-lightning/blob/7c7e50ca4702a5b35bc1b80d44bca7606552093a/pytorch_lightning/trainer/trainer.py#L825

To Reproduce

Steps to reproduce the behavior:

1. Add print statements to prepare_data
2. Train a lightning model with either slurm or elastic training
3. See that it's being called multiple times.

Expected behavior

Expected prepare_data to only be called once per node.

[edenlightning]

@Borda any idea how to fix?