"RuntimeError: Address already in use" when running multiple multi-gpu training (DDP).

[kyoungrok0517]

Describe the bug
I see "RuntimeError: Address already in use" error message if I try to run two multi-gpu training session (using ddp) at the same time.

To Reproduce
Run two multi-gpu training session at the same time.

Expected behavior
Able to run two multi-gpu training session at the same time.

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu 18.04
• Pytorch 1.3.0
• CUDA 10.1

[williamFalcon]

you have to set master_port yourself if running 2 ddp sessions on the same machien at the same time. This is a PyTorch limitation. in all the other ddp settings you wouldn’t have to worry about this.

https://pytorch-lightning.readthedocs.io/en/latest/Trainer/Distributed%20training/#multi-node

[kyoungrok0517]

Great. Thanks!