instable GitHub action cache

[Borda]

🐛 Bug

there is some issue with GH action and caching as it is randomly failing with using Horovod

To Reproduce

#1709 (comment)

[tgaddair]

If the goal is to keep the cache to speed things up, my suggestion would be to add a couple checks in the "Install dependencies" step in the workflow:

1. Check the version of torch before installing requirements.txt, save in Bash variable.
2. Install requirements.txt
3. Check the version of torch after, save in Bash variable.
4. If torch version before != version after, then uninstall Horovod.
5. Install requirements-extra.txt.

This way, we should be able to leverage the cache to speed things up when nothing changes, without running into incompatibilities when the torch version is upgraded.

I can put together a PR for this.

[Borda]

@tgaddair PR would be great!
I am a bit suspicions as you mentioned that the cache is not loaded properly so thinking about opening an issue with https://github.com/actions/cache

[Borda]

I see the issue now, the problem is that the min requirement of horovod is the actual one so the loaded cache even with --upgrade is satisfied
https://github.com/PyTorchLightning/pytorch-lightning/pull/1709/checks?check_run_id=641571597