Consider egg-info/requires.txt when determining requirements hash

[dgilland]

Consider the scenario where install_requires changes in setup.py but isn't reflected in any requirements file (e.g. maybe the requirements file only lists test dependencies and package dependencies only exist in setup.py).

Currently, this does not trigger a venv recreate since tox-battery only checks deps defined in tox.ini, and tox itself doesn't trigger a recreate either in this case (at least in my initial testing I didn't see this). This can result in reusing a venv with an old dependency installed from setup.py.

What if tox-battery used those dependencies in addition to the ones in the requirements files to generate its requirements hash? It's possible to parse the contents of <pkgname>.egg-info/requires.txt to get a list of dependencies defined in setup.py so that approach may work in most cases.

Thoughts?

[signalpillar]

Hi @dgilland , I've recently created same issue here as I was thinking about the same. And, for instance, in my projects - I don't operate with requirement files much mostly project dependencies are in the setup.py. So I like the idea and need it as well. Actually not only we need it community has a big discussion here about this.

I've just found also this project and it should support the functionality we need, I need to figure out how it works (looks like https://github.com/Yelp/venv-update is used).

I need to investigate about .egg-info/requires.txt a bit, would like to understand more. Also I was thinking about mocking setup function that worked for me earlier in similar tasks to get meta-data from setup.py.

[asottile]

Hi friend (noticed the star) -- venv-update (pip-faster) instruments (monkeypatches) pip to hook into the dependency resolution and has two goals:

• speed up cached wheels that are ==d in requirements files
• prune installed packages that are no longer depended on (--prune)

The venv-update plugin of tox-pip-extensions simply adds another stage which calls pip-faster install ... --prune with the total set of dependencies after tox has handled installation. There's some slight overhead of rerunning setup.py for each invocation, but having up-to-date dependencies is way worth it.

Happy to elaborate :)

[dgilland]

@asottile I wasn't able to use tox-pip-extensions because it required that install_command not be set in tox.ini. But for my needs, I need to include --process-dependencies in the pip install command since some of our setup.py files use dependency_links due to internal Python dependencies. having install_command set resulted in tox-pip-extensions complaining about that setting not being allowed.

@signalpillar I spent some time trying to track down how to read the setup() metadata using setuptools but I could only find the .egg-info/requires.txt method (granted I may have missed something). If you know of a way to mock setup to get that information more easily, then that'd be great!

[asottile]

tox-pip-extensions does most of its work by modifying install. You could get it to work still by using environment variables to control pip (via setenv) probably

[dgilland]

@asottile Thanks for the suggestion! Using setenv = PIP_PROCESS_DEPENDENCY_LINKS = 1, I was able to incorporate tox-pip-extensions into my tox workflow.