Add PEP 518 build-requires support

[jaraco]

Although the PEP 518 support added by pip 10 goes a long way toward supporting the use cases of installing dependencies for the "install" and "wheel [build]" steps, it is of much less help for many of the other use-cases that setuptools setup_requires facilitates. A few prominent ones:

• setup.py test
• setup.py pytest (and other custom commands)
• setup.py check_docs

But more generally, anything that invokes setup.py for its distutils-based behaviors.

@pganssle has proposed that Setuptools should present a CLI that also supports PEP 518 for this use-case, and I'm convinced something like that is needed, as I also as hard as I try am finding it difficult to convert projects away from the tried-and-true paradigms to whole new ones without real issues arising.

But it also occurs to me that there is a project that has support for installing packages on demand and which might be a suitable bridge for setup.py, and that's this project (RWT).

RWT already builds on pip and builds throw away environments for the duration of a command (not dissimilar to what setup-requires does). And RWT was written with supplanting many setup-requires use-cases.

What RWT doesn't have is support for PEP 518 build requirements, but since it's a thin wrapper around pip, if pip were to supply an argument to pip install, namely --build-reqs, RWT could build on that also, and it would remove the need for multiple packages to have build-requirement installer support.

Here's how it would work. If invoked on a directory containing a pyproject.toml (or referencing such a file), pip would install the build requirements:

(myenv) $ pip install --build-reqs .
... pip installs build requirements defined in pyproject.toml (permanently)

But in the case where a user does not wish to install the build requirements into the environment, but does wish to invoke a setup.py command with the build requirements present, they could use rwt as the tool to do so:

$ rwt --build-reqs . -- setup.py test
... rwt invokes `install --build-reqs` into a temporary directory, adds to PYTHONPATH, then invokes setup.py test.

I was originally thinking that pip should expose a new command install-build-reqs, but then I realized that rwt only wraps pip install, so a parameter to pip install makes a lot more sense.

Alternatively, rwt could implement this support more directly (without any modifications to pip) by adding a parameter that (a) bypasses default behavior of passing parameters to pip and (b) reads the pyproject.toml and passes the parsed deps to pip install instead.

Advantages of this proposed approach:

• separation of concerns (temporary, isolated install of build deps from command execution)
• substantial code re-use without needing to extract a library for PEP-518
• applies to many more use-cases outside of setuptools (works with other build tools)
• developer does not need setuptools as a prerequisite (only pip and rwt... and only pip if rwt were to become pip run as proposed in Feature: incorporate pip-run as pip run pypa/pip#3971)
• setuptools won't need to build yet another tool for installing transient requirements

Disadvantages:

• Developer would need yet another tool (rwt) to gain the functionality (barring Feature: incorporate pip-run as pip run pypa/pip#3971).
• rwt design doesn't keep the build environment around the way setup-requires does, so the startup cost is paid at each invocation (a cost which is substantially mitigated in some cases by pip caches).

@pganssle - if you haven't already, would you give rwt a trial run (I suggest having it installed in your global environment; it's lightweight and often immensely useful) and after let me know what you think about the proposal.

@benoit-pierre I'd be interested in your opinion also.

[benoit-pierre]

I have a WIP for adding support to pip for using pip install -r pyproject.toml (when the basename is pyproject.toml, the build-system.requires key is automatically used, otherwise -r file.toml[key1,key2] can also be used).

FWIW, I also have a WIP for using pip instead of easy_install for Distribution.fetch_build_egg; basically using pip wheel --no-dep req to build a wheel and installing it as an egg.

Note: preferring setuptools' pkg_resources over pip's vendored version here would slightly lower the costs of using rwt when all requirements are already available.

[benoit-pierre]

• WIP support for TOML requirements in pip: pypa/pip@master...benoit-pierre:toml_requirements
• WIP support for using pip over easy_install in setuptools: pypa/setuptools@master...benoit-pierre:pip_wheels (note the need for a patched pip to workaround utils: fix has_leading_dir corner case pypa/pip#5794)

[jaraco]

I have a WIP for adding support to pip

That would solve this issue without any changes to pip-run (formerly rwt).

[jaraco]

I'm not sure this feature is needed here. I'm going to close this unless someone else wants to revive it.