Feature Request: Allow reading dependencies (and dev-dependencies) from requirements.txt file

[ntninja]

While it isn't a huge maintanance burden it would be nice if we could share our requirements files between tox, CI & packaging.

I guess this should either be done by adding tox/PIP-style syntax to for requires:

requires = -r requirements.txt

or adding a separate requires-file option. I'd prefer the first option because it is similar to how tox handles this stuff (it was the first thing I tried anyways) and because it makes it more obvious that the external file belongs to the same requirement list as the other stuff possibly listed.

[Carreau]

I think this might need to wait for pep517 to be accepted an released as then we'll move away from flit.ini to pyproject.toml (better not add features now right?)

The other question is how to do things the right way: requirements.txt is mostly to pin dependencies for applications, so should (in theory) be relatively orthogonal to what's in setup.py/flit.ini hopefully once pyproject.toml is usable we can solve this issue !

[ntninja]

Do we have a timescale when PEP-517 & PEP-518 will be implemented in pip & tox?
Until tox fully integrates with PEP-517 and uses get_requires_for_build_wheel for dependency resolving, the dependency list will have to be duplicated (correct me if I'm wrong).

Don't get me wrong: Duplicating our short dependency list is definitely not a huge burden, but I'd like to know where we're at on this.

[takluyver]

pip master already implements 518, I think, and I hope that supporting 517 will be relatively rapid once the PEP is agreed on. I don't know who maintains tox, or how interested they are in these discussions.

[ntninja]

Yes, I found your PEP-518 PR for PIP and have now opened an issue with detailed instructions on tox' source repository (maybe you can quickly skim it in case I got anything wrong on the technical side?): tox-dev/tox#573

We'll see what their stance on this is…

[merwok]

This would be great too for workflows using pip-tools to generate completed pinned requirements from a source list of direct dependencies (requirements.in → requirements.txt)

[takluyver]

I believe tox now supports PEP 517, so it should be able to get requirements from a flit package.

The requirements you specify with flit are abstract requirements (see Donald's blog post about this). requirements.txt is typically used to specify concrete, pinned dependencies. So reading that doesn't make sense.

You could write a tool which reads dependencies specified for Flit and creates a list of pinned versions for them. They're configured in Flit's section of the pyproject.toml file, so you're at the mercy of Flit changing the format for dependencies, but I don't see that happening in the near future.

Or look at other tools like poetry and pipenv which handle translating abstract dependencies into concrete ones.

[merwok]

The requirements you specify with flit are abstract requirements

Agreed for a library, but I’m making a web app with flit packaging so I must still use pip-tools and pip to manage requirements. I understand if this isn’t in flit’s target audience though; I may look at adding a flit reader to pip-compile.

That said, it was really nice to see entry points and install -s work just fine! This may be my first project without a commit fix MANIFEST.in 😆

[takluyver]

It's primarily designed around library development, and the kind of small command-line tools that we often manage like libraries. Things like poetry & pipenv probably handle application dependencies better. But if you come up with better ways to use Flit for that kind of scenario, do post them here. 🙂

[merwok]

(For what it’s worth, mixing pip-tools and a flit config to work on a web app that needs installation (because it needs entry points, uses src layout, uses tox with install step, etc) hits too many issues to be a good method. Using poetry is probably the right call for this.)

[romanchyla]

leaving comment here just in case it helps someone; I'm using the following which works

credit: https://stackoverflow.com/questions/32688688/how-to-write-setup-py-to-include-a-git-repository-as-a-dependency

[project]
...
dependencies = [
    'rutils @ git+https://github.com/romanchyla/rutils@#egg=rutils',
]

[build-system]
requires = ["setuptools >= 58.6.0", "wheel", "flit_core >=3.2,<4", "ppsetuptools==2.0.2"]
#build-backend = "setuptools.build_meta"
build-backend = "flit_core.buildapi"

I find the distinction between abstract/concrete dependencies to be way too abstract to be pragmatic and/or useful; when we are developing libraries we also tend to develop applications that use those libraries; it's a messy world, so our tools need to be simple but not simpler than necessary...