[question] How to handle dependencies on Python modules in Conan packages?

[IgorIvanov-Kudan]

What is your question?

Suppose there is a C++ lib X that requires some Python module Y to be built. The simplest use case is that it uses Y for code generation or transformation, by running some Python script during CMake config or build. How should one approach providing / specifying such dependency in a recipe for X?

(Here is the example from the library we are currently trying to package - it runs python -c "import em ; em.invoke(...)" during CMake configuration to do some codegen and requires package empy to be installed for that)

Conan has some automation to work with system package managers like apt or pacman - which is nice because one can control their behavior, as to whether or not it should actually install something (even using sudo), or just notify of missing packages and fail. Is there anything like that for pip with similar configuration possibilities? If not, what would be the preferred method to install the package "unconditionally"?

There is this SO answer by @uilianries from 2019, which suggests to just import pip ; pip.main(['install', 'Y']) - which feels slightly unsafe and is probably outdated by now. Even if it is not, I believe the proper way would be to subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'Y']), as the pip.main approach generates warnings from Python:

[2023-09-22T08:45:09.348Z] WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
[2023-09-22T08:45:09.348Z] Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
[2023-09-22T08:45:09.348Z] To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
[2023-09-22T08:45:09.348Z] Defaulting to user installation because normal site-packages is not writeable

Additionaly one might as well want to separate "build-time module dependency" from "runtime module dependency". For example, if Y is only used for code generation, it might make sense to install it into a separate virtual environment - and do that only when you build the package from sources. But if X would actually require Python for its runtime, then Y should be installed to the same Python that is being used by the rest of the project...

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[samuel-emrys]

There's a proposal to introduce features into conan to support the management of python virtual environments that I've been working to mature in #11601. The current blocker is that the cpython recipe has not been migrated to conan 2 yet, however, I have prototyped this functionality in some recipes that I currently have outside of conan-center: https://github.com/samuel-emrys/python-virtualenv

This will essentially allow you to specify your python requirements in your conanfile and install them into a virtual environment in your conan cache. The programs installed into this virtual environment will be made available through cmake targets.