ninja==1.11.1.2 build fails when built with --no-binary=ninja (or --no-binary=:all:)

[TimotheusBachinger]

Since ninja==1.11.1.2, we cannot built numpy anymore from source as it will fail with:

LookupError: https://files.pythonhosted.org/packages/b4/49/4f1a79f99f4c3eb5d22f943bba14832923bb44423254d5089d38a9f6da63/ninja-1.11.1.2.tar.gz (from https://pypi.org/simple/ninja/) (requires-python:>=3.7) is already being built: ninja>=1.8.2 from https://files.pythonhosted.org/packages/b4/49/4f1a79f99f4c3eb5d22f943bba14832923bb44423254d5089d38a9f6da63/ninja-1.11.1.2.tar.gz

numpy does not pin ninja in its build dependencies, which explains why the build is now failing:
https://github.com/numpy/numpy/blob/b1e6ccd14b173b922f5d6a11bb252f4141dc42cd/requirements/build_requirements.txt#L3

Do you have any idea what causes this?

[johningve]

I'm also having issues with this new version. It seems like ninja is calling itself recursively:

[mayeut]

@TimotheusBachinger, it seems like the bootstrapping issue mentioned in scikit-build/cmake-python-distributions#503 (comment)

ninja requires cmake and cmake is requiring ninja.
Installing the system cmake should solve the issue.

Any reason you're not getting / using binary wheels ? (specific platform maybe ?)

@johningve, this doesn't seem like the same issue, can you please open a different issue with the reproducer please ?

[TimotheusBachinger]

@mayeut thanks for pointing into that direction - this could really have caused the issue in our case (and due to heavy caching by bazel, to use it looked like the new ninja version...)

[TimotheusBachinger]

I am re-opening the issue again as the cmake theory was a dead end for us.
Our quick fix for now it to use a constrain when building from source: Checkmk/checkmk@0316464

For us it is 100% the new version of ninja==1.11.1.2 which leds to the build failure.

[mayeut]

@TimotheusBachinger,

In order to better understand the problem and provide a fix:

Any reason you're not getting / using binary wheels ? Force build from source or specific platform maybe ?

[TimotheusBachinger]

@mayeut we're building for around 12 different platforms (debian, ubuntu, sles, centos, alma...) therefore we're building from source, yes (--no-binary=":all:")

[mayeut]

ok, so I have an idea where the issue is coming from in that case. I will try something after day work to reproduce and hopefully fix this.
It's still a bootstrapping issue which this time should be handled by the new build system used by Ninja PyPI distribution.

[TimotheusBachinger]

@mayeut sounds good, thanks for the effort and no hurry. We're fine with the constraints workaround.

[mayeut]

Here's a quick status:
The build backend switched over to scikit-build-core. scikit-build-core is aware of platforms where binary wheels are available for ninja and will insert a dependency on ninja in the build requirements if a satisfying version is not found on the system.
In a --no-binary=":all:" (or just --no-binary=ninja) condition, trying to request ninja while building ninja triggers a recursive build of ninja (caught by the build frontend: ... is already being built: ninja>=1.8.2 ...).

I'll propose something a bit better than what's already there in #276 and let our resident expert review this.

[cdce8p]

I'm seeing the same issue when trying to build ninja itself. A reproducer in case that's useful

docker pull python:3.13.0
docker run -it --rm python:3.13.0 bash

pip install -v --no-binary ninja ninja==1.11.1.2

A temporary workaround is to use --no-build-isolation.

docker run -it --rm python:3.13.0 bash

pip install scikit-build-core "hatch-fancy-pypi-readme>=22.3" setuptools-scm
pip install -v --no-binary ninja --no-build-isolation ninja==1.11.1.2

[harmstyler]

Is there any progress on this? Seems like there's an open Pull Request. Does this need extra help?

[mayeut]

This should be fixed with version 1.11.1.3 now available on PyPI