“invalid command 'bdist_wheel'” on first execution #10

adamchainz · 2024-02-17T06:07:36Z

Using macOS, Python 3.12, running the test suite for django-upgrade, which has package = wheel.

On the first run, environments all fail with:

.pkg: install_requires_for_build_wheel> /Users/chainz/.pyenv/versions/3.12.1/bin/uv pip install wheel
.pkg: build_wheel> python /Users/chainz/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py312: packaging backend failed (code=usage: _backend.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: _backend.py --help [cmd1 cmd2 ...]
   or: _backend.py --help-commands
   or: _backend.py cmd --help

...

error: invalid command 'bdist_wheel'
Backend: run command build_wheel with args {'wheel_directory': '/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/dist', 'config_settings': None, 'metadata_directory': None}
Backend: Wrote response {'code': "usage: _backend.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]\n   or: _backend.py --help [cmd1 cmd2 ...]\n   or: _backend.py --help-commands\n   or: _backend.py cmd --help\n\nerror: invalid command 'bdist_wheel'", 'exc_type': 'SystemExit', 'exc_msg': "usage: _backend.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]\n   or: _backend.py --help [cmd1 cmd2 ...]\n   or: _backend.py --help-commands\n   or: _backend.py cmd --help\n\nerror: invalid command 'bdist_wheel'"} to /var/folders/20/lzgtdyzs7wj5fc_90w1h3bhc0000gn/T/pep517_build_wheel-nzkrwy_j.json

Full log

py312: remove tox env folder /Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/py312
.pkg: remove tox env folder /Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg
py312: venv> /Users/chainz/.pyenv/versions/3.12.1/bin/uv venv -p 3.12 --seed /Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/py312/.venv
py312: install_deps> /Users/chainz/.pyenv/versions/3.12.1/bin/uv pip install -r requirements/py312.txt
.pkg: venv> /Users/chainz/.pyenv/versions/3.12.1/bin/uv venv -p 3.12 /Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv
.pkg: install_requires> /Users/chainz/.pyenv/versions/3.12.1/bin/uv pip install setuptools
.pkg: _optional_hooks> python /Users/chainz/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /Users/chainz/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: install_requires_for_build_wheel> /Users/chainz/.pyenv/versions/3.12.1/bin/uv pip install wheel
.pkg: build_wheel> python /Users/chainz/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py312: packaging backend failed (code=usage: _backend.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: _backend.py --help [cmd1 cmd2 ...]
   or: _backend.py --help-commands
   or: _backend.py cmd --help

error: invalid command 'bdist_wheel'), with SystemExit: usage: _backend.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: _backend.py --help [cmd1 cmd2 ...]
   or: _backend.py --help-commands
   or: _backend.py cmd --help

error: invalid command 'bdist_wheel'
Traceback (most recent call last):
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 534, in _parse_command_opts
    cmd_class = self.get_command_class(command)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/dist.py", line 713, in get_command_class
    return _Distribution.get_command_class(self, command)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 844, in get_command_class
    raise DistutilsModuleError("invalid command '%s'" % command)
distutils.errors.DistutilsModuleError: invalid command 'bdist_wheel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 172, in setup
    ok = dist.parse_command_line()
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 475, in parse_command_line
    args = self._parse_command_opts(parser, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/dist.py", line 865, in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 536, in _parse_command_opts
    raise DistutilsArgError(msg)
distutils.errors.DistutilsArgError: invalid command 'bdist_wheel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chainz/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pyproject_api/_backend.py", line 90, in run
    outcome = backend_proxy(parsed_message["cmd"], **parsed_message["kwargs"])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pyproject_api/_backend.py", line 32, in __call__
    return getattr(on_object, name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/build_meta.py", line 404, in build_wheel
    return self._build_with_temp_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
    self.run_setup()
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 1, in <module>
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/.venv/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 174, in setup
    raise SystemExit(gen_usage(dist.script_name) + "\nerror: %s" % msg)
SystemExit: usage: _backend.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: _backend.py --help [cmd1 cmd2 ...]
   or: _backend.py --help-commands
   or: _backend.py cmd --help

error: invalid command 'bdist_wheel'
Backend: run command build_wheel with args {'wheel_directory': '/Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg/dist', 'config_settings': None, 'metadata_directory': None}
Backend: Wrote response {'code': "usage: _backend.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]\n   or: _backend.py --help [cmd1 cmd2 ...]\n   or: _backend.py --help-commands\n   or: _backend.py cmd --help\n\nerror: invalid command 'bdist_wheel'", 'exc_type': 'SystemExit', 'exc_msg': "usage: _backend.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]\n   or: _backend.py --help [cmd1 cmd2 ...]\n   or: _backend.py --help-commands\n   or: _backend.py cmd --help\n\nerror: invalid command 'bdist_wheel'"} to /var/folders/20/lzgtdyzs7wj5fc_90w1h3bhc0000gn/T/pep517_build_wheel-nzkrwy_j.json
py312: FAIL ✖ in 1.15 seconds

On a re-run, environments succeed.

This happens whether or not I add uv_seed = true to [testenv].

The text was updated successfully, but these errors were encountered:

gaborbernat · 2024-02-19T01:28:49Z

Can you please update the run output with https://github.com/tox-dev/tox-uv/releases/tag/1.3.0 using tox -rvvv. Thanks!

reinout · 2024-02-19T19:56:35Z

I'm having the same problem (I think), perhaps my output can help, too? (Also mac, 3.12)
I've run just one environment to keep it a bit shorter:

$ tox -rvvv -e coverage > output_first.txt 2>&1
$ tox -vvv -e coverage > output_second.txt 2>&1

The first one fails with the bdist_wheel error, the second one runs fine.

output_first.txt
output_second.txt

The relevant tox.ini part, in case that helps: https://github.com/nens/nens-meta/blob/f9eaffa3f38b3b25dd84350453386d817b584c5d/tox.ini#L53-L69

Perhaps more relevant is my pyproject.toml, https://github.com/nens/nens-meta/blob/f9eaffa3f38b3b25dd84350453386d817b584c5d/pyproject.toml , it could be that requires = ["setuptools>=69"] in the [build-system] is relevant.

reinout · 2024-02-19T19:59:34Z

Oh, I install tools like tox with pipx (and then pipx inject tox tox-uv in this case) so that tox+tox-uv is nicely isolated from the rest of the python environment: that might help or not.

gaborbernat · 2024-02-20T00:57:21Z

Ok, I think I understand what's happening here. I'm not sure if it's a setuptools or pyproject-api bug, but the short of it is that pyproject-api, does not restart the backend subprocess runs. The environment starts out with just setuptools installed into it. We ask the backend what dependencies it needs, comes back with wheels, and we then ask it to build a wheel. The problem is that at this point, setuptools does not reload its registered commands... so us installing the wheel package is not picked up.

Workarounds for now, either will work:

Define wheel in your build backend in your pyproject.toml:

[build-system]
build-backend = "build"
requires = [ "wheel", "setuptools"]

Define wheel as packaging dependency in tox.ini:

[testenv:.pkg]
uv_seed = true
deps = wheel

pyproject-api is not following this recommendation https://peps.python.org/pep-0517/#build-environment:

Frontends should call each hook in a fresh subprocess, so that backends are free to change process global state (such as environment variables or the working directory). A Python library will be provided which frontends can use to easily call hooks this way.

But note that is a recommendation and not a requirement. So who's in wrong here between pyproject-api and setuptools is unclear here.

adamchainz · 2024-02-20T07:17:37Z

Thank you for the investigation. I tried the workarounds.

The first works swimmingly, though I’d rather not affect the public install definition if possible.

The second one fixes only the Python 3.12 environment, the same Python version that I have tox and uv installed in. It leaves the others still failing with “invalid command 'bdist_wheel'”. I discovered I needed to explicitly list all the .pkg environments to fix them:

[testenv:.pkg{,-cpython311,-cpython310,-cpython39,-cpython38}]
uv_seed = true
deps = wheel

For now, I’ll just use the first one without committing.

reinout · 2024-02-20T12:52:10Z

Thanks! The first workaround works for me.
Like @adamchainz , the second workaround works only for the "default" python, not for the other python versions.

👍 btw for trying to get uv into tox so quickly after uv was published!

gaborbernat · 2024-02-20T20:02:15Z

I think the only reason this is not failing in tox, is because virtualenv does this patching https://github.com/pypa/virtualenv/blob/main/src/virtualenv/create/via_global_ref/_virtualenv.py#L1. And here we switched to uv venv that doesn't do that. As for paths ahead, I think there's nothing we can do here; though in tox we should have a use_fresh_process = false flag that one can turn on for backends that require this. Created tox-dev/tox#3227 to track that.

gaborbernat · 2024-03-06T00:06:26Z

I think the only reason this is not failing in tox, is because virtualenv does this patching pypa/virtualenv@main/src/virtualenv/create/via_global_ref/_virtualenv.py#L1. And here we switched to uv venv that doesn't do that. As for paths ahead, I think there's nothing we can do here; though in tox we should have a use_fresh_process = false flag that one can turn on for backends that require this. Created tox-dev/tox#3227 to track that.

This feature is now release https://tox.wiki/en/4.14.0/config.html#fresh_subprocess can give it a go.

reinout · 2024-03-08T10:15:53Z

I wanted to test out fresh_subprocess, but I cannot reproduce the original issue anymore. I did "pipx upgrade" tox (+tox-uv) beforehand, of course. The combination of a new tox + tox-uv + uv (and probably a somewhat fresher setuptools) apparently fixed it even without needing fresh_subprocess? I'll keep a look-out for other projects that might still fail. Anyway, "it works for me" :-)

Perhaps @adamchainz can reproduce the original issue and see if the fix works?

adamchainz · 2024-03-08T12:01:01Z

tox 4.14.1 made fresh_subprocess default to True for setuptools, so you don’t need to declare it explicitly: https://tox.wiki/en/latest/changelog.html#v4-14-1-2024-03-06 . @gaborbernat made this change after my bug report: tox-dev/tox#3235 .

So yeah it “just works” now.

reinout · 2024-03-08T13:48:18Z

:-) Perfect. Thanks!

gaborbernat added bug help wanted labels Feb 18, 2024

gaborbernat mentioned this issue Feb 20, 2024

Support enabling fresh subprocess runs for packaging tox-dev/tox#3227

Closed

gaborbernat closed this as completed Feb 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

“invalid command 'bdist_wheel'” on first execution #10

“invalid command 'bdist_wheel'” on first execution #10

adamchainz commented Feb 17, 2024

gaborbernat commented Feb 19, 2024

reinout commented Feb 19, 2024

reinout commented Feb 19, 2024

gaborbernat commented Feb 20, 2024 •

edited

Loading

adamchainz commented Feb 20, 2024

reinout commented Feb 20, 2024

gaborbernat commented Feb 20, 2024

gaborbernat commented Mar 6, 2024

reinout commented Mar 8, 2024

adamchainz commented Mar 8, 2024

reinout commented Mar 8, 2024

“invalid command 'bdist_wheel'” on first execution #10

“invalid command 'bdist_wheel'” on first execution #10

Comments

adamchainz commented Feb 17, 2024

gaborbernat commented Feb 19, 2024

reinout commented Feb 19, 2024

reinout commented Feb 19, 2024

gaborbernat commented Feb 20, 2024 • edited Loading

adamchainz commented Feb 20, 2024

reinout commented Feb 20, 2024

gaborbernat commented Feb 20, 2024

gaborbernat commented Mar 6, 2024

reinout commented Mar 8, 2024

adamchainz commented Mar 8, 2024

reinout commented Mar 8, 2024

gaborbernat commented Feb 20, 2024 •

edited

Loading