build is not thread safe #849
Assignees
Labels
bug:normal
affects many people or has quite an impact
level:medium
rought estimate that this might be neither easy nor hard to implement
Milestone
Comments
gaborbernat
added
the
level:medium
samdbmg
added a commit
to bbc/rd-apmm-python-lib-mediatimestamp
that referenced
this issue
Jul 3, 2018
Works around a tox race condition when generating Python sdists. Tox bug: tox-dev/tox#849
samdbmg
added a commit
to bbc/rd-apmm-python-lib-mediajson
that referenced
this issue
Jul 3, 2018
Works around a tox race condition when creating Python sdists. Tox bug: tox-dev/tox#849
|
So here's an easy solution for this. Add a flag that once added will make the builds unique by using a unique build folder. To stop such builds generating too many folders remove these unique build folders when the tools exists. People wanting to see the produced sdist can run tox without this extra flag. |
gaborbernat
added a commit
that referenced
this issue
Aug 9, 2018
At the moment the build phase is not thread safe. Running multiple instances of tox will highly likely cause one of the instances to fail as the two processes will step on each others toe while creating the sdist package. This is especially annoying in CI environments where you want to run tox targets in parallle (e..g Jenkins). By passing in ``--parallel--safe-build`` flag tox automatically generates a unique dist folder for the current build. This way each build can use it's own version built package (in the install phase after the build), and we avoid the need to lock while building. Once the tox session finishes remove such build folders to avoid ever expanding source trees when using this feature.
|
Fixed via #934 . |
DmytroLitvinov
referenced
this issue
in DmytroLitvinov/kuna
Aug 20, 2018
### Update [tox](https://pypi.org/project/tox) from **3.1.3** to **3.2.1**. <details> <summary>Changelog</summary> ### 3.2.0 ``` ------------------ Features ^^^^^^^^ - Switch pip invocations to use the module ``-m pip`` instead of direct invocation. This could help avoid some of the shebang limitations. - by :user:`gaborbernat` (`935 <https://github.com/tox-dev/tox/issues/935>`_) - Ability to specify package requirements for the tox run via the ``tox.ini`` (``tox`` section under key ``requires`` - PEP-508 style): can be used to specify both plugin requirements or build dependencies. - by :user:`gaborbernat` (`783 <https://github.com/tox-dev/tox/issues/783>`_) - Allow to run multiple tox instances in parallel by providing the ``--parallel--safe-build`` flag. - by :user:`gaborbernat` (`849 <https://github.com/tox-dev/tox/issues/849>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/tox - Changelog: https://pyup.io/changelogs/tox/ - Docs: https://tox.readthedocs.org/ </details> ### Update [Sphinx](https://pypi.org/project/Sphinx) from **1.7.6** to **1.7.7**. <details> <summary>Changelog</summary> ### 1.7.7 ``` ===================================== Bugs fixed ---------- * 5198: document not in toctree warning when including files only for parallel builds * LaTeX: reduce "Token not allowed in a PDF string" hyperref warnings in latex console output (refs: 5236) * LaTeX: suppress "remreset Warning: The remreset package is obsolete" in latex console output with recent LaTeX (refs: 5237) * 5234: PDF output: usage of PAPER environment variable is broken since Sphinx 1.5 * LaTeX: fix the :confval:`latex_engine` documentation regarding Latin Modern font with XeLaTeX/LuaLateX (refs: 5251) * 5280: autodoc: Fix wrong type annotations for complex typing * autodoc: Optional types are wrongly rendered * 5291: autodoc crashed by ForwardRef types * 5211: autodoc: No docs generated for functools.partial functions * 5306: autodoc: ``getargspec()`` raises NameError for invalid typehints * 5298: imgmath: math_number_all causes equations to have two numbers in html * 5294: sphinx-quickstart blank prompts in PowerShell ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/sphinx - Changelog: https://pyup.io/changelogs/sphinx/ - Homepage: http://sphinx-doc.org/ </details> ### Update [cryptography](https://pypi.org/project/cryptography) from **2.3** to **2.3.1**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/cryptography - Changelog: https://pyup.io/changelogs/cryptography/ - Repo: https://github.com/pyca/cryptography </details>
guykisel
referenced
this issue
in guykisel/inline-plz
Sep 24, 2018
This PR updates [tox](https://pypi.org/project/tox) from **3.1.2** to **3.4.0**. <details> <summary>Changelog</summary> ### 3.4.0 ``` ------------------ Bugfixes ^^^^^^^^ - add ``--exists-action w`` to default pip flags to handle better VCS dependencies (`pip documentation on this <https://pip.pypa.io/en/latest/reference/pip/exists-action-option>`_) - by :user:`gaborbernat` (`503 <https://github.com/tox-dev/tox/issues/503>`_) - instead of assuming the Python version from the base python name ask the interpreter to reveal the version for the ``ignore_basepython_conflict`` flag - by :user:`gaborbernat` (`908 <https://github.com/tox-dev/tox/issues/908>`_) - PEP-517 packaging fails with sdist already exists, fixed via ensuring the dist folder is empty before invoking the backend and `pypa/setuptools 1481 <https://github.com/pypa/setuptools/pull/1481>`_ - by :user:`gaborbernat` (`1003 <https://github.com/tox-dev/tox/issues/1003>`_) Features ^^^^^^^^ - add ``commands_pre`` and ``commands_post`` that run before and after running the ``commands`` (setup runs always, commands only if setup suceeds, teardown always - all run until the first failing command) - by :user:`gaborbernat` (`167 <https://github.com/tox-dev/tox/issues/167>`_) - ``pyproject.toml`` config support initially by just inline the tox.ini under ``tool.tox.legacy_tox_ini`` key; config source priority order is ``pyproject.toml``, ``tox.ini`` and then ``setup.cfg`` - by :user:`gaborbernat` (`814 <https://github.com/tox-dev/tox/issues/814>`_) - use the os environment variable ``TOX_SKIP_ENV`` to filter out tox environment names from the run list (set by ``envlist``) - by :user:`gaborbernat` (`824 <https://github.com/tox-dev/tox/issues/824>`_) - always set ``PIP_USER=0`` (do not install into the user site package, but inside the virtual environment created) and ``PIP_NO_DEPS=0`` (installing without dependencies can cause broken package installations) inside tox - by :user:`gaborbernat` (`838 <https://github.com/tox-dev/tox/issues/838>`_) - tox will inject some environment variables that to indicate a command is running within tox: ``TOX_WORK_DIR`` env var is set to the tox work directory, ``TOX_ENV_NAME`` is set to the current running tox environment name, ``TOX_ENV_DIR`` is set to the current tox environments working dir - by :user:`gaborbernat` (`847 <https://github.com/tox-dev/tox/issues/847>`_) - While running tox invokes various commands (such as building the package, pip installing dependencies and so on), these were printed in case they failed as Python arrays. Changed the representation to a shell command, allowing the users to quickly replicate/debug the failure on their own - by :user:`gaborbernat` (`851 <https://github.com/tox-dev/tox/issues/851>`_) - skip missing interpreters value from the config file can now be overridden via the ``--skip-missing-interpreters`` cli flag - by :user:`gaborbernat` (`903 <https://github.com/tox-dev/tox/issues/903>`_) - keep additional environments config order when listing them - by :user:`gaborbernat` (`921 <https://github.com/tox-dev/tox/issues/921>`_) - allow injecting config value inside the ini file dependent of the fact that we're connected to an interactive shell or not - by :user:`gaborbernat` (`947 <https://github.com/tox-dev/tox/issues/947>`_) - do not build sdist if skip install is specified for the envs to be run - by :user:`gaborbernat` (`974 <https://github.com/tox-dev/tox/issues/974>`_) - when verbosity level increases above two start passing through verbosity flags to pip - by :user:`gaborbernat` (`982 <https://github.com/tox-dev/tox/issues/982>`_) - when discovering the interpreter to use check if the tox host Python matches and use that if so - by :user:`gaborbernat` (`994 <https://github.com/tox-dev/tox/issues/994>`_) - ``-vv`` will print out why a virtual environment is re-created whenever this operation is triggered - by :user:`gaborbernat` (`1004 <https://github.com/tox-dev/tox/issues/1004>`_) Documentation ^^^^^^^^^^^^^ - clarify that ``python`` and ``pip`` refer to the virtual environments executable - by :user:`gaborbernat` (`305 <https://github.com/tox-dev/tox/issues/305>`_) - add Sphinx and mkdocs example of generating documentation via tox - by :user:`gaborbernat` (`374 <https://github.com/tox-dev/tox/issues/374>`_) - specify that ``setup.cfg`` tox configuration needs to be inside the ``tox:tox`` namespace - by :user:`gaborbernat` (`545 <https://github.com/tox-dev/tox/issues/545>`_) ``` ### 3.3.0 ``` ------------------ Bugfixes ^^^^^^^^ - fix ``TOX_LIMITED_SHEBANG`` when running under python3 - by :user:`asottile` (`931 <https://github.com/tox-dev/tox/issues/931>`_) Features ^^^^^^^^ - `PEP-517 <https://www.python.org/dev/peps/pep-0517/>`_ source distribution support (create a ``.package`` virtual environment to perform build operations inside) by :user:`gaborbernat` (`573 <https://github.com/tox-dev/tox/issues/573>`_) - `flit <https://flit.readthedocs.io>`_ support via implementing ``PEP-517`` by :user:`gaborbernat` (`820 <https://github.com/tox-dev/tox/issues/820>`_) - packaging now is exposed as a hook via ``tox_package(session, venv)`` - by :user:`gaborbernat` (`951 <https://github.com/tox-dev/tox/issues/951>`_) Miscellaneous ^^^^^^^^^^^^^ - Updated the VSTS build YAML to use the latest jobs and pools syntax - by :user:`davidstaheli` (`955 <https://github.com/tox-dev/tox/issues/955>`_) ``` ### 3.2.1 ``` ------------------ Bugfixes ^^^^^^^^ - ``--parallel--safe-build`` no longer cleans up its folders (``distdir``, ``distshare``, ``log``). - by :user:`gaborbernat` (`849 <https://github.com/tox-dev/tox/issues/849>`_) ``` ### 3.2.0 ``` ------------------ Features ^^^^^^^^ - Switch pip invocations to use the module ``-m pip`` instead of direct invocation. This could help avoid some of the shebang limitations. - by :user:`gaborbernat` (`935 <https://github.com/tox-dev/tox/issues/935>`_) - Ability to specify package requirements for the tox run via the ``tox.ini`` (``tox`` section under key ``requires`` - PEP-508 style): can be used to specify both plugin requirements or build dependencies. - by :user:`gaborbernat` (`783 <https://github.com/tox-dev/tox/issues/783>`_) - Allow to run multiple tox instances in parallel by providing the ``--parallel--safe-build`` flag. - by :user:`gaborbernat` (`849 <https://github.com/tox-dev/tox/issues/849>`_) ``` ### 3.1.3 ``` ------------------ Bugfixes ^^^^^^^^ - A caching issue that caused the ``develop-inst-nodeps`` action, which reinstalls the package under test, to always run has been resolved. The ``develop-inst-noop`` action, which, as the name suggests, is a no-op, will now run unless there are changes to ``setup.py`` or ``setup.cfg`` files that have not been reflected - by stephenfin (`909 <https://github.com/tox-dev/tox/issues/909>`_) Features ^^^^^^^^ - Python version testenvs are now automatically detected instead of comparing against a hard-coded list of supported versions. This enables ``py38`` and eventually ``py39`` / ``py40`` / etc. to work without requiring an upgrade to ``tox``. As such, the following public constants are now deprecated (and scheduled for removal in ``tox`` 4.0: ``CPYTHON_VERSION_TUPLES``, ``PYPY_VERSION_TUPLES``, ``OTHER_PYTHON_INTERPRETERS``, and ``DEFAULT_FACTORS`` - by :user:`asottile` (`914 <https://github.com/tox-dev/tox/issues/914>`_) Documentation ^^^^^^^^^^^^^ - Add a system overview section on the index page that explains briefly how tox works - by :user:`gaborbernat`. (`867 <https://github.com/tox-dev/tox/issues/867>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/tox - Changelog: https://pyup.io/changelogs/tox/ - Docs: https://tox.readthedocs.org/ </details>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Starting
tox -e py27andtox -e py36in parallel can cause the build to break as thesdistbuild is not thread-safe. We should use some lock to stop this from becoming an issue.The text was updated successfully, but these errors were encountered: