Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] Maintain separate tox --notest and tox calls in CIs #38

Closed
webknjaz opened this issue Dec 31, 2020 · 0 comments
Closed

[FR] Maintain separate tox --notest and tox calls in CIs #38

webknjaz opened this issue Dec 31, 2020 · 0 comments

Comments

@webknjaz
Copy link
Contributor

webknjaz commented Dec 31, 2020

Follow-up for https://github.com/pypa/setuptools/pull/2486/files#r546904721

Many CI setups do this nowadays and I consider it the best practice.

  • CIs usually dump a lot of logs together but they also have a concept of steps. When looking at logs pages, steps are normally collapsed and it's easy to navigate to the step of interest with a single click. They are mostly mapped to shell commands but what if one command encapsulates several high-level actions. It's preferable to separate those actions as different steps in CI setups so that things like runnings tests and setting up the environment/deps would not show up together. Moreover, some CIs even differentiate types of failures during the install and test stages (Travis CI shows FAILURE in one case and ERROR in the other, for example).
    When opening the testing step, it's harder to visually find the useful part if it starts with the pip install logs. This affects the CI UX.
    Plus, CIs usually have per-step timing info which is quite helpful when it comes to figuring out slow things.
  • In the past, there was a nasty tox bug that looked like the test invocation was failing, hiding that the real reason was the install step. Making it separate allowed to actually identify what the problem was.
  • This is an idiom that the tox maintainers themselves follow: https://github.com/tox-dev/azure-pipelines-template/blob/30f07f8/run-tox-env.yml#L143. I may have started this tradition, though.
jaraco added a commit that referenced this issue Jan 9, 2021
@jaraco jaraco closed this as completed Jan 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants