Make --parallel and --result-json work together #1159
Comments
|
yeah this is a known issue and limitation for now, thanks for raising it, we'll try to fix it but it's not trivial so don't expect it anytime soon though (3-4 months probably more realistic) |
|
To be fair I view this more a feature request though, as the parallel mode wanted to be a replacement of detox, and that did not have result json support. What's your reason for relying on the result JSON? (haven't had many people using it, so excited to find some actual use cases 😎) |
|
We basically run a bunch of our tests in parallel containers, saving the results json to a shared workspace, then we have a final job that runs that will list all environments from tox and make sure that we successfully executed all tests. This isn't a "did all envs pass" test, but a "did all envs run" test. Also, do you have any more details around why it doesn't work and why it isn't trivial? (or is there a place where that conversation already happened that you can link to?) |
|
I don't follow how the parallel flag and parallel containers work together, can you detail? And how did this work before the parallel flag? will lay out the details tomorrow when I'm by a proper keyboard. |
|
@gaborbernat sorry I meant to share more information but was stepping away from keyboard for a bit. Here is a typical CircleCI pipeline for us: https://circleci.com/workflow-run/5025703b-143d-4bc8-81be-f14baa3f1234 We run each of those integration tests via tox with Basically, each job runs it's own integration tests in our container, along with any additional containers needed for running that specific test (e.g. memcached, redis, etc), then Right now we are not running any environments in parallel via If there is a better way, I am all ears as well, since I mostly inherited this, but it does seem to "just work" :) If that makes sense? We do it this way to try and have our integration tests run in their own job, it is easier for us to maintain that way, and helps speed things up a bit to run these in parallel. For context, we have 720 tox environments in total.
|
|
The way parallel works now is the main tox instance spins up child tox instances to perform the child builds. At the moment there's no merge functionality implemented, so the parallel run would need to generate different reports and then merge it together (around here https://github.com/tox-dev/tox/blob/master/src/tox/session/commands/run/parallel.py#L88). I would say would not be really hard to do thinking more in detail about it, but someone would need to do it (and we have a few higher priority issues, our small team needs to focus). If you can make a PR that would be welcomed 👍 the sequential runs write the json here, we would probably want to move this into the sequential runner https://github.com/tox-dev/tox/blob/master/src/tox/session/__init__.py#L241 |
obestwalter
changed the title
When running Tox 3.7.0 with
--paralleland--result-jsontogether the result json file does not contain all the information from the test run.Example:
Our tox.ini
Not parallelize
/tmp/flask.results
Parallelize
/tmp/flask.results
Please let me know if there is anything else I can provide to help diagnose/resolve this!
The text was updated successfully, but these errors were encountered: