Cancelling after a predefined number of failed tests

[Evangelink]

Summary

It could be interesting to automatically cancel test suite execution after a specified number of tests have failed.

Background and Motivation

It might not always be useful to run all tests and "waste" computation time. It would be interesting to have some option to allow cancelling the full test application after a specified number of tests have failed.

Proposed Feature

Introduce a built-in feature that will cancel test execution after a specified number of tests have failed. Option could be called --max-failed-test or --fail-limit.

Alternative Designs

None

[MarcoRossignoli]

Is something you saw somewhere?
I wonder in which scenario could be useful.

[Evangelink]

Is something you saw somewhere?

Some javascript and some python test tools :) VS has an option stop on first error so I guess some people use that for inner loop development.

[MarcoRossignoli]

On first error I think make sense...specify a number on an expecting growing tests suite is not easy, I mean decide the number and keep it up to date.

[cremor]

@MarcoRossignoli Why would you need to keep the number "up to date"?

I would just set that new number to something around 20.
If fewer tests fail then I want all fails with details. Because I might have broken multiple things.
If more tests fail then any additional failing tests above that number will not give me more information because I'll never check the details of all of them. Instead I will always just check a few, fix them, and then run all tests again.

That stays true regardless of the number of tests in the project/solution. The number is "how many test results would I look at in detail".

[MarcoRossignoli]

I would just set that new number to something around 20.

It's not fully clear to me how a user can decide this number, if I have 100k or tests 20 is really small if I have 100 tests it's big...so the fact that this number doesn't distinguish the name or the logical group name of the tests(you could fail 2 tests with a dataset attribute of 10 cases so at the end you ran 2 tests) I wonder why 20 is different than 1. What's the goal? Stop soon as possible or? What if actually we stop at 20 but tests would fail are 100? So you fail to 20 you fix that 20 and maybe other 50 will be out, you re-run and you fail again.

What's the suppose benefit to specify something > 1 for a "real"(big) test suite? Maybe make sense but I'd like to understand how the feature will help.

[cremor]

It's not fully clear to me how a user can decide this number

It's the number of fails over which the number itself doesn't matter any more.
In other words: It's the maximum number where it makes at least a little bit of sense to check the test results of all of them.

What's the suppose benefit to specify something > 1 for a "real"(big) test suite?

If only a few tests fail then I want to check all of them individually before running all tests again.
If I added two bugs, which cause a few test fails, then I can see and fix both bugs before running all tests again.

Maybe it's me, but that's how I work. When I execute all tests locally I'll just stop the execution after many tests failed. But as long as the fail number stays low, I keep it running.
And when the tests are executed on a build server then I wouldn't want to block it unnecessary with running tests which I'll never check anyway in the case with many fails. But again, with only a few fails then I want the information that exactly those tests failed.

[MarcoRossignoli]

Maybe it's me, but that's how I work. When I execute all tests locally I'll just stop the execution after many tests failed. But as long as the fail number stays low, I keep it running.

Got it, this make sense it's interacting usage and restart the tests is "quick", in case of CI means repush and wait for the CI to reach that point, looks slow.

In other words: It's the maximum number where it makes at least a little bit of sense to check the test results of all of them.

Yeah this was clear, I was thinking that "a little bit of a sense" could be related to the amount of tests in the suite.

Anyway if we decide to have the number of tests I won't oppose if also @Evangelink is in agreement with the usage/scenario, it could have a usage and anyway cover the 1 test failure too.

[nohwnd]

To me it also makes sense locally, if I run my tests from commandline (which I don't prefer if there is a better UI in editor), it that case I don't get 600 test failures, that I won't be able to fix in that single run, and I would prefer to see 1-20 fails, so I can gradually fix them without for the whole suite to finish every time.

On server I would find this option useful only when there is a threshold for tests that fail with the same error. E.g. 100 tests failing with "Cannot find nuget package xxx, aborting", but this exact example feels like approaching the problem from the wrong direction, instead the test should be able to return a Fatal result when common dependency is not preset, which will stop the run.

But maybe there are other usecases where this would make sense on server as well.

[cremor]

The use case on the server for me is to reduce "wasted" agent run time. My company only has a limited amount of parallel builds. Often multiple builds are queued which would get started earlier if the run with many failures would stop earlier.

[Evangelink]

Some links of the same feature in different languages:

• Jest: https://jestjs.io/docs/cli#--bailn
• Ward: https://ward.readthedocs.io/en/latest/guide/running_tests.html#cancelling-after-a-number-of-failures-with-fail-limit