test: add --repeat option to tools/test.py

Manual backport from master to 4.x stream, original
commit message follows.

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Fedor Indutny <[email protected]>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <[email protected]>
Reviewed-By: joaocgreis - João Reis <[email protected]>
Reviewed-By: Johan Bergström <[email protected]>

Author: mhdawson

tools/test.py

@@ -774,8 +774,7 @@ def AddTestsToList(self, result, current_path, path, context, arch, mode):
       tests = self.GetConfiguration(context).ListTests(current_path, path,
                                                        arch, mode)
       for t in tests: t.variant_flags = v
-      result += tests
-
+      result += tests * context.repeat
 
   def GetTestStatus(self, context, sections, defs):
     self.GetConfiguration(context).GetTestStatus(sections, defs)
@@ -828,7 +827,8 @@ def GetTestStatus(self, context, sections, defs):
 class Context(object):
 
   def __init__(self, workspace, buildspace, verbose, vm, expect_fail,
-               timeout, processor, suppress_dialogs, store_unexpected_output):
+               timeout, processor, suppress_dialogs,
+               store_unexpected_output, repeat):
     self.workspace = workspace
     self.buildspace = buildspace
     self.verbose = verbose
@@ -838,6 +838,7 @@ def __init__(self, workspace, buildspace, verbose, vm, expect_fail,
     self.processor = processor
     self.suppress_dialogs = suppress_dialogs
     self.store_unexpected_output = store_unexpected_output
+    self.repeat = repeat
 
   def GetVm(self, arch, mode):
     if arch == 'none':
@@ -1369,6 +1370,9 @@ def BuildOptions():
       default="")
   result.add_option('--temp-dir',
       help='Optional path to change directory used for tests', default=False)
+  result.add_option('--repeat',
+      help='Number of times to repeat given tests',
+      default=1, type="int")
   return result
 
 
@@ -1533,7 +1537,8 @@ def Main():
                     options.timeout,
                     processor,
                     options.suppress_dialogs,
-                    options.store_unexpected_output)
+                    options.store_unexpected_output,
+                    options.repeat)
   # First build the required targets
   if not options.no_build:
     reqs = [ ]