Feature/python test runner (#419)

* start implementing a python launch controller

* make it work for the first time

* try launching outside of oskar.

* no more pipes needed

* adjust report directory

* fix paths, thread naming.

* fallback if no env is configured

* lint

* more work on cluster etc

* silence, proper error message for missing variable

* convert params

* lint

* fix slot

* fix arangosh.conf, launching of subsequent testruns

* try to launch it from fish

* implement 7zip

* add modules to the docker container

* more printing

* fix handling

* Add pip3

* Fix typo

* Typo 2

* handle INNERWORKDIR

* fix missing line break

* export settings

* fix typo

* on windows skip !windows tests

* lint, refactor, simplify

* install 7z

* export core directory

* work on fish integration

* similarize for new python job scheduler

* work on reprot generating

* try to implement timeout

* also upload 7z and txt

* also upload 7z and txt

* fix deadline

* fix workspace handling

* fix temporary directory handling

* make sure out temp directory exists

* RTFM fail

* don't put it to the workspace

* implement gtest invoking

* cleanup

* sort, lint

* prefer INNERWORKDIR

* implement writing test.log

* implement html report

* bring back function deletet to early

* install the windows boomerang handler on top level

* fix include

* fix reference

* print before killing shit

* work on timeout

* finish deadline handling, rename script

* fix exit code handling

* lint

* thanks @mpoeter for ps aid

* make the thread identifier the test plus a growing number

* implement central final deadline, which will kick in after 2 minutes

* remove debug output

* use /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin to locate python

* wintendo next try

* wintendo next try

* wintendo go home

* fix calculation of hard time limit

* make sure nobody changes the exit code to good

* add monkey patches

* cleanup deadline

* ignore exceptions if no process is there

* deadline handling: prioritize incomming lines over timeout counting

* fix directory handling

* work on result presentation

* cleanup

* let the file remain open for further info

* fix environment variable handling

* documentation

* fix port handling

* work on deadline

* fix hard deadline handling

* make it 20s

* need more time

* list processes so we may guess whats actually going on

* kill all, then waitpid all

* make threads provide half a slot.

* be sure to catch

* resume just in case, then kill

* resume just in case, then kill

* ignore resume errors

* increase volume

* lint

* lint

* catch more

* add multipliers

* more load, print load avg

* fix sorting by prio - biggest values first

* cleanup crash report for size

* if test indicates its been crashing create report as well.

* more threat to the machine.

* timeout

* fix typo

* delete tzdata subdir first

* use load and sockets for throttle control

* install required python libs

* only see for load [0, 1]

* increase container version

* anounce deadline at start

* don't print to logfile

* give better feedback if arangosh fails to launch in first place, thangs @maierlars for bringing up the topic

* Update helper.linux.fish

* tschuess ruby

* re-sync to be stock RTA

* fix container numbers, adjust #3

* sync to rta

* resync

* this is not needed anymore

* add --fix-missing

* fresh python?

* revert to tar.gz

* chaos tests in nightlies demand for longer timeouts, since tests run longer.

* Update README.md

Co-authored-by: Jan <[email protected]>

* Update README.md

Co-authored-by: Jan <[email protected]>

* Update README.md

Co-authored-by: Jan <[email protected]>

* Update README.md

Co-authored-by: Jan <[email protected]>

* Update README.md

Co-authored-by: Jan <[email protected]>

* remove more old stuff

* ignore encoding errors

* increase timeout to hard self kill

* switch to one environment variable name

* env

* limit the amount of coredumps

* ignore access denied to open sockets

* if we need to wait for the system to cool down on start...

* make sure we don't come back good if nothing launched at all

* them tiny boxes need more time

* need more time

* add deadline status to testfailurs.txt

* need more time

* beautify testfailures.txt

* give machine estimate reasons at the start of the run

* case may matter

* one more environment variable

* anounce test directory

* switch sequence, print first

* one more var exported

* add disk i/o to the output

* better work with M1 performance cores

* print other sequence; enable more load[1]

* more threads doesn't cut it

* print platform

* precise M1 detection

* two places on mac to collect cores

* properly append

* fix default directory

* use iso-ish datetime format for filenames

Co-authored-by: Vadim <[email protected]>
Co-authored-by: Jan <[email protected]>
Co-authored-by: Markus Pfeiffer <[email protected]>

Author: 4 people

jenkins/helper/test_launch_controller.py

@@ -2,9 +2,9 @@
 """ read test definition, and generate the output for the specified target """
 import argparse
 from datetime import datetime, timedelta
+import platform
 import os
 from pathlib import Path
-import platform
 import pprint
 import signal
 import sys
@@ -37,6 +37,14 @@
 
 IS_WINDOWS = platform.win32_ver()[0] != ""
 IS_MAC = platform.mac_ver()[0] != ""
+if IS_MAC:
+    # Put us to the performance cores:
+    # https://apple.stackexchange.com/questions/443713
+    from os import setpriority
+    PRIO_DARWIN_THREAD  = 0b0011
+    PRIO_DARWIN_PROCESS = 0b0100
+    PRIO_DARWIN_BG      = 0x1000
+    setpriority(PRIO_DARWIN_PROCESS, 0, 0)
 
 pp = pprint.PrettyPrinter(indent=4)
 
@@ -309,9 +317,22 @@ def __init__(self, definition_file):
             self.timeout = int(os.environ['timeLimit'.upper()])
         elif 'timeLimit' in os.environ:
             self.timeout = int(os.environ['timeLimit'])
-        if psutil.cpu_count() <= 8:
+
+        if psutil.cpu_count(logical=False) <= 8:
             print("Small machine detected, quadrupling deadline!")
             self.timeout *= 4
+        self.no_threads = psutil.cpu_count()
+        self.available_slots = round(self.no_threads * 2) #logical=False)
+        if IS_MAC and platform.processor() == "arm" and psutil.cpu_count() == 8:
+            self.no_threads = 6 # M1 only has 4 performance cores
+            self.available_slots = 10
+        if IS_WINDOWS:
+            self.max_load = 0.85
+            self.max_load1 = 0.75
+        else:
+            self.max_load = self.no_threads * 0.9
+            self.max_load1 = self.no_threads * 0.95
+
         self.deadline = datetime.now() + timedelta(seconds=self.timeout)
         self.hard_deadline = datetime.now() + timedelta(seconds=self.timeout + 660)
         if definition_file.is_file():
@@ -322,21 +343,14 @@ def __init__(self, definition_file):
             for target in ['RelWithdebInfo', 'Debug']:
                 if (bin_dir / target).exists():
                     bin_dir = bin_dir / target
-        self.no_threads = psutil.cpu_count()
-        self.available_slots = round(self.no_threads * 2) #logical=False)
-        if IS_WINDOWS:
-            self.max_load = 0.85
-            self.max_load1 = 0.75
-        else:
-            self.max_load = self.no_threads * 0.9
-            self.max_load1 = self.no_threads * 0.9
-        # self.available_slots += (psutil.cpu_count(logical=True) - self.available_slots) / 2
         print(f"""Machine Info:
  - {psutil.cpu_count(logical=False)} Cores / {psutil.cpu_count(logical=True)} Threads
+ - {platform.processor()} processor architecture
  - {psutil.virtual_memory()} virtual Memory
  - {self.max_load} / {self.max_load1} configured maximum load 0 / 1
  - {self.available_slots} test slots
  - {str(TEMP)} - temporary directory
+ - current Disk I/O: {str(psutil.disk_io_counters())}
 """)
         self.cfgdir = base_source_dir / 'etc' / 'relative'
         self.bin_dir = bin_dir
@@ -426,13 +440,15 @@ def __init__(self, cfg):
         self.success = True
         self.crashed = False
         self.cluster = False
+        self.datetime_format = "%Y-%m-%dT%H%M%SZ"
 
     def print_active(self):
         """ output currently active testsuites """
         with self.slot_lock:
-            print("Running: " + str(self.running_suites) +
+            print(str(psutil.getloadavg()) + "<= Load " +
+                  "Running: " + str(self.running_suites) +
                   " => Active Slots: " + str(self.used_slots) +
-                  " => Load: " + str(psutil.getloadavg()))
+                  " => Disk I/O: " + str(psutil.disk_io_counters()))
         sys.stdout.flush()
 
     def done_job(self, parallelity):
@@ -454,7 +470,8 @@ def launch_next(self, offset, counter):
         load = psutil.getloadavg()
         if ((load[0] > self.cfg.max_load) or
             (load[1] > self.cfg.max_load1)):
-            print(F"Load to high: {str(load)} waiting before spawning more")
+            print(F"{str(load)} <= Load to high; waiting before spawning more - Disk I/O: " +
+                  str(psutil.disk_io_counters()))
             return False
         with self.slot_lock:
             self.used_slots += self.scenarios[offset].parallelity
@@ -617,13 +634,16 @@ def generate_crash_report(self):
             core_max_count = 15 # 3 cluster instances
         core_dir = Path.cwd()
         core_pattern = "core*"
+        if IS_WINDOWS:
+            core_pattern = "*.dmp"
+        system_corefiles = []
         if 'COREDIR' in os.environ:
             core_dir = Path(os.environ['COREDIR'])
+        else:
+            core_dir = Path('/var/tmp/') # default to coreDirectory in testing.js
         if IS_MAC:
-            core_dir = Path('/cores')
-        if IS_WINDOWS:
-            core_pattern = "*.dmp"
-        files = sorted(core_dir.glob(core_pattern))
+            system_corefiles = sorted(Path('/cores').glob(core_pattern))
+        files = sorted(core_dir.glob(core_pattern)) + system_corefiles
         if len(files) > core_max_count:
             count = 0
             for one_crash_file in files:
@@ -633,7 +653,7 @@ def generate_crash_report(self):
                     one_crash_file.unlink(missing_ok=True)
         is_empty = len(files) == 0
         if self.crashed or not is_empty:
-            crash_report_file = get_workspace() / datetime.now(tz=None).strftime("crashreport-%d-%b-%YT%H.%M.%SZ")
+            crash_report_file = get_workspace() / datetime.now(tz=None).strftime(f"crashreport-{self.datetime_format}")
             print("creating crashreport: " + str(crash_report_file))
             sys.stdout.flush()
             shutil.make_archive(str(crash_report_file),
@@ -642,7 +662,7 @@ def generate_crash_report(self):
                                 core_dir.name,
                                 True)
             self.cleanup_unneeded_binary_files()
-            binary_report_file = get_workspace() / datetime.now(tz=None).strftime("binaries-%d-%b-%YT%H.%M.%SZ")
+            binary_report_file = get_workspace() / datetime.now(tz=None).strftime(f"binaries-{self.datetime_format}")
             print("creating crashreport binary support zip: " + str(binary_report_file))
             sys.stdout.flush()
             shutil.make_archive(str(binary_report_file),
@@ -657,7 +677,7 @@ def generate_crash_report(self):
 
     def generate_test_report(self):
         """ regular testresults zip """
-        tarfile = get_workspace() / datetime.now(tz=None).strftime("testreport-%d-%b-%YT%H.%M.%SZ")
+        tarfile = get_workspace() / datetime.now(tz=None).strftime(f"testreport-{self.datetime_format}")
         print("Creating " + str(tarfile))
         sys.stdout.flush()
         shutil.make_archive(self.cfg.run_root / 'innerlogs',