QA-273: Feature/python test runner

README.md

@@ -234,6 +234,100 @@ if supported, set number of concurrent builds to `PARALLELISM`
 
 ## Testing
 
+`jenkins/helper/test_launch_controller.py` is used to control multiple test executions.
+
+### Its dependencies over stock python3 are:
+ - psutil to control subprocesses
+ - py7zr (optional) to build 7z reports instead of tar.bz2
+
+### It's reading these environment variables:
+- `INNERWORKDIR` - as the directory to place the report files
+- `WORKDIR` - used instead if `INNERWORKDIR` hasn't been set.
+- `TEMP` - temporary directory if not `INNERWORKDIR`/ArangoDB
+- `TMPDIR` and `TEMP` are passed to the executors.
+- `TSHARK` passed as value to `--sniffProgram`
+- `DUMPDEVICE` passed as value to `--sniffDevice`
+- `SKIPNONDETERMINISTIC` passed on as value to `--skipNondeterministic` to the testing.
+- `SKIPTIMECRITICAL` passed on as value to `--skipTimeCritical` to the testing.
+- `BUILDMODE` passed on as value to `--buildType` to the testing.
+- `DUMPAGENCYONERROR` passed on as value to `--dumpAgencyOnError` to the testing.
+- `PORTBASE` passed on as value to `--minPort` and `--maxPort` (+99) to the testing. Defaults to 7000
+- `SKIPGREY` passed on as value to `--skipGrey` to the testing.
+- `ONLYGREY` passed on as value to `--onlyGrey` to the testing.
+- `TIMELIMIT` is used to calculate the execution deadline starting point in time.
+- `COREDIR` the directory to locate coredumps for crashes
+- `LDAPHOST` to enable the tests with `ldap` flags.
+- any parameter in `test-definition.txt` that starts with a `$` is expanded to its value.
+
+### Its Parameters are:
+ - `PATH/test-definition.txt` - (first parameter) test definitions file from the arangodb source tree
+   (also used to locate the arangodb source)
+ - `-f` `[launch|dump]` use `dump` for syntax checking of `test-definition.txt` instead of executing the tests
+ - `--validate-only` don't run the tests
+ - `--help-flags` list the flags which can be used in `test-definition.txt`:
+    - `cluster`: this test requires a cluster
+    - `single`: this test requires a single server
+    - `full`: this test is only executed in full tests
+    - `!full`: this test is only executed in non-full tests
+    - `gtest`: only the gtests are to be executed
+    - `ldap`: ldap
+    - `enterprise`: this test is only executed with the enterprise version
+    - `!windows`: test is excluded from ps1 output
+ - `--cluster` filter `test-definition.txt` for all tests flagged as `cluster`
+ - `--full` - all tests including those flagged as `full` are executed.
+ - `--gtest` - only gtests are executed
+ - `--all` - output unfiltered
+
+### Syntax in `test-definition.txt`
+Lines consist of these parts:
+```
+testingJsSuiteName flags params suffix -- args to testing.js
+```
+where 
+- `flags` are listed above in `--help-flags`
+- params are:
+  - weight - sequence priority of test, 250 is the default.
+  - wweight - execution slots to book. defaults to 1, if cluster 4.
+  - buckets - split testcases to be launched in concurent chunks
+  Specifying a `*` in front of the number takes the default and multiplies it by the value.
+- suffix - if a testsuite is launched several times, make it distinguishable
+  like shell_aql => shell_aql_vst ; Bucket indexes are appended afterwards.
+- `--` literally the two dashes to split the line at.
+- `args to testing.js` - anything that `./scripts/unittest --help` would print you.
+
+### job scheduling
+To utilize all of the machines resources, tests can be run in parallel. The `execution_slots` are 
+set to the number of the physical cores of the machine (not threads).
+`wweight` is used to add the currently expected load by the tests to be no more than `execution_slots`.
+
+For managing each of these parallel executions of testing.js, worker threads are used. The workers
+themselves will spawn a set of I/O threads to capture the output of testing.js into a report file.
+
+The life cycle of a testrun will be as follows:
+
+ - the environment variable `TIMELIMIT` defines a *deadline* to all the tests, how much seconds should be allowed.
+ - tests are running in worker threads.
+ - main thread keeps control, launches more worker threads, once machine bandwith permits, but only every 5s as closest to not overwhelm the machine while launching arangods.
+ - tests themselves have their timeouts; `testing.js` will abort if they are reached.
+ - workers have a progressive timeout, if it doesn't hear back from `testing.js` for 999999999s it will hard kill and abort. [currently high / not used!]
+ - if workers have no output from `testing.js` they check whether the *deadline* is reached.
+ - if the *deadline* is reached, `SIG_INT`[* nix] / `SIG_BREAK`[windows] is sent to `testing.js` to trigger its *deadline* feature.
+ - the reached *deadline* will be indicated to the `testfailures.txt` report file and the logfile of the test in question.
+ - with *deadline* engageged, `testing.js` can send no more subsequent requests, nor spawn processes => eventually testing will abort.
+ - force shutdown of arangod Instances will reset the deadline, SIG_ABRT arangods, and try to do core dump analysis.
+ - workers continue reading pipes from `testing.js`, but once no chars are comming, `waitpid()` checks with a 1s timout whether `testing.js` is done and exited.
+ - if the worker reaches `180` counters of `waitpid()` invocations it will give up. It will hard kill `testing.js` and all other child processes it can find.
+ - this should unblock the workers I/O threads, and they should exit.
+ - the `waitpid()` on `testing.js` should exit, I/O threads should be joined, results should be passed up to the main thread.
+ - so the workers still have a slugish interpretation of the *deadline*, giving them the chance to collect as much knowledge about the test execution as posible.
+ - meanwhile the main thread has a *fixed* deadline: 5 minutes after the `TIMELIMIT` is reached.
+ - if not all workers have indicated their exit before this final deadline:
+   - the main thread will start killing any subprocesses of itself which it finds.
+   - after this wait another 20s, to see whether the workers may have been unblocked by the killing
+ - if not, it shouts "Geronimoooo" and takes the big shotgun, and force-terminates the python process which is running it. This will kill all threads as well and terminate the process.
+ - if all workers have indicated their exit in time, their threads will be joined.
+ - reports will be generated.
+
 ## Packaging
 
     makeRelease

containers/buildUbuntu3.docker/Dockerfile

@@ -7,13 +7,13 @@ ENV COMPILER_VERSION 9
 
 ENV CLANG_VERSION 12
 
-RUN apt-get update && \
+RUN apt-get update --fix-missing && \
     apt-get install -y software-properties-common && \
     add-apt-repository -y ppa:ubuntu-toolchain-r/test && \
     apt-get update && \
-    apt-get upgrade -y
-
-RUN apt-get install -y build-essential gcc-${COMPILER_VERSION} g++-${COMPILER_VERSION} cmake make bison flex python python3-distutils ccache git libjemalloc-dev vim exuberant-ctags gdb fish ruby ruby-httparty ruby-rspec psmisc sudo debhelper debconf jq wget libdb-dev curl gnupg2 gcovr prometheus && gem install persistent_httparty && apt-get clean
+    apt-get upgrade -y && \
+    apt-get install -y build-essential gcc-${COMPILER_VERSION} g++-${COMPILER_VERSION} cmake make bison flex python3-psutil python3-distutils ccache git libjemalloc-dev vim exuberant-ctags gdb fish psmisc sudo debhelper debconf jq wget libdb-dev curl gnupg2 gcovr prometheus &&\
+    apt-get clean
 
 RUN curl -L https://github.com/mozilla/sccache/releases/download/v0.2.15/sccache-v0.2.15-${ARCH}-unknown-linux-musl.tar.gz | tar xvz -C /tmp && mv /tmp/sccache-v0.2.15-${ARCH}-unknown-linux-musl/sccache /usr/bin/sccache && chmod +x /usr/bin/sccache && rm -rf /tmp/sccache-v0.2.15-${ARCH}-unknown-linux-musl
 

containers/buildUbuntu4.docker/Dockerfile

@@ -1,3 +1,4 @@
+
 FROM ubuntu:20.04
 MAINTAINER Max Neunhoeffer <[email protected]>
 
@@ -7,12 +8,12 @@ ENV COMPILER_VERSION 9
 
 ENV CLANG_VERSION 12
 
-RUN apt-get update && \
+RUN apt-get update --fix-missing && \
     apt-get install -y software-properties-common && \
     apt-get update && \
     apt-get upgrade -y
 
-RUN apt-get install -y build-essential gcc-${COMPILER_VERSION} g++-${COMPILER_VERSION} cmake make bison flex python ccache git libjemalloc-dev vim exuberant-ctags gdb fish ruby ruby-httparty ruby-rspec psmisc sudo debhelper debconf jq wget libdb-dev curl gnupg2 gcovr prometheus && gem install persistent_httparty && apt-get clean
+RUN apt-get install -y build-essential gcc-${COMPILER_VERSION} g++-${COMPILER_VERSION} cmake make bison flex python ccache git libjemalloc-dev vim exuberant-ctags gdb fish psmisc sudo debhelper debconf jq wget libdb-dev curl gnupg2 gcovr prometheus && apt-get clean
 
 RUN curl -L https://github.com/mozilla/sccache/releases/download/v0.2.15/sccache-v0.2.15-${ARCH}-unknown-linux-musl.tar.gz | tar xvz -C /tmp && mv /tmp/sccache-v0.2.15-${ARCH}-unknown-linux-musl/sccache /usr/bin/sccache && chmod +x /usr/bin/sccache && rm -rf /tmp/sccache-v0.2.15-${ARCH}-unknown-linux-musl
 
@@ -32,4 +33,6 @@ RUN git config --global --add safe.directory /work/ArangoDB
 RUN git config --global --add safe.directory /work/ArangoDB/enterprise
 RUN git config --global --add safe.directory /work/ArangoDB/docs
 
+RUN apt-get install -y python3-pip && pip3 install py7zr psutil 
+
 CMD [ "/usr/bin/fish" ]

containers/buildUbuntu5.docker/Dockerfile

@@ -7,12 +7,12 @@ ENV COMPILER_VERSION 10
 
 ENV CLANG_VERSION 12
 
-RUN apt-get update && \
+RUN apt-get update --fix-missing && \
     apt-get install -y software-properties-common && \
     apt-get update && \
     apt-get upgrade -y
 
-RUN apt-get install -y build-essential gcc-${COMPILER_VERSION} g++-${COMPILER_VERSION} cmake make bison flex python ccache git libjemalloc-dev vim exuberant-ctags gdb fish ruby ruby-httparty ruby-rspec psmisc sudo debhelper debconf jq wget libdb-dev curl gnupg2 gcovr prometheus && gem install persistent_httparty && apt-get clean
+RUN apt-get install -y build-essential gcc-${COMPILER_VERSION} g++-${COMPILER_VERSION} cmake make bison flex python ccache git libjemalloc-dev vim exuberant-ctags gdb fish psmisc sudo debhelper debconf jq wget libdb-dev curl gnupg2 gcovr prometheus && apt-get clean
 
 RUN curl -L https://github.com/mozilla/sccache/releases/download/v0.2.15/sccache-v0.2.15-${ARCH}-unknown-linux-musl.tar.gz | tar xvz -C /tmp && mv /tmp/sccache-v0.2.15-${ARCH}-unknown-linux-musl/sccache /usr/bin/sccache && chmod +x /usr/bin/sccache && rm -rf /tmp/sccache-v0.2.15-${ARCH}-unknown-linux-musl
 
@@ -37,4 +37,6 @@ RUN git config --global --add safe.directory /work/ArangoDB/3rdParty/iresearch
 RUN git config --global --add safe.directory /work/ArangoDB/3rdParty/rocksdb
 RUN git config --global --add safe.directory /work/ArangoDB/3rdParty/velocypack
 
+RUN apt-get install -y python3-pip && pip3 install py7zr psutil 
+
 CMD [ "/usr/bin/fish" ]

containers/buildUbuntu6.docker/Dockerfile

@@ -7,7 +7,7 @@ ENV COMPILER_VERSION 11
 
 ENV CLANG_VERSION 14
 
-RUN apt-get update && \
+RUN apt-get update --fix-missing && \
     apt-get install -y software-properties-common && \
     apt-get update && \
     apt-get upgrade -y
@@ -39,4 +39,6 @@ RUN git config --global --add safe.directory /work/ArangoDB/3rdParty/iresearch
 RUN git config --global --add safe.directory /work/ArangoDB/3rdParty/rocksdb
 RUN git config --global --add safe.directory /work/ArangoDB/3rdParty/velocypack
 
+RUN apt-get install -y python3-pip && pip3 install py7zr psutil
+
 CMD [ "/usr/bin/fish" ]

helper.fish

@@ -95,7 +95,8 @@ end
 function single ; set -gx TESTSUITE single ; end
 function cluster ; set -gx TESTSUITE cluster ; end
 function resilience ; set -gx TESTSUITE resilience ; end
-function catchtest ; set -gx TESTSUITE catchtest ; end
+function catchtest ; set -gx TESTSUITE gtest ; end
+function gtest ; set -gx TESTSUITE gtest ; end
 if test -z "$TESTSUITE" ; cluster
 else ; set -gx TESTSUITE $TESTSUITE ; end
 
@@ -1484,7 +1485,7 @@ function showConfig
   printf $fmt3 'SkipGrey'       $SKIPGREY      '(skipGrey/includeGrey)'
   printf $fmt3 'OnlyGrey'       $ONLYGREY      '(onlyGreyOn/onlyGreyOff)'
   printf $fmt3 'Storage engine' $STORAGEENGINE '(mmfiles/rocksdb)'
-  printf $fmt3 'Test suite'     $TESTSUITE     '(single/cluster/resilience/catchtest)'
+  printf $fmt3 'Test suite'     $TESTSUITE     '(single/cluster/resilience/gtest)'
   printf $fmt2 'Log Levels'     (echo $LOG_LEVELS)
   echo
   echo 'Package Configuration'
@@ -1943,7 +1944,7 @@ function moveResultsToWorkspace
       mv $WORKDIR/work/coverage $WORKSPACE
     end
 
-    set -l matches $WORKDIR/work/*.{asc,deb,dmg,rpm,tar.gz,tar.bz2,zip,html,csv}
+    set -l matches $WORKDIR/work/*.{asc,testfailures.txt,deb,dmg,rpm,7z,tar.gz,tar.bz2,zip,html,csv}
     for f in $matches
       echo $f | grep -qv testreport ; and echo "mv $f" ; and mv $f $WORKSPACE; or echo "skipping $f"
     end

helper.linux.fish

@@ -15,19 +15,19 @@ set -gx ARCH (uname -m)
 set IMAGE_ARGS "--build-arg ARCH=$ARCH"
 
 set -gx UBUNTUBUILDIMAGE3_NAME arangodb/ubuntubuildarangodb3-$ARCH
-set -gx UBUNTUBUILDIMAGE3_TAG 16
+set -gx UBUNTUBUILDIMAGE3_TAG 17
 set -gx UBUNTUBUILDIMAGE3 $UBUNTUBUILDIMAGE3_NAME:$UBUNTUBUILDIMAGE3_TAG
 
 set -gx UBUNTUBUILDIMAGE4_NAME arangodb/ubuntubuildarangodb4-$ARCH
-set -gx UBUNTUBUILDIMAGE4_TAG 17
+set -gx UBUNTUBUILDIMAGE4_TAG 18
 set -gx UBUNTUBUILDIMAGE4 $UBUNTUBUILDIMAGE4_NAME:$UBUNTUBUILDIMAGE4_TAG
 
 set -gx UBUNTUBUILDIMAGE5_NAME arangodb/ubuntubuildarangodb5-$ARCH
-set -gx UBUNTUBUILDIMAGE5_TAG 10
+set -gx UBUNTUBUILDIMAGE5_TAG 11
 set -gx UBUNTUBUILDIMAGE5 $UBUNTUBUILDIMAGE5_NAME:$UBUNTUBUILDIMAGE5_TAG
 
 set -gx UBUNTUBUILDIMAGE6_NAME arangodb/ubuntubuildarangodb6-$ARCH
-set -gx UBUNTUBUILDIMAGE6_TAG 2
+set -gx UBUNTUBUILDIMAGE6_TAG 3
 set -gx UBUNTUBUILDIMAGE6 $UBUNTUBUILDIMAGE6_NAME:$UBUNTUBUILDIMAGE6_TAG
 
 set -gx UBUNTUPACKAGINGIMAGE arangodb/ubuntupackagearangodb-$ARCH:1
@@ -454,11 +454,11 @@ function oskar
   and if test "$SAN" = "On"
     parallelism 2
     clearSanStatus
-    runInContainer --cap-add SYS_NICE --cap-add SYS_PTRACE (findBuildImage) $SCRIPTSDIR/runTests.fish $argv
+    runInContainer --security-opt seccomp=unconfined --cap-add SYS_NICE --cap-add SYS_PTRACE (findBuildImage) $SCRIPTSDIR/runTests.fish $argv
     set s $status
     set s (math $s + (getSanStatus))
   else
-    runInContainer --cap-add SYS_NICE (findBuildImage) $SCRIPTSDIR/runTests.fish $argv
+    runInContainer --security-opt seccomp=unconfined --cap-add SYS_NICE (findBuildImage) $SCRIPTSDIR/runTests.fish $argv
     set s $status
   end
 

helper.psm1

@@ -42,6 +42,7 @@ Else
         $global:TSHARK = ""
   }
 }
+$ENV:TSHARK=$global:TSHARK
 
 $global:HANDLE_EXE = $null
 If (Get-Command handle.exe -ErrorAction SilentlyContinue)
@@ -65,7 +66,7 @@ Else
 {
   Remove-Item "$global:COREDIR\*" -Recurse -Force
 }
-$global:RUBY = (Get-Command ruby.exe).Path
+$env:COREDIR=$global:COREDIR
 $global:INNERWORKDIR = "$WORKDIR\work"
 $global:ARANGODIR = "$INNERWORKDIR\ArangoDB"
 $global:ENTERPRISEDIR = "$global:ARANGODIR\enterprise"
@@ -582,6 +583,11 @@ Function showConfig
     Write-Host "------------------------------------------------------------------------------"
     Write-Host "Cache Statistics"
     showCacheStats
+    $ENV:SKIPNONDETERMINISTIC = $SKIPNONDETERMINISTIC
+    $ENV:SKIPTIMECRITICAL = $SKIPTIMECRITICAL
+    $ENV:SKIPGREY = $SKIPGREY
+    $ENV:ONLYGREY = $ONLYGREY
+    $ENV:BUILDMODE = $BUILDMODE
     comm
 }
 
@@ -600,7 +606,7 @@ Function resilience
 Function catchtest
 {
     $global:TESTSUITE = "catchtest"
-    $global:TESTSUITE_TIMEOUT = 1800
+    $global:TIMELIMIT = 1800
 }
 If (-Not($TESTSUITE))
 {

jenkins/helper/allure_commons/_allure.py

@@ -0,0 +1,2 @@
+def attach(**kwargs):
+    return

jenkins/helper/asciiprint.py

@@ -0,0 +1,79 @@
+#!env python
+""" removes terminal control sequences and other non ascii characters """
+import unicodedata
+import re
+import sys
+
+is_tty = sys.stdout.isatty()
+PROGRESS_COUNT = 0
+
+# 7-bit C1 ANSI sequences
+ANSI_ESCAPE_B = re.compile(
+    rb"""
+    \x1B  # ESC
+    \xE2  # throbber...
+    \xA0  # throbber...
+    \xA7  # throbber...
+    \x8F  # throbber...
+    \r    # cariage return
+    (?:   # 7-bit C1 Fe (except CSI)
+        [@-Z\\-_]
+    |     # or [ for CSI, followed by a control sequence
+        \[
+        [0-?]*  # Parameter bytes
+        [ -/]*  # Intermediate bytes
+        [@-~]   # Final byte
+    )
+""",
+    re.VERBOSE,
+)
+
+
+def ascii_convert(the_bytes: bytes):
+    """convert string to only be ascii without control sequences"""
+    return ANSI_ESCAPE_B.sub(rb"", the_bytes).decode("utf-8")
+
+
+# 7-bit C1 ANSI sequences
+ANSI_ESCAPE = re.compile(
+    r"""
+    \x1B  # ESC
+    \xE2  # throbber...
+    \xA0  # throbber...
+    \xA7  # throbber...
+    \x8F  # throbber...
+    \r    # cariage return
+    (?:   # 7-bit C1 Fe (except CSI)
+        [@-Z\\-_]
+    |     # or [ for CSI, followed by a control sequence
+        \[
+        [0-?]*  # Parameter bytes
+        [ -/]*  # Intermediate bytes
+        [@-~]   # Final byte
+    )
+""",
+    re.VERBOSE,
+)
+
+
+def ascii_convert_str(the_str: str):
+    """convert string to only be ascii without control sequences"""
+    return ANSI_ESCAPE.sub(rb"", the_str)
+
+
+def ascii_print(string):
+    """convert string to only be ascii without control sequences"""
+    string = ANSI_ESCAPE.sub("", string)
+    print("".join(ch for ch in string if ch == "\n" or unicodedata.category(ch)[0] != "C"))
+
+
+def print_progress(char):
+    """print a throbber alike that immediately is sent to the console"""
+    # pylint: disable=global-statement
+    global PROGRESS_COUNT
+    print(char, end="")
+    PROGRESS_COUNT += 1
+    if not is_tty and PROGRESS_COUNT % 10 == 0:
+        # add a linebreak so we see something in jenkins (if):
+        print("\n")
+    sys.stdout.flush()