Reworks container support (#741)

This PR reworks the tool-suites container support.
The main changes are:
 * Base images are now separated into stages. This allows rebuilding only certain stages.
 * vara-container build has additional flags for updating only varats, the research tool, or the config.
 * Installing research tools into a container is handled by a new protocol
 * vara-container prepare-slurm has been removed. Its functionality is now handled by vara-run. As a side-effect, this does no longer require persistent changes to the benchbuild config.
 * node_dir cleanup in the slurm script template has (hopefully) been fixed
 * Container-related config has been updated.
 * Python 3.10 in the Debian 10 base image

Author: boehmseb

docs/source/tutorials/container_guide.rst

@@ -3,8 +3,16 @@ Container Guide
 
 VaRA-TS supports running experiments inside containers to make them more portable and reproducible.
 This guide walks you through the steps how to prepare and execute experiments in containers and assumes that your system has already been set up for executing containers.
-You can find the instructions for setting up containers on your system :ref:`here <Running BenchBuild in a Container>`.
-More details about the inner workings of the container support in VaRA-TS can be found :ref:`here <Container support>`.
+For additional resources, see the following pages:
+
+- :ref:`Running BenchBuild in a Container` explains how to initially set up containers on your system and how they work with BenchBuild under the hood.
+- :ref:`Container support` explains how VaRA-TS creates and manages reusable base images that can be used by projects.
+- :ref:`vara-container` is the tool that is used for managing these base images.
+- :ref:`Using Containers` explains how container support can be implemented for a project.
+- :ref:`Slurm and Container` demonstrates the container support integration with slurm.
+
+
+
 Note that, by default, the most recent version of VaRA-TS available from PyPI will be used inside the container.
 If you want to use your local development version of VaRA-TS instead, you have to specify this in the ``.varats.yml`` configuration file by setting the value of ``from_source`` to true and the value of ``varats_source`` to the directory containing the source code of the VaRA-TS.
 
@@ -18,11 +26,12 @@ If you want to use your local development version of VaRA-TS instead, you have t
        vara-buildsetup build vara --container=DEBIAN_10
 
    Note that the underlying tools may not support network file systems (i.e., if you are using podman in `rootless mode <https://github.com/containers/podman/blob/master/rootless.md>`_).
-   In the ``.benchbuild.yml`` configuration, you need to set the values of ``root`` and ``runroot`` to a directory that is locally available on the current machine.
+   In the ``.benchbuild.yml`` configuration, you need to set the values of ``container/root`` and ``container/runroot`` to a directory that is locally available on the current machine.
    For example, you can use the following values:
 
    .. code-block:: yaml
 
+        container:
             root:
                 default: !create-if-needed '/scratch/<username>/vara-root/benchbuild/containers/lib'
                 desc: Permanent storage for container images
@@ -50,6 +59,8 @@ If you want to use your local development version of VaRA-TS instead, you have t
        vara-container build
 
    You can use the flag ``-i <base_image>`` to only build a specific base image and should add ``--export`` if you want to use the image with slurm.
+   There are also flags for re-building only parts of an image, e.g., to just update the tool suite.
+   These flags can drastically remove the time it takes to build an image.
 
 3. Running the experiments
 
@@ -58,11 +69,3 @@ If you want to use your local development version of VaRA-TS instead, you have t
    .. code-block:: console
 
        vara-run -E JustCompile --container
-
-   Do not forget to run
-
-   .. code-block:: console
-
-       vara-gen-bbconfig
-
-   and to adapt the values as described in step 1 before running your experiments.

docs/source/vara-ts-api/containers.rst

@@ -1,24 +1,29 @@
 Container support
 =================
 
-BenchBuild allows you to :ref:`run experiments in a container <Running BenchBuild in a container>`.
-The `containers` module handles the correct creation of container images.
-Our containers are structured in layers as follows::
+VaRA-TS allows you to :ref:`run experiments in a container <Container Guide>`.
+The ``containers`` module handles the correct creation of container images.
+Our container images are structured in multiple stages as follows::
 
     ┌───────────────────────────────────┐  \
+    │             Stage 00              │   |
     │ Base Container (e.g., Debian 10)  │   |
+    │          + dependencies           │   |
     └───────────────────────────────────┘   |
                       +                     |
     ┌───────────────────────────────────┐   |
-    │              varats               │   |
+    │             Stage 10              │   |
+    │         varats/benchbuild         │   |
     └───────────────────────────────────┘   |
                       +                      > Base Image
     ┌───────────────────────────────────┐   |
-    │            benchbuild             │   |
+    │             Stage 20              │   |
+    │    Research Tool (e.g., VaRA)     │   |
     └───────────────────────────────────┘   |
                       +                     |
     ┌───────────────────────────────────┐   |
-    │    Research Tool (e.g., VaRA)     │   |
+    │             Stage 30              │   |
+    │          configuration            │   |
     └───────────────────────────────────┘  /
                       +
     ┌───────────────────────────────────┐
@@ -30,6 +35,10 @@ Our containers are structured in layers as follows::
     └───────────────────────────────────┘
 
 
+Each stage results in its own container image.
+This allows us to update only some of the stages to save time when only changes to certain stages are required (especially stage 00 can be very time consuming to build).
+The :ref:`vara-container` tool provides appropriate command line flags to only re-build certain stages.
+
 Base images are built once for each :class:`~varats.containers.containers.ImageBase` with the currently configured research tool using the :ref:`vara-container` tool.
 Projects must :ref:`select one of these base images <Using Containers>` which they are then free to extend with their own project-specific dependencies.
 The directory structure inside a container looks like the following::
@@ -41,7 +50,7 @@ The directory structure inside a container looks like the following::
        ├─ BC_files/       # mount point to <varats-root>/benchbuild/BC_files outside the container
        └─ paper_configs/  # mount point to <varats-root>/paper_configs outside the container
 
-See :ref:`Running BenchBuild in a container` for how to set up the mount points.
+The required mount points are specified automatically when creating a BenchBuild config via :ref:`vara-gen-bbconfig`.
 
 Containers module
 .................

docs/source/vara-ts-api/projects.rst

@@ -197,7 +197,7 @@ environment).
 Using Containers
 ................
 
-To use :ref:`BenchBuild's container support <Running BenchBuild in a container>`, a project has to declare what container image to use.
+To use a project with our :ref:`container support <Container Guide>` it has to declare which container image to use.
 This can be done by assigning a container image to the `CONTAINER` class variable::
 
     CONTAINER = get_base_image(ImageBase.DEBIAN_10)

docs/source/vara-ts-api/tools/vara-container.rst

@@ -1,7 +1,7 @@
 vara-container
 ==============
 
-This tool is used to manage the base container images used when :ref:`running benchbuild in container mode<Running BenchBuild in a container>`.
+This tool is used to manage the base container images used when using our :ref:`container support<Container Guide>`.
 
 .. program-output:: vara-container -h
     :nostderr:
@@ -11,7 +11,8 @@ This tool provides an easy way to select the research tool that is used in the b
 .. program-output:: vara-container select -h
     :nostderr:
 
-With `vara-container build`, you can build all base container images for the current research tool:
+With `vara-container build`, you can build all base container images for the current research tool.
+Additional flags allow to only re-build parts of the base image.
 
 .. program-output:: vara-container build -h
     :nostderr:
@@ -25,16 +26,3 @@ Existing base images can also be deleted:
 
   This command does not delete project and experiment images, it only deletes base images.
   To delete project and experiment images use BenchBuild's `benchbuild container rmi` command.
-
-
-If you plan on running BenchBuild with container support on slurm, the following command helps you set up the correct environment:
-
-.. program-output:: vara-container prepare-slurm -h
-    :nostderr:
-
-
-.. warning::
-  This command changes the location where container images are built and stored in the BenchBuild config.
-  That means that all subsequent container images will reside in that directory including the base images that are built by this command.
-  This also means, that the ``--node_dir`` option you pass to this command must be a path that can be created on both, the machine where this command is executed and on the slurm nodes.
-  In general, it is a good idea to use some subdirectory of ``/tmp`` here, although that means that images and containers may be lost after a reboot.

docs/source/vara-ts/benchbuild.rst

@@ -189,10 +189,10 @@ Using buildah and podman on the Commandline
 ...........................................
 
 For more advanced users, it might be useful to work with buildah and podman directly from the commandline, e.g., when debugging container images.
-In these situations, it can come in handy to create some shell aliases that set the correct `root` and `runroot` to for the buildah and podman commands::
+In these situations, it can come in handy to create some shell aliases that set the correct `root` and `runroot` to for the buildah and podman commands, as well as the `storage_driver` that is set in BenchBuild's config file::
 
-    alias bbuildah='buildah --root $VARATS_ROOT/containers/lib --runroot $VARATS_ROOT/containers/run'
-    alias bpodman='podman --root $VARATS_ROOT/containers/lib --runroot $VARATS_ROOT/containers/run'
+    alias bbuildah='buildah --root $VARATS_ROOT/containers/lib --runroot $VARATS_ROOT/containers/run --storage-driver=vfs'
+    alias bpodman='podman --root $VARATS_ROOT/containers/lib --runroot $VARATS_ROOT/containers/run --storage-driver=vfs'
 
 
 Debugging Container Images
@@ -226,7 +226,7 @@ As an alternative, you can also mount the file system of a container image by fo
 
    .. code-block:: bash
 
-     newontainer=$(bbuildah from <image_id>)
+     newcontainer=$(bbuildah from <image_id>)
 
    This command will print a container id.
 4. Mount the working container (identified by the id you got from the step before) with

docs/source/vara-ts/slurm.rst

@@ -108,52 +108,42 @@ If you understand how BenchBuild uses containers to run experiments you can prep
 
 1. Set up VaRA-TS as described in the normal :ref:`slurm guide <Running with Slurm>` (steps 1 and 2).
 
-2. Set up the BenchBuild container support as described in the :ref:`BenchBuild container documentation <Running BenchBuild in a Container>`.
+2. Make sure that the host systems (from where slurm jobs are submitted), as well as the slurm cluster has rootless buildah and podman installed and configured (don't forget the subuid and subgid mappings for the users submitting the slurm jobs).
+   See :ref:`Running BenchBuild in a Container` for further instructions.
 
-3. Make sure that also the slurm cluster has rootless buildah and podman installed and configured (don't forget the subuid and subgid mappings for the users submitting the slurm jobs).
+3. Prepare the research tool(s) and base images (see the :ref:`container guide<Container Guide>`).
+   Use the ``--export`` flag when generating the base images to make them available to the containers running via slurm.
 
-4. Preparing the research tool(s) for each base container required by your experiments, e.g.:
-
-  .. code-block:: console
-
-       vara-buildsetup build vara --container=DEBIAN_10
-
-5. Rootless containers do not work on NFS (see `here <https://github.com/containers/podman/blob/master/rootless.md>`_), so we have to take some extra steps if we want to run containers via slurm.
-   These steps can be executed easily using the following command (:ref:`documentation <vara-container>`):
-
-   .. code-block:: bash
-
-     vara-container prepare-slurm
-
-   This step also builds the base images.
-
-   If you want to know in detail what happens in this command, take a look at the section :ref:`Prepare-Slurm in Detail`.
-
-6. After the preparation is complete, you can generate the slurm script as follows:
+4. After the preparation is complete, you can generate the slurm script as follows:
 
    .. code-block:: bash
 
      vara-run --slurm --container -E <report_type> <projects>
 
-7. That's it! the script obtained from the previous step can be used like any other slurm script.
+5. That's it! the script obtained from the previous step can be used like any other slurm script.
    You can now make any adjustments to the script if needed or just submit it to slurm as described in the slurm guide (step 5).
    You can also add the flag ``--submit`` to the ``vara-run`` command to directly submit the script to slurm.
 
 
-Prepare-Slurm in Detail
-...........................
+How vara-run handles rootless containers
+........................................
+
+.. note::
+
+    This section is mainly for documentation purposes and intended for advanced users and developers of VaRA-TS.
 
-As explained above, rootless containers do not work on NFS (see `here <https://github.com/containers/podman/blob/master/rootless.md>`_), so we have to take some extra steps if we want to run containers via slurm.
-The recommended way to do this is using the ``vara-container prepare-slurm`` command, but in some situations it might be handy to know what happens under the hood:
+Rootless containers do not work on NFS (see `here <https://github.com/containers/podman/blob/master/rootless.md>`_), so we have to take some extra steps if we want to run containers via slurm.
+These steps are handled automatically by ``vara-run`` but in some situations it might be handy to know what happens under the hood:
 
-    - You need to set the container root and runroot paths to some location that is not on a NFS, e.g., to a directory in ``tmp``:
+    - You need to set the container root and runroot paths to some location that is not on a NFS, e.g., to a directory in ``tmp``.
+      Since this may differ from your local setup, BenchBuild offers its own set of configuration options to set the buildah/podman root and runroot directories on slurm:
 
       .. code-block:: yaml
 
-        container:
-          root:
+        slurm:
+          container_root:
             value: /tmp/<username>/containers/lib
-          runroot:
+          container_runroot:
             value: /tmp/<username>/containers/run
 
     - BenchBuild allows to export and import container images.
@@ -192,14 +182,12 @@ The recommended way to do this is using the ``vara-container prepare-slurm`` com
 
     - Now it is time to generate the slurm script (cf. step 5 of the slurm guide).
       Because of our NFS workarounds, we cannot use the default script provided by BenchBuild, but we need to provide our own script template.
-      You can find our default template in the ``varats.tools`` module.
-      This template is very similar to the original template provided by BenchBuild, but it takes care of pointing all relevant environment variables to the slurm node directory as described in the points above.
+      You can find our default template in the ``varats.tools.templates`` module.
+      This template is very similar to the original template provided by BenchBuild, but it takes care of pointing all relevant environment variables to the slurm node directory as described in the points above and implements a different cleanup strategy for the node directory.
       To activate the template, simply save it to the ``/scratch/<username>/varats/benchbuild`` directory and set the appropriate value in the BenchBuild config:
 
       .. code-block:: yaml
 
         slurm:
           template:
             value: /scratch/<username>/varats/benchbuild/slurm_container.sh.inc
-
-      You can now continue with generating the slurm script as described above.

requirements.txt

@@ -1,5 +1,5 @@
 argparse-utils>=1.2.0
-benchbuild>=6.6.3
+benchbuild>=6.6.4
 click>=8.0.2
 distro>=1.5.0
 graphviz>=0.14.2