Informatics Matters Job Tester ("jote")

The Squonk2 Job Tester (jote) is a Python utility used to run unit tests that are defined in Data Manager job implementation repositories against the job's container image, images that are typically built from the same repository.

jote is designed to run job implementations in a file-system environment that replicates what they find when they're run by the Data Manager. But jobs are not running in the same operating-system environment, e.g. they are not bound by the same processor and memory constraints they'll encounter in the Data Manager, which runs in Kubernetes.

To use jote you will need to install docker-compose (v1 or v2).

A successful test should give you confidence that it should work in the Data Manger but without writing a lot of tests you'll never be completely confident that it will always run successfully.

jote is a tool we designed to provide us with confidence that we can deploy jobs to a Data Manager instance and know that they're basically fit for purpose. Jobs that have no tests will not normally be deployed to the Data Manager.

To use a job in Squonk you need to create at least one manifest file and one job definition file. These reside in the data-manager directory of the repository you're going to test. jote expects the default manifest file to be called manifest.yaml but you can use a different name and have more than one.

If you want to provide your own Squonk jobs and corresponding job definitions our Virtual Screening repository (https://github.com/InformaticsMatters/virtual-screening) is a good place to start. The repository is host to a number of job-based container images and several manifests and job definition files.

Here's an example manifest from a recent Virtual Screening repository:

---
kind: DataManagerManifest
kind-version: '2021.1'

job-definition-files:
- im-virtual-screening.yaml
- rdkit.yaml
- xchem.yaml

Each Manifest must list at least one file. To be included in Squonk every job must contain at least one test. jote runs the tests but also ensures the repository structure is as expected and applies strict rules for the formatting of the YAML files.

Both jote and the Data Manager rely on the schemas that can be found in our Job Decoder repository (https://github.com/InformaticsMatters/data-manager-job-decoder).

Here's a snippet from a job definition file illustrating a job (max-min-picker) that has a test called simple-execution.

The test defines an input option (a file) and some other command options. The checks section is used to define the exit criteria of the test. In this case the container must exit with code 0 and the file diverse.smi must be found in the generated test directory, i.e it must exist and contain 100 lines. jote will fail the test unless these checks are satisfied:

jobs:
  [...]
  max-min-picker:
    [...]
    tests:
      simple-execution:
        inputs:
          inputFile: data/100000.smi
        options:
          outputFile: diverse.smi
          count: 100
        checks:
          exitCode: 0
          outputs:
          - name: diverse.smi
            checks:
            - exists: true
            - lineCount: 100

Running tests

Run jote from the root of a clone of the Data Manager Job implementation repository that you want to test:

jote

You can display the utility's help with:

jote --help

Jote container network

jote tests are executed on the network data-manager_jote. This is defined in the docker-compose file that it generates to run your tests.

Built-in variables

Job definition command-expansion provided by the job decoder relies on a number of built in variables. Some are provided by the Data Manager when the job runs under its control (i.e. DM_INSTANCE_DIRECTORY) others are provided by jote to simplify testing.

The set of variables injected into the command expansion by jote are: -

• DM_INSTANCE_DIRECTORY. Set to the path of the simulated instance directory created by jote, normally created by the Data Manager
• CODE_DIRECTORY. Set to the root of the repository that you're running the tests in. This is a convenient variable to locate your out-of-container nextflow workflow file, which is likely to be in the root of your repository

Ignoring tests

Occasionally you may want to disable some tests because they need some work before they're complete. To allow you to continue testing other jobs under these circumstances you can mark individual tests and have them excluded by adding an ignore declaration:

jobs:
  [...]
  max-min-picker:
    [...]
    tests:
      simple-execution:
        ignore:
        [...]

You don't have to remove the ignore declaration to run the test in jote. If you want to see whether an ignored test now works you can run jote for specific tests by using --test and naming the ignored test you want to run. When a test is named explicitly it is run, regardless of whether ignore has been set or not.

Test run levels

Tests can be assigned a run-level. Run-levels are numerical value (1..100) that can be used to group your tests. You can use the run-level as an indication of execution time, with short tests having low values and time-consuming tests with higher values.

By default all tests that have no run-level defined and those with a run-level of 1 are executed. If you set the run-level for longer-running tests to a higher value, e.g. 5, these will be skipped. To run these more time-consuming tests you specify the run-level when running jote using --run-level 5.

When you give jote a run-level only tests up to and including the level, and those without any run-level, will be run.

You define the run-level in the root block of the job's test specification:

jobs:
  [...]
  max-min-picker:
    [...]
    tests:
      simple-execution:
        run-level: 5
        [...]

Test timeouts

jote lets each test run for 10 minutes before cancelling (and failing) them. If you expect that your test needs to run for more than 10 minutes you must use the timeout-minutes property in the job definition to define your own test-specific value:

jobs:
  [...]
  max-min-picker:
    [...]
    tests:
      simple-execution:
        timeout-minutes: 120
        [...]

You should try and avoid creating too many long-running tests. If you cannot, consider whether it's a appropriate to use run-level to avoid jote running them by default.

Test groups

Tests are normally executed and the environment torn-down between them. If you have tests that depend on the results from a prior test you can run tests as a group, which preserves the project directory between the tests.

To run a sequence of test (as a group) you need to define a test-group in your Job Definition file and then refer to that group in your test. Here, we define a test group called experiment-a, at the top of the definition file:

test-groups:
- name: experiment-a

We then place a test in that group with a run-group declaration in the corresponding test block:

jobs:
  max-min-picker:
    [...]
    tests:
      test-a:
        run-groups:
        - name: experiment-a
          ordinal: 1

We need to provide an ordinal value. This numeric value (from 1 ..N) puts the test in a specific position in the test sequence. When tests are placed in a run-group you have to order your tests, i.e. declare that test-a follows test-b. This is done with unique ordinals for each test in the run-group. A test with ordinal 1 will run before a test with ordinal 2. Ordinals have to be unique within a run-group.

You can run the tests for a specific group by using the --run-group option:

jote --run-group experiment-a

Running additional containers (group testing)

Test groups provide an ability to launch additional support containers during testing. You might want to start a background database for example, that can be used by tests in your test-group. To take advantage of this feature you just need to provide a docker-compose file (in the Job definition data-manager directory) and name that file in you r``test-groups`` declaration.

Here we declare a docker-compose file called docker-compose-experiment-a.yaml:

test-groups:
- name: experiment-a
  compose:
    file: docker-compose-experiment-a.yaml

The compose filename must begin docker-compose and end .yaml.

The compose file is run before any tests in the corresponding test group have been run and will be stopped after the last test in the group.

The compose file you provide is run in a detached state so jote does not wait for the containers to start (or initialise). As the first test in the test group can begin very soon after the compose file is started you can minimise the risk that your containers are not ready for the tests by adding a fixed delay between jote starting the compose file and running the first test:

test-groups:
- name: experiment-a
  compose:
    file: docker-compose-experiment-a.yaml
    delay-seconds: 10

Nextflow test execution

Job image types can be simple or nextflow. Simple jobs are executed in the container image you've built and should behave much the same as they do when run within the Data Manager. Nextflow jobs on the other hand are executed using the shell, relying on Docker as the execution run-time for the processes in your workflow.

Be aware that nextflow tests run by jote run under different conditions compared to when it runs under the Data Manager's control. In the Data Manager nextflow jobs will be executed within a Kubernetes environment. When run by jote nextflow is expected using the operating system shell. This introduces a variability that you need to take into account - i.e. under jote the nextflow controller runs in the shell, and are not executed in the same environment or under the same memory or processor constraints.

You might need to provide a custom nextflow configuration file for your tests to run successfully. You do this by adding a nextflow-config-file declaration in the test. Here, we name the file nextflow-test.config:

jobs:
  max-min-picker:
    [...]
    tests:
      simple-load:
        nextflow-config-file: nextflow-test.config
        [...]

The config file must be located in the Job repository's data-manager directory.

Prior to running the corresponding test jote copies it to the Job's project directory as the file nextflow.config (a standard file expected by nextflow).

jote will not let you have a nextflow config in your home directory as any settings found there would be merged with the file jote writes, potentially disturbing the execution behaviour.

Note

It's your responsibility to install a suitable nextflow that's available for shell execution. jote expects to be able to run nextflow when executing the corresponding command that's defined in the job definition.

Installation

jote is published on PyPI and can be installed from there:

pip install im-jote

This is a Python 3 utility, so try to run it from a recent (ideally 3.10) Python environment.

To use the utility you will need to have installed Docker, docker-compose,

and, if you want to test nextflow jobs, nextflow.

Get in touch

• Report bugs, suggest features or view the source code on GitHub.