GODRIVER-3061 Expand the capabilities of the docker container (#378)

* Clean up docker handling

* syntax

* clean up and add client runner

* clean up and handle load balancer

* fix syntax

* fix run-server

* add an ignore

* address review and add docker cleanup

* Update .evergreen/docker/README.md

Co-authored-by: Preston Vasquez <prestonvasquez@icloud.com>

---------

Co-authored-by: Preston Vasquez <prestonvasquez@icloud.com>

.evergreen/config.yml

@@ -372,7 +372,7 @@ functions:
           ${PREPARE_SHELL}
           set -ex
           cd $DRIVERS_TOOLS/.evergreen/docker
-          ENTRYPOINT=/root/test-entrypoint.sh bash run-local.sh
+          ENTRYPOINT=/root/test-entrypoint.sh bash run-server.sh
           # Generate a test results file
           cd ${PROJECT_DIRECTORY}
           make test
@@ -419,6 +419,15 @@ functions:
           ${PREPARE_SHELL}
           rm -rf $DRIVERS_TOOLS || true
 
+  "cleanup docker":
+    - command: shell.exec
+      params:
+        script: |
+          # Kill all containers
+          docker rm $(docker ps -a -q)&> /dev/null || true
+          # Remove all images
+          docker rmi -f $(docker images -a -q) &> /dev/null || true
+
   "fix absolute paths":
     - command: shell.exec
       params:
@@ -802,11 +811,13 @@ tasks:
       tags: ["latest", "docker"]
       commands:
         - func: "run docker test"
+        - func: "cleanup docker"
 
     - name: "test-oidc"
       tags: ["latest", "oidc"]
       commands:
         - func: "run oidc test"
+        - func: "cleanup docker"
 
     - name: "test-oidc-azure"
       commands:

.evergreen/docker/README.md

@@ -1,9 +1,9 @@
 # Drivers Evergreen Tools Dockerfile
 
-The ``Dockerfile`` and scripts in the subdirector(ies) serve dual purposes.
+The `Dockerfile` and scripts can be used to run a local server in docker container.
 
-- Run a local server in docker container.
-- Extend and run a driver test in a docker container.
+Additionally, you can build a Docker [container](#driver-testing-using-this-docker-container)
+for your Driver that shares the binary files and communicates with this container.
 
 You will need Docker (or podman aliased to Docker) installed and running
 locally.
@@ -13,7 +13,7 @@ locally.
 To run a local server, change to this directory and run:
 
 ```bash
-bash ./run-local.sh
+bash ./run-server.sh
 ```
 
 This will build the docker image and run it with appropriate settings.
@@ -22,22 +22,64 @@ will be passed through to the container.
 The appropriate port(s) will be exposed, allowing you to run local test against
 the running docker container.
 
-The default image can be overridden with `IMAGE``, and the entrypoint with `ENTRYPOINT`.
+The default image can be overridden with `IMAGE`, and the entrypoint with `ENTRYPOINT`.
 To use a specific architecture, use `PLATFORM`, e.g. `--platform linux/amd64`.
 
-## Driver Testing in Docker
+The script also supports the standard environment variables used in mongo-orchestration:
 
-To extend this image and run against a driver test suite, first build the
-image locally.
+```
+AUTH
+SSL
+TOPOLOGY
+LOAD_BALANCER
+STORAGE_ENGINE
+REQUIRE_API_VERSION
+DISABLE_TEST_COMMANDS
+MONGODB_VERSION
+MONGODB_DOWNLOAD_URL
+ORCHESTRATION_FILE
+```
+
+Note that the default `TOPOLOGY` is [`servers`](https://github.com/mongodb-labs/drivers-evergreen-tools/tree/master/.evergreen/orchestration/configs/servers) and the default `ORCHESTRATION_FILE` is `basic.json`. For example, to run a replica using the [auth](https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/orchestration/configs/replica_sets/auth.json) orchestration:
 
 ```bash
-docker build -t drivers-evergreen-tools .
+TOPOLOGY=replica_set ORCHESTRATION_FILE=auth.json bash ./run-server.sh 
 ```
+## Driver Testing using this Docker container
 
-Then, in your `Dockerfile`, use `FROM drivers-evergreen-tools`.
+First, start this container with the appropriate environment variables, running as:
 
-When running your derived image, use `-v $DRIVERS_TOOLS:/root/drivers-evergreen-tools`
-to use the local checkout.
+```bash
+bash ./run-server.sh
+```
 
-In your entry point script, run `run-orchestration.sh` before running your test suite.
-Note that you will probably want to expose the environment variables as is done in `run-local.sh`.
+You may wish to launch other services at this point, like a load balancer or the
+csfle KMIP server.
+
+To launch your driver's Dockerfile, prep the necessary environment variables
+and args, and run:
+
+```bash
+$DRIVERS_TOOLS/.evergreen/docker/run-client.sh $ARGS
+```
+
+You'll have the following env variables available in your container by default.
+
+```
+AUTH
+SSL
+TOPOLOGY
+MONGODB_VERSION
+MONGODB_BINARIES
+CRYPT_SHARED_LIB_PATH
+ORCHESTRATION_FILE
+SKIP_CRYPT_SHARED_LIB
+DRIVERS_TOOLS
+```
+
+In the entry point of your container, ensure to run the following to add the
+crypt shared and other binaries to your PATH:
+
+```bash
+export PATH="$MONGODB_BINARIES:$PATH"
+```

.evergreen/docker/run-client.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Run a driver test in a docker container that targets the 
+# the server running in docker.
+#
+set -eu
+
+# Docker related variables.
+IMAGE=${TARGET_IMAGE:-ubuntu20.04}
+PLATFORM=${DOCKER_PLATFORM:-}
+# e.g. --platform linux/amd64
+
+# Mongo orchestration related variables.
+MONGODB_VERSION=${MONGODB_VERSION:-latest}
+TOPOLOGY=${TOPOLOGY:-replica_set}
+ORCHESTRATION_FILE=${ORCHESTRATION_FILE:-basic.json}
+SKIP_CRYPT_SHARED_LIB=${SKIP_CRYPT_SHARED_LIB:-false}
+AUTH=${AUTH:-""}
+SSL=${SSL:-""}
+
+# Internal variables.
+ROOT_DRIVERS_TOOLS=/root/drivers-evergeen-tools
+MONGODB_BINARIES="ROOT_DRIVERS_TOOLS/.evergreen/docker/$IMAGE/mongodb/bin"
+
+# Build up the arguments.
+ARGS="$PLATFORM --rm -i"
+ARGS+=" -e MONGODB_VERSION=$MONGODB_VERSION -e TOPOLOGY=$TOPOLOGY"
+ARGS+=" -e SSL=$SSL -e AUTH=$AUTH"
+ARGS+=" -e MONGODB_BINARIES=$MONGODB_BINARIES"
+ARGS+=" -e CRYPT_SHARED_LIB_PATH=$MONGODB_BINARIES/mongosh_crypt_v1.so"
+ARGS+=" -e ORCHESTRATION_FILE=$ORCHESTRATION_FILE"
+ARGS+=" -e SKIP_CRYPT_SHARED_LIB=$SKIP_CRYPT_SHARED_LIB"
+ARGS+=" -e DRIVERS_TOOLS=$ROOT_DRIVERS_TOOLS"
+
+# Ensure host.docker.internal is available on MacOS.
+if [ "$(uname -s)" = "Darwin" ]; then
+    ARGS+=" -e MONGODB_URI=mongodb://host.docker.internal"
+fi
+
+# Ensure host network is available on Linux.
+if [ "$(uname -s)" = "Linux" ]; then
+    ARGS+=" --network=host"
+fi
+
+# If there is a tty, add the -t arg.
+test -t 1 && ARGS+=" -t"
+
+# Map the cwd to /src and map in DRIVERS_TOOLS.
+ARGS+=" -v `pwd`:/src"
+ARGS+=" -v $DRIVERS_TOOLS:/root/drivers-evergreen-tools"
+
+# Launch client docker container.
+docker run $ARGS $@

.evergreen/docker/run-server.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+#
+# Run a local MongoDB orchestration inside a docker container
+#
+set -eu
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+NAME=drivers-evergreen-tools
+ENTRYPOINT=${ENTRYPOINT:-/root/local-entrypoint.sh}
+IMAGE=${TARGET_IMAGE:-ubuntu20.04}
+PLATFORM=${DOCKER_PLATFORM:-}
+# e.g. --platform linux/amd64
+
+docker build $PLATFORM -t $NAME $IMAGE
+cd $SCRIPT_DIR/../..
+
+# Remove existing mongodb files
+rm -rf $SCRIPT_DIR/$IMAGE/mongodb
+
+# Handle environment variables.
+AUTH=${AUTH:-noauth}
+SSL=${SSL:-nossl}
+TOPOLOGY=${TOPOLOGY:-server}
+LOAD_BALANCER=${LOAD_BALANCER:-}
+STORAGE_ENGINE=${STORAGE_ENGINE:-}
+REQUIRE_API_VERSION=${REQUIRE_API_VERSION:-}
+DISABLE_TEST_COMMANDS=${DISABLE_TEST_COMMANDS:-}
+MONGODB_VERSION=${MONGODB_VERSION:-latest}
+MONGODB_DOWNLOAD_URL=${MONGODB_DOWNLOAD_URL:-}
+ORCHESTRATION_FILE=${ORCHESTRATION_FILE:-basic.json}
+
+# Build up the args.
+ARGS="$PLATFORM --rm -i"
+ARGS+=" -e MONGODB_VERSION=$MONGODB_VERSION"
+ARGS+=" -e TOPOLOGY=$TOPOLOGY"
+ARGS+=" -e AUTH=$AUTH"
+ARGS+=" -e SSL=$SSL"
+ARGS+=" -e ORCHESTRATION_FILE=$ORCHESTRATION_FILE"
+ARGS+=" -e LOAD_BALANCER=$LOAD_BALANCER"
+ARGS+=" -e STORAGE_ENGINE=$STORAGE_ENGINE"
+ARGS+=" -e REQUIRE_API_VERSION=$REQUIRE_API_VERSION"
+ARGS+=" -e DISABLE_TEST_COMMANDS=$DISABLE_TEST_COMMANDS"
+ARGS+=" -e MONGODB_DOWNLOAD_URL=$MONGODB_DOWNLOAD_URL"
+
+# Expose the required ports.
+if [ "$TOPOLOGY" == "server" ]; then
+    ARGS+=" -p 27017:27017"
+elif [ -n "$LOAD_BALANCER" ]; then
+    ARGS+=" -p 27017:27017 -p 27018:27018 -p 27019:27019 -p 27050:27050 -p 27051:27051"
+else
+    ARGS+=" -p 27017:27017 -p 27018:27018 -p 27019:27019"
+fi
+
+# If there is a tty, add the -t arg.
+test -t 1 && ARGS+=" -t"
+
+# Map in the DRIVERS_TOOLS directory.
+ARGS+=" -v `pwd`:/root/drivers-evergreen-tools"
+
+# Launch server docker container.
+docker run $ARGS $NAME $ENTRYPOINT

.evergreen/docker/ubuntu20.04/Dockerfile

@@ -22,8 +22,8 @@ ARG GROUP_ID
 
 ENV DRIVERS_TOOLS=/root/drivers-evergreen-tools
 ENV PROJECT_ORCHESTRATION_HOME=/root/drivers-evergreen-tools/.evergreen/orchestration
-ENV MONGODB_BINARIES=/root/mongodb/bin
-ENV MONGODB_BINARY_ROOT=/root
+ENV MONGODB_BINARIES=/root/drivers-evergreen-tools/.evergreen/docker/ubuntu20.04/mongodb/bin
+ENV MONGODB_BINARY_ROOT=/root/drivers-evergreen-tools/.evergreen/docker/ubuntu20.04/
 ENV MONGO_ORCHESTRATION_HOME=/root
 ENV DOCKER_RUNNING=true
 

.evergreen/download-mongodb.sh

@@ -760,6 +760,22 @@ download_and_extract ()
       rm -rf $DRIVERS_TOOLS/legacy-shell-download
       echo "Download legacy shell from 5.0 ... end"
    fi
+
+   # Define SKIP_CRYPT_SHARED=1 to skip downloading crypt_shared. This is useful for platforms that have a
+   # server release but don't ship a corresponding crypt_shared release, like Amazon 2018.
+   if [ -z "${SKIP_CRYPT_SHARED:-}" ]; then
+      if [ -z "$MONGO_CRYPT_SHARED_DOWNLOAD_URL" ]; then
+         echo "There is no crypt_shared library for distro='$DISTRO' and version='$MONGODB_VERSION'".
+      else
+         echo "Downloading crypt_shared package from $MONGO_CRYPT_SHARED_DOWNLOAD_URL"
+         download_and_extract_crypt_shared "$MONGO_CRYPT_SHARED_DOWNLOAD_URL" "$EXTRACT" CRYPT_SHARED_LIB_PATH
+         echo "CRYPT_SHARED_LIB_PATH:" $CRYPT_SHARED_LIB_PATH
+         if [ -z $CRYPT_SHARED_LIB_PATH ]; then
+            echo "CRYPT_SHARED_LIB_PATH must be assigned, but wasn't" 1>&2 # write to stderr"
+            exit 1
+         fi
+      fi
+   fi
 }
 
 # download_and_extract_crypt_shared downloads and extracts a crypt_shared package into the current directory.

.evergreen/run-orchestration.sh

@@ -47,6 +47,17 @@ else
 fi
 download_and_extract "$MONGODB_DOWNLOAD_URL" "$EXTRACT" "$MONGOSH_DOWNLOAD_URL" "$EXTRACT_MONGOSH"
 
+# Write the crypt shared path to the expansion file if given.
+if [ -n $CRYPT_SHARED_LIB_PATH ]; then
+    cat <<EOT >> mo-expansion.yml
+CRYPT_SHARED_LIB_PATH: "$CRYPT_SHARED_LIB_PATH"
+EOT
+
+  cat <<EOT >> mo-expansion.sh
+export CRYPT_SHARED_LIB_PATH="$CRYPT_SHARED_LIB_PATH"
+EOT
+fi
+
 DL_END=$(date +%s)
 MO_START=$(date +%s)
 
@@ -94,10 +105,12 @@ perl -p -i -e "s|ABSOLUTE_PATH_REPLACEMENT_TOKEN|${DRIVERS_TOOLS}|g" $ORCHESTRAT
 
 # Docker does not enable ipv6 by default.
 # https://docs.docker.com/config/daemon/ipv6/
+# We also need to use 0.0.0.0 instead of 127.0.0.1
 if [ -n "$DOCKER_RUNNING" ]; then
   cp $ORCHESTRATION_FILE /root/config.json
   export ORCHESTRATION_FILE=/root/config.json
   sed -i "s/\"ipv6\": true,/\"ipv6\": false,/g" $ORCHESTRATION_FILE
+  sed -i "s/\"127\.0\.0\.1\,/\"0.0.0.0\,/g" $ORCHESTRATION_FILE
 fi
 
 export ORCHESTRATION_URL="http://localhost:8889/v1/${TOPOLOGY}s"
@@ -118,27 +131,6 @@ URI=$(${PYTHON:?} -c 'import json; j=json.load(open("tmp.json")); print(j["mongo
 echo 'MONGODB_URI: "'$URI'"' > mo-expansion.yml
 echo $URI > $DRIVERS_TOOLS/uri.txt
 printf "\nCluster URI: %s\n" "$URI"
-# Define SKIP_CRYPT_SHARED=1 to skip downloading crypt_shared. This is useful for platforms that have a
-# server release but don't ship a corresponding crypt_shared release, like Amazon 2018.
-if [ -z "${SKIP_CRYPT_SHARED:-}" ]; then
-  if [ -z "$MONGO_CRYPT_SHARED_DOWNLOAD_URL" ]; then
-    echo "There is no crypt_shared library for distro='$DISTRO' and version='$MONGODB_VERSION'".
-  else
-    echo "Downloading crypt_shared package from $MONGO_CRYPT_SHARED_DOWNLOAD_URL"
-    download_and_extract_crypt_shared "$MONGO_CRYPT_SHARED_DOWNLOAD_URL" "$EXTRACT" CRYPT_SHARED_LIB_PATH
-    echo "CRYPT_SHARED_LIB_PATH:" $CRYPT_SHARED_LIB_PATH
-    if [ -z $CRYPT_SHARED_LIB_PATH ]; then
-      echo "CRYPT_SHARED_LIB_PATH must be assigned, but wasn't" 1>&2 # write to stderr"
-      exit 1
-    fi
-  cat <<EOT >> mo-expansion.yml
-CRYPT_SHARED_LIB_PATH: "$CRYPT_SHARED_LIB_PATH"
-EOT
-cat <<EOT >> mo-expansion.sh
-export CRYPT_SHARED_LIB_PATH="$CRYPT_SHARED_LIB_PATH"
-EOT
-  fi
-fi
 
 MO_END=$(date +%s)
 MO_ELAPSED=$(expr $MO_END - $MO_START)