Deployment of pyiron workflow environment

This repository provides the configuration files, needed for the deployment of a pre-configured jupyterhub to run pyiron workflows.

Content of the repository

File	Description
config.json	a json file containing the configurable environment variables: keycloak client details, name of pyiron docker images, users' resources, and Postgres database Password
static.json	a json file containing the non-configurable environment variables
nginx/pyiron.conf	A template of the configuration file for the Nginx reverse proxy
docker-compose.yml	the compose file, which runs jupyterhub and its Postgres database services.

Deployment Guide

The infrastructure, OS and other required packages

As an infrastructure, the deployment requires:

• A server running a Linux operating system
• Installation of docker engine and docker-compose
• Adequate resources for running jupyterhub: (>2GB of RAM + 2 VCPU)
• The users' resources on the server should be proportional to the number of users (~2GB of RAM, 2VCPU, 10GB of storage per users)

Assumptions

• For the authentication of the users, keycloak is assumed as the authentication provider. Therefore, a client id and secret are needed.
• Here, it is assumed that all the jobs are run on the same server as the jupyterhub.
• The configuration of the hostnames is assumed to be done separately by the admin, e.g. creating the A-record, etc

pyiron docker images

In the configuration file config.json, it is needed to pass in a set of pyiron docker image. pyiron offers various docker images corresponding to its modules, atomistics, continuum, md, ... . The docker images are available on docker hub: https://hub.docker.com/u/pyiron, and the corresponding dockerfiles can also be found via https://github.com/pyiron/docker-stacks.

Nginx configuration

For the Nginx reverse proxy, you need to provide a configuration file, e.g. pyiron.conf. A template of such file is provided in the repository under repo_path/nginx/pyiron.conf
After adding the domain, and the path to the SSL certificates and key (as described here), you should copy it to the nginx directory:

cp nginx/pyiron.conf ../data/nginx/

Of course, here it is assumed that the PMD-S core repo is the parent directory of the pyiron deployment directory.

Adding pyiron client to Keycloak

For configuring the hub to authenticate the users via keycloak, you need to create a client in the keycloak instance. You can create the client via the following steps:

1. Signing in as the admin of the keycloak instance,
2. Select your desirable realm
3. Select clients from the left panel
4. On the top right-hand side, select create.
5. Enter a client ID, and a root URL. For the URL, you should provide the url of pyiron.
6. click save.
7. In the setting tab, change the access type to confidential, and save.
8. From the credential tab, you can obtain the client secret.

For configuring pyiron, you need the keycloak domain, the realm, the client id, and the client secret.

The process of the deployment

It is assumed that the current working directory is the root of PMD-S core repository.

1. cloning the current git repository.

   git clone https://github.com/materialdigital/pyiron_workflow_environment_deployment.git pyiron/
   cd pyiron

2. Providing the values for the keys in the config.json file. The keys are:

• OAUTH2_TOKEN_URL: Keycloak Tocken URL; here you need only to change the domain and the realm
• OAUTH2_AUTHORIZE_URL:Keycloak authorize URL; here you need only to change the domain and the realm
• OAUTH2_USERDATA_URL: Keycloak userdate URL; here you need only to change the domain and the realm
• OAUTH_CALLBACK_URL: Keycloak call back URL; here you need only to change the domain
• CLIENT_ID: The client ID defined in Keycloak
• CLIENT_SECRET: The secret for the client, provided from the Keycloak instance
• PYIRON_BASE: The relavant information of pyiron_base image in the form of image_name:tag, e.g. pyiron/base:2021-06-04
• PYIRON_ATOMISTIC: The relavant information of pyiron_atomistics image in the form of image_name:tag, e.g. pyiron/pyiron:2021-06-04
• PYIRON_CONTINUUM: The relavant information of pyiron_continuum image in the form of image_name:tag, e.g. pyiron/continuum:2021-06-04.
• PYIRON_EXPERIMENTAL: The relavant information of pyiron_experimental image in the form of image_name:tag, e.g. pyiron/experimental:2021-06-04
• MEM_LIMIT: The limiting amount of RAM per user
• CPU_LIMIT: The limiting amount of VCPU per user
• ADMIN_USER: The username of jupyterhub admin, this username should be consistent with the username in the keycloak instance
• POSTGRES_PASSWORD: A password for the postgres database

3. run the config.py script from the pmd-server parent directory.
   • in the case that python3 is installed on the host OS:

      python3 ../scripts/configure.py

   • in the case of no python3 installation on the host OS:

      docker run --rm -v $PWD/:/tmp/ -v $PWD/../scripts/configure.py:/tmp/configure.py -w /tmp  python:3-alpine  python configure.py

4. run docker-compose script via

      docker-compose up -d

HPC connection (will be added soon)

In principal, the pyiron docker containers can submit jobs to the cluster according to pyiron documentation in here. This feature will be added to pyiron docker images in the next release.

Room for modification

Here we assumed a semi-automated deployment with minimal changes needed from the side of IT adminstrators. However, many things can be changed, such as:

• building a customized jupyterhub; this gives the possibility to change the jupyterhub configuration
• In the current setup, we assumed to have four jupyter environments based on: pyiron_base, pyiron_atomistics, and pyiron_continuum, pyiron_experimental. This list can be extended in the case of a customized jupyterhub build.