Developer Guide

Intro

Here you can find a guide on how to contribute to the Discourse Analysis Tool Suite (DATS) as a developer.

The DATS is developed as a client-server application and gets deployed via docker-compose. Since backend and frontend development is very different, this guide is split in two.

The setup described here uses docker compose to run auxiliary software like PostgreSQL, Redis etc., while running the backend and/or frontend outside of docker in a more traditional development environment.

Setting up the developer environment

You will need a Linux machine with Docker, optimally with NVIDIA Container Toolkit for GPU support. Other operation systems are not supported!

1. Install VSCode for development. We curated a list of extensions that are required for developing DATS and wrote several launch configurations that help with the development.

2. Clone this repository git clone git@github.com:uhh-lt/dats.git

3. Install the recommended VSCode plugins. Once you open the dats folder, a popup asks if the recommended extensions should be installed automatically. Answer with yes.

4. Install python environment. We use micromamba to manage python versions.

   1. See micromamba install instructions here. We use these commands curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj bin/micromamba, then ./bin/micromamba shell init -s bash -r ~/micromamba. Make sure you install mircomamba in your home direction (cd ~). After installing, it is required to restart the terminal, otherwise, the new micromamba command is not recognized.
   2. We recommend to add an alias to your .bashrc: alias mm='micromamba'. Use this to edit the file nano ~/.bashrc.
   3. Install the environment: First navigate to backend with cd backend then install the dats environment with micromamba env create -f environment.yml
   4. Activate the environment with micromamba activate dats
   5. Install additional dependencies from our ray_model_worker service: pip install -r src/app/preprocessing/ray_model_worker/requirements.txt
   6. Finally, install the ray dependency: pip install ray==2.32.0
   7. Set the newly create dats python environment as the Python Interpreter in VS Code. Therefore, use the shortcut CTRL/CMD + SHIFT + P and type Select Interpreter. From the dropdown, select the micromamba dats environment.

5. Install the Node.js environment. We use node version manager (nvm)

   1. See nvm install instructions here. We use these commands curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash. Make sure you install nvm in your home direction (cd ~). After installing, it is required to restart the terminal, otherwise, the new nvm command is not recognized.
   2. Install Node 20.17.0: nvm install 20.17.0
   3. Install the environment: First navigate to frontend with cd frontend then install the dependencies with npm install

6. Install pre-commit. We use pre-commit for automatic code linting for the backend. (Prettier is used for the frontend).

   1. Install the pre-commit dependency with pip install pre-commit.
   2. Initialize pre-commit with pre-commit install

Done! Now you have everything installed to get started with DATS development.

Docker Guide

We start with setting up the services running inside docker containers:

1. Configure the docker containers. In the docker directory, you will find a .env.example file to configure various settings. Start by making a copy of it: cp .env.example .env. Now modify the .env file accordingly:
   1. Change UID and GID to match user and group id of your current user. This will prevent permission problems with volumes mapping files into the container. You can find your users IDs with id.
   2. Change COMPOSE_PROJECT_NAME to prevent collisions with other docker compose projects running on the same machine.
   3. Change all values ending in _EXPOSED to configure the various ports your services will be available on, taking care to prevent conflicts with other services running on your machine. We will configure the backend later to access these ports.
   4. With COMPOSE_PROFILES you can change which services will be started. For development, we recommend to remove backend and frontend from this list. We won't need it because we will start the backend and frontend ourselves, outside of docker. Add development to the list.
   5. We recommend to use RAY_CONFIG=config_cpu.yaml for development.
   6. Use pwgen to create the JWT_SECRET
2. Run the script ./setup-folders.sh to automatically create directories for storing and caching ML models as well as uploaded documents.
3. Run docker compose up -d to start all docker containers. Use docker compose ps to check that all containers are running. On the first start, this will take quite a while, as the container will download some large AI models.

Resetting docker services

For a full, complete restart of the docker containers, we recommend the following procedure:

1. docker compose down -v - Stop all containers and remove the volumes.
2. rm -r backend_repo - Remove all uploaded user data
3. ./setup-folders - Create the backend_repo folder again
4. docker compose pull - Download the lastest docker containers of DATS
5. docker compose up -d - Restart all docker containers

Backend Guide

Intro and Structure

The backend is written in Python and is organized in directories each responsible for different tasks.

• backend/src/api -> REST API endpoints consumed by the frontend (or other clients)
• backend/src/app/core -> core backend logic such as the data model, internal service modules, search and analysis functionality
• backend/src/app/docprepro -> document preprocessing logic
• backend/src/test -> unit, integration, and e2e tests
• backend/src/configs -> configuration files to customize the backend behavior (handle with care!)

Development setup

1. Tell the backend how to reach the various services running in docker. In backend/.env.example, you can find an example configuration. Make a copy of this config with cp .env.example .env. Next, set the _PORT settings to match the _EXPOSED ports from before in docker/.env. Change the REPO_ROOT string to point to the docker/backend_repo folder inside your repository. Finally, change the API_PORT. Use pwgen to create the JWT_SECRET.
2. (optional) Load the backend configuration into your shell using set -o allexport; source backend/.env; set +o allexport.
3. In VSCODE, navigate to 'Run and Debug' and launch 'backend'.

Building for Production With Docker

This section describes deploying the backend API to production.

One container is built for the backend-api as well as the celery-background-jobs-worker as they share the same dependencies. The startup command (entrypoint) decides if the container is serving the API or running as a worker. The different entrypoints can be found at backend/src/*_entrypoint.sh, where every script corresponds to exactly one worker type. The backend source code is copied into the container, so are the config files in backend/src/configs, which can be altered to configure various settings.

This is what our release github action does:

1. Build docker container: docker build -f Dockerfile -t uhhlt/dats_backend:{version}.
2. Push docker container: docker push uhhlt/dats_backend:{version}

Data Model

The Data Model (DM) (located at backend/src/app/core/data/) is the core of the DATS and represents all it's entities. The DM is based on sqlalchemy and pydantic. Although sqlalchemy is (mostly) database agnostic, the intended database for the DATS DM is PostgreSQL.

The entities (i.e. database tables) are based on sqlalchemy and are located in backend/src/app/core/data/orm -- ORM stands for Object-Relational Mapping and defines the bridge between python classes and objects and the database. To perform Create-Read-Update-Delete (CRUD) operations we use the interfaces defined in backend/src/app/core/data/crud. To transfer entities between the frontend and the backend (and also often internally in the backend) we make use the DTOs (Data Transfer Objects) based on pydantic, which are located in backend/src/app/core/data/dto.

How to extend the Data Model

In this guide, you will learn all steps necessary to extend the DATS Data Model.

Please always have a look at implementations of other existing entities for examples and coding/naming conventions!

Step 1: Define the ORM

To define the ORM, first create a new file in backend/src/app/core/data/orm. For examples and further help, have a look at other files defined in this directory and the sqlalchemy documentation.

If you need an ObjectHandle (basically a pointer) for that entity (e.g. to enable Memos for that entity), you also have to extend the ObjectHandleORM accordingly.

Step 2: Register the ORM and create a migration

To register the ORM, import the class in the import_all_orms module (backend/src/app/core/data/db/import_all_orms.py). Then tell alembic to look at that file and generate a migration for models that are not present in the database yet. In VSCODE, navigate to

alembic revision --autogenerate -m "short description"

Step 3: Define the DTOs

To define the DTOs, first, create a new file in backend/src/app/core/data/dto with the same filename as the ORM file. In that file, add the DTOs as separate classes that inherit from each other (if possible).

Step 4: Implement CRUD

To implement the CRUD object, first, create a new file in backend/src/app/core/data/crud with the same filename as the ORM and DTO file. In that file, create a CRUD class that inherits from CRUDBase with the DTOs from the previous step. This already provides basic CRUD operations. If you need to customize or add operations, implement the respective methods in this class.

Frontend Guide

Intro and Structure

The frontend is written in TypeScript using React, bootstrapped with Create React App. It is organized in directories each responsible for different tasks.

• frontend/src/api -> custom TanStack Query hooks to communicate with the backend via the automatically generated API client in openapi.
• frontend/src/components -> reusable UI components
• frontend/src/features -> components that implement logic and access the API to build a feature. These features are used across the app and are not specific to a certain route.
• frontend/src/layouts -> different page layouts used by the routing library.
• frontend/src/plugins -> configuration files to customize the behavior of various plugins.
• frontend/src/router -> configuration of the React Router.
• frontend/src/store -> configuration of the global store - Redux Toolkit.
• frontend/src/view -> the main directory of the app. Every subfolder corresponds to a route. All features specific only to the corresponding route are implemented here.

Development Setup

1. Configure the frontend. Copy .env.development.example to .env.development and copy .env.production.example to .env.production. In frontend/.env.development, configure VITE_APP_SERVER and VITE_APP_CONTENT with the values from the previous step (API_EXPOSED and CONTENT_SERVER_EXPOSED). Change PORT.
2. (optional) Load the frontend configuration into your current shell using set -o allexport; source .env.development; set +o allexport.
3. Run npm run dev to start the frontend. Check https://localhost:{PORT} to see if everything is working.
4. Run npm run tsc-watch to watch for errors during development. Run this command in a seperate terminal.

Build for Production With Docker

When deploying the frontend, the React code is first bundled and then served via an NGINX web server. In production mode, the NGINX web server also acts as a reverse proxy for communication with the backend. frontend/.env.production/ is configured to make the backend api available at /api and the content server available at /content. The matching configuration of the NGINX web server is located at /docker/nginx.conf.

1. Build docker container: docker build -f Dockerfile -t uhhlt/dats_frontend:latest .
2. (optional) Push docker container: docker push uhhlt/dats_frontend:latest

Communication with the backend

The backend uses FastAPI to serve an accessible API that follows the OpenAPI standard. We consume this API by automatically generating an OpenAPI client with OpenAPI Typescript Codegen and build hooks around that with TanStack Query, that can be conveniently used in all components.

In case the backend is updated and offers new API endpoints, the following steps must be performed to make them available in the frontend:

1. npm run update-api - download the new OpenAPI specification of the backend.
2. npm run generate-api - generate the new client. This command deletes everything in frontend/src/api/openapi, generates new code, and formats it with prettier.
3. npm run dev - start the frontend again.
4. Implement a new Hook in frontend/src/api.

Deployment

Docker-compose is used to orchestrate the frontend, API, celery workers databases and other services that are used in the DATS Tool Suite.

1. Configure ports and credentials in /docker/.env
2. (optional) Note the environment variable DATS_BACKEND_CONFIG, which links to backend/src/configs/default_localhost_dev.yaml per default, additional adjustments can be made.
3. Set-up the folder structure with sh /docker/setup-folders.sh. This script creates models_cache for storing various ML models, spacy_models for storing various spaCy models for different languages, backend_repo for storing uploaded documents and tika that stores Java executables for Apache TIKA.
4. Deploy the application with docker compose up -d. Check with docker compose ps that all containers are running.

Versioning

We use Semantic Versioning as explained below:

Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes
2. MINOR version when you add functionality in a backward compatible manner
3. PATCH version when you make backward compatible bug fixes

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

For reference, see https://semver.org/.

Releasing a New Version

In the root of the repository, run:

bin/release.sh 1.0.3

This will update all necessary files, create a commit and matching tag, and push it. The github action will then take care of building containers and creating a github release.