nf-shard

nf-shard is an open source user interface for monitoring Nextflow runs, searching historical runs and analysing metrics. It was designed as drop-in replacement for community nf-tower, however it does not aim to replace Enterprise nf-tower.

Usage in Nextflow

Once nf-shard is running to integrate it into your Nextflow project you should add similar snippet to your nextflow.config. The exact form of this snippet will be displayed once you start nf-shard.

tower {
    enabled = true
    accessToken = "non-empty"
    endpoint = "http://localhost:3000/api"
}

Run

Following instructions allows to run nf-shard locally. To run nf-shard you PostgreSQL database.

Method 1 - docker-compose

docker-compose will spin up PostgreSQL database and the server.

git clone git@github.com:AugustDev/nf-shard.git
cd nf-shard
docker-compose --profile all up

If you have PostgreSQL running externally then you only need to launch the server. In this case update your .env with POSTGRES_URI and run

git clone git@github.com:AugustDev/nf-shard.git
cd nf-shard
docker-compose --profile server up

Add tower server details in your Nextflow config.

tower {
    enabled = true
    accessToken = "non-empty"
    endpoint = "http://localhost:3000/api"
}

Done! If you navigate to http://localhost:3000 and run Nextflow workflow you should be able to see progress in the UI.

Note - accessToken can by any non-empty string.

Method 2 - yarn

If you already have running PostgreSQL database, you can run build the project using yarn package manager. To specify your database login edit .env. If you are developer you would prefer using this approach.

yarn
yarn migrate
yarn build
yarn run

Not that yarn migrate requires connection to databse, so you should update .env file.

To run PostgreSQL for local development you can use

docker-compose --profile db up

Since connection to PostgreSQL now happens outside of docker, you should update your .env to specify localhost

POSTGRES_URI=postgresql://postgres:yourpassword@localhost:5432/postgres?schema=public

Method 3 - Vercel

One click deployment using Vercel requires having PostgreSQL access. Video instructions below. In the video I use Neon.tech free PostgreSQL version, but any provider works.

Stack

• NextJS/React/Typescript
• PostgreSQL
• Prisma ORM

I was picking the a stack with large communities to maximise open source contribution from the Nextflow community.

Features

• Slack integration
• Indexed search by workflow ID, run name, user name, tag, projeect name, before date, after date.
• Search supports multiple AND conditions.
• Ability to attach multiple tags for each workflow. Tags are visible in the UI and can be used in search.
• Workspaces
• List of historical runs
• Run details
• nf-tower plugin compatible API.
• Store and submit pipeline jobs from Shard UI (beta requires deploying shard-worker)

Roadmap

• Authentication
• Dedicated plugin
• Upload/view execution logs
• Download reports
• tRPC client/server communication

Development

Useful during development

docker buildx build --platform linux/amd64 -t nf-shard:semver . --load
docker run -it nf-shard:semver /bin/sh

Author

Augustinas Malinauskas