The Task Parallel System Composer (TaPaSCo)

Master Branch Status: Dev Branch Status:

Introduction

Specialized accelerators in a heterogeneous system play a vital role in providing enough compute power for current and upcoming computational tasks. Field-programmable gate arrays (FPGA) are an established platform for such custom and highly specialized accelerators. However, an accelerator implementation alone is only part of the way to a usable system. In order to be used as a specialized co-processor in a heterogeneous setup, the accelerator still needs to be integrated into the overall system and requires a connection to the host (typically a software-programmable CPU) and often also external memory.

The open-source TaPaSCo (Task-Parallel System Composer) framework was created to serve exactly this purpose: The fast integration of FPGA-based accelerators into heterogeneous compute platforms or systems-on-chip (SoC) and their connection to relevant components on the FPGA board.

TaPaSCo can support developers in all steps of the development process of heterogeneous systems:

• TaPaSCo Toolflow: from cores resulting from High-Level Synthesis or cores manually written in an HDL, a complete FPGA-design can be created. TaPaSCo will automatically connect all processing elements to the memory- and host-interface and generate a complete bitstream.

• TaPaSCo Runtime API: allows to interface with accelerator from software and supports operations such as transferring data to the FPGA memory, pass values to accelerator cores and control the execution of the processing elements.

Next to the setup and usage instructions in this README, you can find additional information about TaPaSCo in the tutorial videos and the scientific publications describing and using TaPaSCo.

We welcome contributions from anyone interested in this field, check the contributor's guide for more information.

Supported FPGA devices

• Zynq-based: PYNQ-Z1, ZC706, ZedBoard, Ultra96V2, ZCU102
• PCIe cards: VC709, NetFPGA-SUME, VCU108, VCU118, VCU1525, Alveo U250, Alveo U280, BittWare XUP-VVH, PRO DESIGN HAWK, VCK5000

System Requirements

TaPaSCo is known to work in this environment:

• Intel x86_64 arch
• Linux kernel 4.4+
• CentOS 8, Fedora 30+, Ubuntu 16.04+
• Fedora 24/25 does not support debug mode due to GCC bug
• Bash Shell 4.2.x+

Other setups likely work as well, but are untested.

Prerequisites for Toolflow

To use TaPaSCo, you'll need working installations of

• Vivado Design Suite 2017.4 or newer
• Java SDK 8 - 11
• git
• python3
• GCC newer than 5.x.x for C++11 support
• OPTIONAL: Local Installation of gradle 5.0+, if you do not want to use the included wrapper.

If you want to use the High-Level Synthesis flow for generating custom IP cores, you will also need:

• Vivado HLS 2017.4+ or Vitis HLS 2020.2+

Check that at least the following are in your $PATH:

• vivado - If not source path/to/vivado/settings64.sh
• git
• bash
• [vivado_hls,vitis_hls] - Since Vivado 2018.1 this is included in vivado

When using Ubuntu, ensure that the following packages are installed:

• unzip
• zip
• git
• findutils
• curl
• default-jdk

apt-get -y install unzip git zip findutils curl default-jdk

When using Fedora, ensure that the following packages are installed:

• which
• java-openjdk
• findutils

dnf -y install which java-openjdk findutils

Prerequisites for Simulation

• Vivado Design Suite 2021 or newer
• Questa Simulator 2021 or newer
• python3
• pip3

TaPaSCo-Toolflow Setup

Using the prebuilt packages, the setup of TaPaSCo is very easy:

1. Create or open a folder, which you would like to use as your TaPaSCo workspace. Within this folder, run the TaPaSCo-Initialization-Script which is located in /opt/tapasco/tapasco-init-toolflow.sh. This will setup your current folder as TAPASCO_WORK_DIR. It will also create the file tapasco-setup.sh within your current directory.
2. Source tapasco-setup.sh.

If you want to use a specific (pre-release) version or branch, you can do the following:

1. Clone TaPaSCo: git clone https://github.com/esa-tu-darmstadt/tapasco.git
2. Optionally Checkout a corresponding branch: git checkout <BRANCH>
3. Create or open a folder, which you would like to use as your TaPaSCo workspace. Within this folder, run the TaPaSCo-Initialization-Script tapasco-init.sh which is located in the root-folder of your cloned repo. This will setup your current folder as TAPASCO_WORK_DIR. It will also create the file tapasco-setup.sh within your workdir.
4. Source tapasco-setup.sh to setup the TaPaSCo-Environment.
5. Build the TaPaSCo-Toolflow using tapasco-build-toolflow.

Whenever you want to use TaPaSCo in the future, just source the corresponding workspace using the tapasco-setup.sh. This also allows you to have multiple independent TaPaSCo-Workspaces.

Prerequisites for compiling the runtime

Ubuntu:

apt-get -y install build-essential linux-headers-generic python3 cmake libelf-dev git rpm protobuf-compiler

Fedora:

dnf -y install kernel-devel make gcc gcc-c++ elfutils-libelf-devel cmake python3 libatomic git rpm-build protobuf-compiler

Arch:

pacman -S linux-headers make gcc libelf libatomic_ops cmake python3 git protobuf

Rust:

The runtime uses Rust and requires a recent version of it. The versions provided by most distributions is too old. We recommend the official way of installing Rust through rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o /tmp/rustup.sh && sh /tmp/rustup.sh -y
source ~/.cargo/env

TaPaSCo-Runtime Setup

If you want to use a specific (pre-release) version or branch, you can do the following:

1. Clone TaPaSCo: git clone https://github.com/esa-tu-darmstadt/tapasco.git
2. Optionally Checkout a corresponding branch: git checkout <BRANCH>
3. Create or open a folder, which you would like to use as your TaPaSCo workspace. Within this folder, run the TaPaSCo-Initialization-Script tapasco-init.sh which is located in the root-folder of your cloned repo. This will setup your current folder as TAPASCO_WORK_DIR. It will also create the file tapasco-setup.sh within your workdir.
4. Source tapasco-setup.sh to setup the TaPaSCo-Environment.
5. Build the TaPaSCo-Toolflow using tapasco-build-libs.
6. Optionally If you want to use the simulation features build the Toolflow using tapasco-build-libs -s. Currently the only supported simulator is Questa.

All of this is not necessary when using the prebuilt packages. In that case, the corresponding libraries and files are installed as usual for your OS. Simulation support is currently not available with prebuilt packages.

Getting Started - Build a TaPaSCo design

1. Import your kernels
   • HDL flow: tapasco import path/to/ZIP as <ID> -p <PLATFORM> will import the corresponding ZIP file as a new HDL-based core. The Kernel-ID is set from and the optional flag -p <PLATFORM> determines for which platform the kernel will be available. If it is omitted, it will be made available for all platforms which may take a lot of time.
   • HLS flow: tapasco hls <KERNEL> -p <PLATFORM> will perform hls according to the kernel.json. The resulting HLS-based core will be made available for the platform given by -p <PLATFORM>. Again, -p can be omitted. HLS-Kernels are generally located in $TAPASCO_WORKDIR/kernel. If you want to add kernels you can create either symlink or copy them into the folder. Additionally, the folder can be temporarily changed using the optional --kernelDir path/to/kernels flag like this: tapasco --kernelDir path/to/kernels hls <KERNEL> -p <PLATFORM>
2. Create a composition: tapasco compose [<KERNEL> x <COUNT>] @ <NUM> MHz -p <PLATFORM>
3. Load the bitstream: tapasco-load-bitstream <BITSTREAM>
4. Implement your host software
   • C API
   • C++ API

You can get more information about commands with tapasco --help and the corresponding subpages with tapasco --help <TOPIC>

Getting Started - Build a Software-Interface

1. Design your Accelerator using HLS/HDL according to the previous section.
2. Load your bitstream: tapasco-load-bitstream my-design.bit --reload-driver. To do this, you have to source vivado and tapasco-setup.sh.
3. Write a C/C++ executable that interfaces with your design accordingly. To get a better understanding of this, you might want to refer to the collection of examples and the corresponding README which is located in $TAPASCO_HOME/runtime/examples
4. Build and Compile your Software.

Getting Started - Build a Boot Image

This repository provides a script to generate boot images for some common AMD Xilinx rSoC boards. Refer to the dedicated README for more information.

Using the Simulation

1. Design the Accelerator using HLS/HDL for the platform sim.
2. The TaPaSCo-Toolflow will generate a ZIP-file in place of a bitstream.
3. Load the design and start the simulation with tapasco-start-sim path/to/zip
   • After being started, the simulation can be stopped using CTRL-C
   • Consequent simulations of that design can be started simply using tapasco-start-sim
   • New designs need to be loaded again by following step 3.
   • The currently used Questa-Simulator offers the option to interact with the simulation via a GUI. To start the simulation in GUI-mode the flag --gui needs to be used.
   • To speedup datatransfers to the simulated design the flag --unsafe-sim can be used. Using this flag, the simulation will allow multiple write requests to be scheduled by the host software disabling the possibility to match errors in the simulator to the coresponding write-request in the host software.
4. By default the simulation listens on port 4040 for incoming connections of the TaPaSCo-Runtime. Before starting your software-interface, make sure that it can connect to the simulation. You can specify that port in the runtime by setting the environment variable SIM_PORT and by using the flag --sim-port with the tapasco-start-sim command.
   • If simulation and your software-interface are running on the same host, there shouldn't be an issue
   • If simulation and software-interface are running on different host, port 4040 can be forwarded via ssh using ssh -L 4040:localhost:4040 simulation-host on the host, where the software-interface should run.
5. Run your Software
   • Make sure to select the correct TaPaSCo kernel-device in your software when instantiating the Tapasco Class/Structure.

Acknowledgements

TaPaSCo is based on ThreadPoolComposer, which was developed by us as part of the REPARA project, a Framework Seven (FP7) funded project by the European Union.

We would also like to thank Bluespec, Inc. for making their Bluespec SystemVerilog (BSV) tools available to us and their permission to distribute the Verilog code generated by the Bluespec Compiler (bsc).

Publications

A List of publications about TaPaSCo or TaPaSCo-related research can be found here.

If you want to cite TaPaSCo, please use the following information:

[Heinz2021a] Heinz, Carsten, Jaco Hofmann, Jens Korinth, Lukas Sommer, Lukas Weber, and Andreas Koch. 2021. The Tapasco Open-Source Toolflow. In Journal of Signal Processing Systems.

Releases

We provided pre-compiled packages for many popular Linux distributions. All packages are build for the x86_64 variant.

Distribution	Kernel Driver	Kernel Driver (Debug)	Runtime	Runtime (Debug)	Toolflow
Ubuntu 18.04	Download	Download	DEB	DEB	DEB
Ubuntu 20.04	Download	Download	DEB	DEB	DEB
Ubuntu 22.04	Download	Download	DEB	DEB	DEB
Rocky Linux 8	Download	Download	RPM	RPM	RPM
Fedora 36	Download	Download	RPM	RPM	RPM