Popcorn Linux Compiler Toolchain, Copyright Systems Software Research Group at
Virginia Tech, 2017.

For more information, please visit http://popcornlinux.org or e-mail Rob Lyerly
(rlyerly@vt.edu).

--------
Overview
--------

The goal of the heterogeneous compiler toolchain is to prepare multi-ISA
binaries for migration through a series of analyses and transformations.  We
utilize and extend clang/LLVM in order to prepare heterogeneous binaries.  We
also use a Java-based tool to prepare custom linker scripts to align the
generated binaries.  Finally, there are a number of additional libraries needed
for migration and runtime state transformation.

We need to prepare the binary so that before and after migrating between
architectures, the application is able to find the required code and data to
seamlessly continue execution.  This is done by both using a common layout
(where it is possible without significant performance overhead) and state
transformation (where state is dictated by the ISA or where a common format
would be too costly in performance).

The toolchain operates as follows:

1. Parsing/LLVM bitcode generation (clang) - clang frontend, mostly unmodified
   except to pass the generated LLVM IR to multiple ISA-specific backends

2. Middle-end analysis, refactoring & optimization - in addition to standard
   optimizations, run several passes which adjust the linkage of some variables
   and notify architecture-specific backends to generate stack frame metadata

3. Backend - modified LLVM backend(s) generate custom data and function
   location information and compile bitcode to object-code

4. Linker - modified gold linker generates detailed data and function linking
   information

5. Alignment - Java tool uses information provided by gold linker to generate
   linker scripts that align data & function symbols across binaries

6. Metadata Generation - parse information generated by LLVM backends and add
   stack transformation metadata to binaries (must be run post-alignment as it
   requires final symbol layouts!)

In order to have functionally-identical implementations for all compiled source
code, the binary must be compiled into a single IR representation, which is
then used by each of the backends to generate architecture-specific code:

                           ----------------
                           | Orig. Source |
                           ----------------
                                   |
                                   | (clang)
                                   V
                           ----------------
                           |    LLVM IR   |
                           ----------------
                                   |
                                   | (opt)
                                   V
                           ----------------
                           | Optimized IR |
                           ----------------
                                   |
                   ---------------------------------  (arch-specific backend)
                   |               |               |
                   V               V               V
            ---------------                 ---------------
            | aarch64 bin |       ...       |   x86 bin   |
            ---------------                 ---------------

Therefore, we compile to LLVM IR using aarch64 as the target (as x86-64 is much
more flexible and will be able to successfully codegen any IR generated for
aarch64) and pass the single set of IR through each of the backends.

The toolchain's installation folder is organized as shown below, along with a
brief introduction about important sub-folders:

root
 \
  common - source/headers common between components in different folders
  |
  lib - libraries needed by the compiler and/or the compiled application
   \
    passes - passes needed by LLVM's middle-end
  |
  patches - patches for LLVM & gold
  |
  tool - post-compilation binary tools for alignment & metadata-generation
  |
  util - various utilities, including a Makefile template, for heterogeneous
         applications
   \
    scripts - scripts for patch generation, testing and running binaries

See the README file in each subdirectory for more information, and the INSTALL
file for installation instructions.

-------------
Prerequisites
-------------

Note: the toolchain has been tested on x86-64 with Ubuntu 14.04 and up.  It
should work on other architectures and distributions, but may require
installing alternate packages and some extra hacking.  It is *highly*
recommended that applications be built on x86-64.

+ gcc and g++ 4.8 or higher, for both aarch64 & x86-64
  - On Ubuntu 14.04 & higher, x86-64, install the following packages:
      build-essential
      gcc-aarch64-linux-gnu
      g++-aarch64-linux-gnu
+ flex & bison (x86-64 only), needed by binutils
  - On Ubuntu 14.04 & higher, x86-64, install the following packages:
      flex
      bison
+ OpenJDK, version 1.7 or higher
  - On Ubuntu 14.04 & higher, x86-64, install the following packages:
      default-jre
      default-jdk
+ Python, v2.7 and v3
  - v2 is needed for installation script
  - v3 is needed for scripts in util/scripts
  - Both are pre-installed on Ubuntu 14.04
+ Cmake
  - On Ubuntu 14.04 & higher, x86-64, install the following packages:
      cmake
+ SVN
  - On Ubuntu 14.04 & higher, x86-64, install the following packages:
      subversion

-----------
Limitations
-----------

The current version of the toolchain only works for aarch64 + x86-64.
Applications must be written in C (although anything for which LLVM IR can be
generated should be supported), and inline assembly is not supported.  All
optimizations except auto-vectorization and frame-pointer elimination (required
by stackmap intrinsic) are supported.