Skip to content

Latest commit

 

History

History
385 lines (334 loc) · 21.3 KB

Installation.md

File metadata and controls

385 lines (334 loc) · 21.3 KB
Raw

Installation

[TOC]

Document Sections

  • Requirements
  • Install using CMake: the preferred way
  • Install with (basic) GNU Make
  • Install with Python Pip
  • Install with Spack
  • Install on MS Windows platform
  • Install folder structure
  • Linking to Tasmanian: CMake Package Config
  • Known issues

Requirements

Minimum requirements to use Tasmanian:

Recommended additional features:

Optional features:

Note: with the exception of the MAGMA library and some of the more exotic compilers, the rest of the software is included in most Linux distributions (or HomeBrew) or there are vendor supported software repositories.

Feature Tested versions Recommended
gcc 11 - 13 any
clang 14 - 18 any
cmake 3.19 - 3.25 3.22
python 3.10 - 3.12 any
anaconda 5.3 5.3
OpenBlas 0.2.18 - 3.08 any
ESSL 6.2 6.2
CUDA 11.8 - 12.6 avoid 12.3-12.4
ROCm 5.7 - 6.2 any
DPC++ 2024.2 2024
libiomp 5.0 5.0
MAGMA 2.5.1 - 2.6.1 2.6.1
Doxygen 1.8.13 1.8.13
MPI 2.1, 3.1 3.1

Install using CMake: the preferred way

The preferred way to install Tasmanian is to use the included CMake build script, which requires CMake version 3.19, ROCm capabilities require CMake 3.21.

  • The commands for an out-of-source CMake build:
  mkdir Build
  cd Build
  cmake <options> <path-to-Tasmanian-source>
  make
  make test
  make install
  make test_install
  • Standard CMake options are accepted:
  -D CMAKE_INSTALL_PREFIX:PATH=<install-prefix> (install folder for the make install command)
  -D CMAKE_BUILD_TYPE:STRING=<Debug/Release>    (set debug flags or default optimization flags)
  -D BUILD_SHARED_LIBS:BOOL=<ON/OFF>            (pick shared or static libs)
  -D CMAKE_CXX_COMPILER:PATH=<path>             (specify the C++ compiler, or HIP/ROCm compiler)
  -D CMAKE_CUDA_COMPILER:PATH=<path>            (specify the CUDA nvcc compiler)
  -D CMAKE_CXX_FLAGS:STING=<flags>              (set additional flags)
  • List of Tasmanian specific CMake options (all default to OFF):
  -D Tasmanian_ENABLE_OPENMP:BOOL=<ON/OFF>      (recommended)
  -D Tasmanian_ENABLE_BLAS:BOOL=<ON/OFF>        (recommended)
  -D Tasmanian_ENABLE_PYTHON:BOOL=<ON/OFF>      (recommended)
  -D Tasmanian_ENABLE_RECOMMENDED:BOOL=<ON/OFF> (enable the above and the -O3 flag)
  -D Tasmanian_ENABLE_CUDA:BOOL=<ON/OFF>        (stable)
  -D Tasmanian_ENABLE_HIP:BOOL=<ON/OFF>         (stable)
  -D Tasmanian_ENABLE_DPCPP:BOOL=<ON/OFF>       (stable)
  -D Tasmanian_ENABLE_MAGMA:BOOL=<ON/OFF>       (stable)
  -D Tasmanian_MATLAB_WORK_FOLDER:PATH=""       (stable)
  -D Tasmanian_ENABLE_DOXYGEN:BOOL=<ON/OFF>     (stable)
  -D Tasmanian_ENABLE_FORTRAN:BOOL=<ON/OFF>     (stable)
  -D Tasmanian_ENABLE_MPI:BOOL=<ON/OFF>         (stable)

  • Acceleration options:

    • OpenMP allows Tasmanian to use more than one CPU core, which greatly increases the performance. While many of the Tasmanian algorithms have been parallelized, the buildin C++ algorithms are usually sequential. This is most notable in the case of std::sort but affects others as well. Some compilers support parallel standard algorithms but those in turn reuqire additional compiler flags.
      • for GCC add -D_GLIBCXX_PARALLEL to the CMAKE_CXX_FLAGS
    • Basic Linear Algebra Subroutines (BLAS) is a standard with many implementations, e.g., https://www.openblas.net/; optimized BLAS improves the performance when using evaluate commands on grids with many points or working with models with many outputs
    • Linear Algebra PACKage (LAPACK) is a set of advanced solver, eigen-solvers and decomposition methods that build upon BLAS and is usually included in the same package, e.g., OpenBLAS, MKL and ESSL; within Tasmanian, the name BLAS in CMake or run-time options indicate the dependence and usage of both BLAS and LAPACK
    • CUDA is a C++ language extension that allows Tasmanian to leverage the computing power of Nvidia GPU devices, which greatly enhances the performance of evaluateFast() and evaluateBatch() and a few other calls
    • HIP/ROCm is very similar to CUDA but uses AMD GPU devices instead, Tasmanian supports a HIP backend
    • SYCL/DPC++/OneAPI is the Intel alternative to CUDA and HIP, Tasmanian supports a SYCL/DPC++ backend
    • Matrix Algebra on GPU and Multicore Architectures (MAGMA) is a library for GPU accelerated linear algebra developed at the University of Tennessee at Knoxville
    • MPI allows the use of distributed memory in Bayesian inference, parallel surrogate model construction, and send/receive grid through an MPI communicators

  • The Tasmanian_ENABLE_RECOMMENDED option searches for OpenMP, BLAS, and Python, enables the options (if possible) and also sets the -O3 flag

  • Tasmanian_ENABLE_FORTRAN enables the Fortran 2003 interface

    • Fortran 90 interface is deprecated, but can be enabled with -DTasmanian_ENABLE_FORTRAN90=ON

  • Additional interfaces are available, beyond the default C/C++ library and the command line tools:

    • Python and Fortran require appropriate interpreter and compiler
    • The MATLAB/Octave interface requires a work-folder with read/write permission for temporary files. The interface is enabled by setting Tasmanian_MATLAB_WORK_FOLDER to a valid read/write location.

  • The Doxygen option will build the HTML documentation

  • Options to adjust the testing environment: by default Tasmanian testing will use the system provided OpenMP parameters and run tests on all visible GPU devices; specific number of threads and device can be selected (note that only the testing environment is affected):

 -D Tasmanian_TESTS_OMP_NUM_THREADS=<number-of-threads-for-testing> (only used with OpenMP)
 -D Tasmanian_TESTS_GPU_ID=<cuda-device-id-for-testing>             (only used with CUDA)
  • Additional commands to guide the CMake find_package() modules:
  -D Python_EXECUTABLE:PATH            (specify the Python interpreter)
  -D CMAKE_CUDA_COMPILER:PATH          (specify the CUDA nvcc compiler)
  -D Tasmanian_ENABLE_ROCM:PATH        (specify the search path for HIP)
  -D CMAKE_Fortran_COMPILER:PATH       (specify the Fortran compiler)
  -D MKLROOT:PATH                      (specify the path to the oneMKL installation)
  -D Tasmanian_MAGMA_ROOT:PATH         (specify the path to the MAGMA installation)
  -D MPI_CXX_COMPILER:PATH=<path>      (specify the MPI compiler wrapper)
  -D MPI_Fortran_COMPILER:PATH=<path>  (needed for MPI with Fortran)
  -D MPIEXEC_EXECUTABLE:PATH=<path>    (needed for MPI testing)

  • The MAGMA option can be passed without the Tasmanian_ prefix and if not specified explicitly, it will be read from the OS environment.

  • Option for automated download of MAGMA, the package will be downloaded, build, and installed together with Tasmanian in the same prefix.

  -D Tasmanian_MAGMA_DOWNLOAD:BOOL=<ON/OFF> (works only on Linux)

  • The ROCm capabilities NO LONGER require the CMake CXX compiler to be set to hipcc.

    • by default, Tasmanian will search for hip in /opt/rocm /opt/rocm/hip following the ROCm documentation
    • additional search paths can be added using CMAKE_PREFIX_PATH
    • the CXX compiler, if not set automatically, should be set to ROCm clang, e.g., /opt/rocm/llvm/bin/clang++
    • if Tasmanian_ENABLE_ROCM is defined and set to a directory, it will be added to CMAKE_PREFIX_PATH to search for roc::hip

  • The OneAPI capabilities require:

    • the compiler has to be set to Intel ipcx, Tasmanian will automatically add the -fsycl flag (when needed)
    • Tasmanian_ENABLE_BLAS is set to ON
    • BLAS is set to the CPU version of MKL, e.g., using BLAS_LIBRARIES or BLA_VENDOR or MKLROOT environment variable

  • Alternatives allowing to directly specify libraries and bypass find_package() altogether:

  -D BLAS_LIBRARIES
  -D LAPACK_LIBRARIES
  • Extra options are available in case CMake fails to find a required dependency, e.g., find_package() sometimes fails to acknowledge that the ACML implementation of BLAS depends on both libgfortran and libgomp; the manual options below should not be necessary in the majority of cases:
  -D Tasmanian_EXTRA_LIBRARIES:LIST     (add more libraries as dependencies)
  -D Tasmanian_EXTRA_INCLUDE_DIRS:PATH  (add more include paths to search for headers)
  -D Tasmanian_EXTRA_LINK_DIRS:PATH     (appends more link paths to search for libraries)
  • Options helpful to Tasmanian developers:
  -D DOXYGEN_INTERNAL_DOCS=YES  (include the documentation of the Tasmanian internals)

Install with (basic) GNU Make

The core capabilities of Tasmanian can be build with a few simple GNU Make commands. The basic build engine is useful for quick testing and exploring Tasmanian, or if CMake is unavailable or unwanted. Acceleration options other than OpenMP are not supported in the basic mode, CMake is so common these days that it should be the default way to install.

  • Using GNU Make with g++ and optionally /usr/bin/env python
  make
  make test     (will fail if /usr/bin/env python is missing the numpy or ctypes modules)
  make matlab   (optional: sets matlab work folder to ./tsgMatlabWorkFolder/)
  make examples
  make clean

In the basic mode, the source folder will become the installation folder, i.e., the libraries, executables and Python modules will be build in the source folder and the headers will be copied to the include sub-folder.

Install with Python Pip

Tasmanian is included in the Python Pip index: https://pypi.org/project/Tasmanian/

  python3 -m pip install Tasmanian --user   (user installation)
  python3 -m pip install Tasmanian          (virtual env installation)

The Tasmanian module is not a regular Python-only project but a wrapper around the C++ libraries, note the following:

  • The compiled binaries are cached by pip and --no-cache-dir option must be used to change the options.
  • Only user installations are supported, installation for all users is possible with CMake but not Pip.
  • Python virtual environments are supported, as well as Linux, Mac and Windows operating systems.
  • By default, Tasmanian will install with Tasmanian_ENABLE_RECOMMENDED=ON which will safe-auto-enable BLAS and OpenMP
    • environment variable (see below) can be used to disable the recommended options and switch to manual enable/disable
  • The Pip installer will accept Tasmanian options specified in the environment variables:
Environment Option will translate to               CMake Options
export Tasmanian_ENABLE_RECOMMENDED=OFF            -D Tasmanian_ENABLE_RECOMMENDED=OFF

export Tasmanian_ENABLE_BLAS=<blas-lapack-libs>    -D Tasmanian_ENABLE_BLAS=ON
                                                   -D BLAS_LIBRARIES=<blas-lapack-libs>
                                                   -D LAPACK_LIBRARIES=<blas-lapack-libs>
export Tasmanian_ENABLE_CUDA=<cuda-nvcc>           -D Tasmanian_ENABLE_CUDA=ON
                                                   -D CMAKE_CUDA_COMPILER=<cuda-nvcc>
export Tasmanian_ENABLE_DPCPP=<dpcpp>              -D Tasmanian_ENABLE_DPCPP=ON
                                                   -D CMAKE_CXX_COMPILER=<dpcpp>
export Tasmanian_ENABLE_MAGMA=<magma-root>         -D Tasmanian_ENABLE_MAGMA=ON
                                                   -D Tasmanian_MAGMA_ROOT_DIR=<magma-root>
export Tasmanian_ENABLE_MPI=<mpicxx>               -D Tasmanian_ENABLE_MPI=ON
                                                   -D MPI_CXX_COMPILER=<mpicxx>
export Tasmanian_MATLAB_WORK_FOLDER=<path>         -D Tasmanian_MATLAB_WORK_FOLDER=<path>
  • CMake also accepts several standard environment variables:
Environment Option          Example values
export BLA_VENDOR           OpenBLAS or MKL
export MKLROOT              /opt/intel/oneapi/mkl/latest
  • note that the Tasmanian specific environment variables work only with Python Pip
  • Example virtual install looks like this:
python3 -m venv tasmanian_virtual_env                   # create a virtual environment
source ./tasmanian_virtual_env/bin/activate             # activate the virtual environment
export Tasmanian_ENABLE_CUDA=/usr/local/cuda/bin/nvcc   # specify the CUDA compiler
python -m pip install Tasmanian --no-cache-dir          # will install Tasmanian with CUDA
python -m Tasmanian                                     # print the install log
tasgrid -v                                              # print the available CUDA devices

Additional notes:

  • under MS Windows the environment variables can be set from Advanced System Settings, but the paths should use Linux style of back-slashes, e.g., C:/Program Files/CUDA/bin/nvcc.exe
  • scikit build sometimes lags in support for the latest MS Visual Studio, regular CMake install works fine
  • under OSX some users have reported segfaults when using a pip install, the problem does not appear when using CMake, the issue is under investigation

Install with Spack

Tasmanian is also included in Spack: https://spack.io/

 spack install tasmanian+openmp+blas+cuda+magma+python+fortran

Install on MS Windows platform

Tasmanian has been tested with MS Visual Studio 2019.

  • First use the CMake GUI to set the folders and options
  • Then use the command prompt (cmd.exe) to enter the build folder
  cd <cmake-build-folder>
  cmake --build . --config Release
  ctest -C Release
  cmake --build . --config Release --target install
  • Both Debug and Release are supported config modes, but do not use them simultaneously, pick only one Release or Debug.

Install on Mac platform

As of Tasmanian 8.1, the Mac OSX installation handles the same as Linux regardless for both Intel and M-chips.

Install folder structure

Tasmanian follows standard Linux conventions, the install path could be set to /usr/local/, although it is recommended to install in a location inside the user home folder to avoid potential system-wide conflicts.

  • Install folder structure:
  <install-path>/bin/                       (tagrid executable tool)
  <install-path>/lib/                       (shared or static libraries)
  <install-path>/lib/Tasmanian/             (cmake package-config files)
  <install-path>/lib/pythonX.Y/             (python module)
  <install-path>/include/                   (headers .h and .hpp, and Fortran .mod)
  <install-path>/share/Tasmanian            (bash env scripts, install log, table)
  <install-path>/share/Tasmanian/examples/  (reference examples)
  <install-path>/share/Tasmanian/matlab/    (matlab scripts)
  <install-path>/share/Tasmanian/python/    (copy of <install-path>/lib/pythonX.Y)
  <install-path>/share/Tasmanian/testing/   (a CMake script for post-install testing)

Additional notes:

  • A summary of all compile options is stored in
  <install-path>/share/Tasmanian/Tasmanian.log
  • The log file can be displayed with wither command:
  <install-path>/bin/tasgrid -log  (using the command line tool)
  python -m Tasmanian              (only if Python is enabled)
  • The executable and library paths, as well as the Python path can be set in bash by sourcing
  source <install-path>/share/Tasmanian/TasmanianENVsetup.sh
  • When using pip, the python module location may be different depending on the OS and the Python virtual environment.
  • The Python module is version independent, i.e., the files work with all tested Python versions; the share/Tasmanian/python copy allows for easy and version independent access to the Tasmanian modules.
  • Under MS Windows the shared libraries (e.g., the .dll files) are installed in bin.

Linking to Tasmanian: CMake Package Config

Tasmanian will install CMake package-config files in <install-path>/lib/Tasmanian, the files will contain all necessary information to import the Tasmanian targets into other CMake projects using the CMake command:

  find_package(Tasmanian 8.1 PATHS "<install-path>")

See the included CMakeLists.txt in <install-path>/share/Tasmanian/examples.

Note that the PATHS do not have to be specified explicitly if the TasmanianENVsetup.sh is sourced or if the Tasmanian_ROOT environment variable is set. The correct find_package() command is displayed in the log (see the previous section).

The imported targets will be named:

  Tasmanian::Tasmanian  (links to the C++ libraries)
  Tasmanian::tasgrid    (imported executable pointing to the command line tool)
  Tasmanian::Fortran    (links to the C++ libraries and the Fortran wrappers)

In addition, the following variables will be set:

  Tasmanian_PYTHONPATH         (path to the Python module, if Python was enabled)
  Tasmanian_MATLAB_WORK_FOLDER (path to the MATLAB work folder, if set during build)
  Tasmanian_MATLABPATH         (path to the MATLAB scripts, if MATLAB was enabled)
  Tasmanian_<component>_FOUND  (set to ON for each available component)

The possible components are:

  SHARED STATIC OPENMP BLAS CUDA HIP DPCPP MAGMA MPI PYTHON MATLAB FORTRAN

The modules correspond to shared and static libraries and the cmake options used during build.

All available components will be included even if the component is not explicitly requested. Requesting components can help catch errors early in the build process and/or print useful log messages. For example:

  find_package(Tasmanian 8.1 REQUIRED SHARED PYTHON CUDA OPTIONAL_COMPONENTS OPENMP)

In the above example:

  • an error will be generated if Tasmanian was build with static libraries, no CUDA or no Python support
  • a status message will report whether Tasmanian was build with OpenMP support
  • requesting incompatible components will always fail, e.g., CUDA with ROCM or SHARED and STATIC

Known Issues

Several known issues and work-around fixes:

  • Apple OSX has poor support for OpenMP, especially when using M-chips (apple silicon)
    • OpenMP is not supported by the default compiler (xcode)
    • gcc with OpenMP support can be installed through Homebrew, but the performance gains from multi-threading are low
  • the automated MAGMA download reports the wrong GPU architecture enabled
    • ignore the MAGMA message the build correctly respects CMAKE_CUDA_ARCHITECTURES
  • The addon tests sometime fail due to thread scheduling
    • The overhead associated with thread scheduling is much larger than the simple test models used, which leads to unrealistically large fluctuations in sample run-time, which in turn leads to randomness in the results, most notably on machines with few cpu cores.
    • Rerun the tests and/or installation to see if the problem is persistent
  • Mixing the GCC and Clang compilers and linkers sometimes fails with an error about the architecture
    • use shared libraries only, i.e., -D BUILD_SHARED_LIBS=ON in CMake
  • Some older versions of the PGI compiler fails when using optimization -O2
    • use the -O1 instead, or the newest version of the compiler
  • CUDA 12.3 and 12.4 seem to have a bug in cuSparse
    • whenever possible, avoid those versions of CUDA