Arkouda Prerequisite & Client Install Guide

This guide will walk you through the environment configuration for Arkouda to operate properly. Arkouda can be run on Linux, Windows, and MacOS.

There is no Linux Chapel install, so the first two steps in the Linux Arkouda install are to install the Chapel dependencies followed by downloading and building Chapel. The Ubuntu and RHEL Chapel installations are different for installing Chapel dependencies, particularly regarding older RHEL distro versions such as CentOS 7. Specifically, the gcc compiler on RHEL distros such as CentOS 7 do not support building Chapel. Consequently, a newer version of the gcc compiler must be installed via the devtoolset-9-gcc-c++ package. In addition, the CentOS Software Collections packager must be installed to enable the newer version of gcc to be leveraged for building Chapel.

Install Chapel (Ubuntu)^{^_toc}

# Update Linux kernel and install Chapel dependencies
sudo apt-get update
sudo apt-get install gcc g++ m4 perl python3 python3-pip python3-venv python3-dev bash make mawk git pkg-config cmake llvm-12-dev llvm-12 llvm-12-tools clang-12 libclang-12-dev libclang-cpp12-dev libedit-dev

# Download latest Chapel release, explode archive, and navigate to source root directory
wget https://github.com/chapel-lang/chapel/releases/download/1.27.0/chapel-1.27.0.tar.gz
tar xvf chapel-1.27.0.tar.gz
cd chapel-1.27.0/

# Set CHPL_HOME
export CHPL_HOME=$PWD

# Add chpl to PATH
source $CHPL_HOME/util/setchplenv.bash

# Set remaining env variables and execute make
# It is recommended to add these variables to a ~/.chplconfig file to prevent having 
# to export them again
export CHPL_COMM=gasnet
# If you're going to be running locally only, use CHPL_COMM=none instead to improve build time
export CHPL_COMM_SUBSTRATE=smp
export CHPL_TARGET_CPU=native
export GASNET_QUIET=Y
export CHPL_RT_OVERSUBSCRIBED=yes
export CHPL_RE2=bundled
export CHPL_LLVM=bundled

# Build Chapel
cd $CHPL_HOME
make

# Build chpldoc to enable generation of Arkouda docs
make chpldoc

# Optionally add the Chapel executable (chpl) to the PATH for all users: /etc/environment
export PATH=$CHPL_HOME/bin/linux64-x86_64/:$PATH

Install Chapel (RHEL)^{^_toc}

For all RHEL distros, Chapel dependencies are installed as follows:

yum update -y && yum install gcc gcc-c++ m4 perl python3 python3-devel \
make gawk git cmake llvm-devel clang clang-devel curl-devel -y

For older RHEL distros with incompatible gcc compiler versions, the following dependencies must be installed:

yum install devtoolset-9-gcc-c++-9.1.1-2.6.el7.x86_64 centos-release-scl -y

The RHEL Chapel build instructions are the same for all distros with the exception of older distros such as CentOS 7; for these, the newer gcc compiler must first be enabled:

source /opt/rh/devtoolset-9/enable

The remaining RHEL Chapel download and build instructions follow those detailed above for Ubuntu Linux.

Python Environment - Anaconda (Linux)^{^_toc}

As is the case with the MacOS install, it is highly recommended to install Anaconda to provide a Python environment and manage Python dependencies:

 wget https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
 sh Anaconda3-2020.07-Linux-x86_64.sh
 source ~/.bashrc
 
# User conda env
conda env create -f arkouda-env.yml
conda activate arkouda

# Developer conda env
conda env create -f arkouda-env-dev.yml
conda activate arkouda-dev

#These packages are not required, but nice to have (these are included with Anaconda3)
conda install jupyter

# Install the Arkouda Client Package
pip install -e . --no-deps

Installation of Arkouda Client Package and dependencies if choosing to not use Anaconda

# without developer dependencies
pip install -e .

# With developer dependencies
pip install -e .[dev]

Windows (WSL2) ^{^_toc}

It is possible to set up a basic arkouda installation on MS Windows using the Windows Subsystem for Linux (WSL2). The general strategy here is to use Linux terminals on WSL to launch the server. If you are going to try this route we suggest using WSL-2 with Ubuntu 20.04 LTS. There are a number of tutorials available online such has MicroSoft's

Key installation points:

Make sure to use WSL2
Ubuntu 20.04 LTS from the MS app store
Don't forget to create a user account and password as part of the Linux install

Once configured you can follow the basic Linux installation instructions for installing Chapel & Arkouda. We also recommend installing Anaconda for windows.

Note: When running make to build Chapel while using WSL, pathing issues to library dependencies are common. In most cases, a symlink pointing to the correct location or library will fix these errors.

An example of one of these errors found while using Chapel 1.27.0 and Ubuntu 20.04 LTS with WSL is:

../../../bin/llvm-tblgen: error while loading shared libraries: libtinfow.so.6: cannot open shared object file: No such file or directory

This error can be fixed by the following command:

sudo ln -s /lib/x86_64-linux-gnu/libtic.so.6.2 /lib/x86_64-linux-gnu/libtinfow.so.6

The general plan is to compile & run the arkouda-server process from a Linux terminal on WSL and then either connect to it with the python client using another Linux terminal running on WSL or using the Windows Anaconda-Powershell.

If running an IDE you can use either the Windows or Linux version, however, you may need to install an X-window system on Windows such as VcXsrv, X410, or an alternative. Follow the setup instructions for whichever one you choose, but keep in mind you may need to update your Windows firewall to allow the Xserver to connect. Also, on the Linux side of the house we found it necessary to add

export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2; exit;}'):0.0

to our ~/.bashrc file to get the display correctly forwarded.

MacOS ^{^_toc}

Prerequisites for Arkouda can be installed using Homebrew or manually. Both installation methods have variances to account for the chipset being run.

Homebrew Installation ^{^_toc}

Chapel Installation ^{^_toc}

brew install chapel

Python Environment ^{^_toc}

While not required, it is highly recommended to install Anaconda to provide a Python3 environment and manage Python dependencies. It is important to note that the install will vary slightly if your Mac is equiped with Apple Silicon. Otherwise, python can be installed via Homebrew and additional dependencies via pip.

Anaconda (Recommended) ^{^_toc}

Arkouda provides 2 .yml files for configuration, one for users and one for developers. The .yml files are configured with a default name for the environment, which is used for example interactions with conda. *Please note that you are able to provide a different name by using the -n or --name parameters when calling conda env create

#Works with all Chipsets (including Apple Silicon)
brew install miniforge
#Add /opt/homebrew/Caskroom/miniforge/base/bin as the first line in /etc/paths

#works with only x86 Architecture (excludes Apple Silicon)
brew install anaconda3
#Add /opt/homebrew/Caskroom/anaconda3/base/bin as the first line in /etc/paths

# User conda env
conda env create -f arkouda-env.yml
conda activate arkouda

# Developer conda env
conda env create -f arkouda-env-dev.yml
conda activate arkouda-dev

#These packages are not required, but nice to have (these are included with Anaconda3)
conda install jupyter

# Install the Arkouda Client Package
pip install -e . --no-deps

Python Only ^{^_toc}

brew install python3
#Add /opt/homebrew/Cellar/python@3.9/3.9.10/bin as the first line in /etc/paths

# Install Arkouda Client Package
# without developer dependencies
pip install -e .

# With developer dependencies
pip install -e .[dev]

Manual Installation ^{^_toc}

Chapel Installation ^{^_toc}

# build chapel in the user home directory with these settings...
export CHPL_HOME=~/chapel/chapel-1.27.0
source $CHPL_HOME/util/setchplenv.bash
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=smp
export CHPL_TARGET_CPU=native
export GASNET_QUIET=Y
export CHPL_RT_OVERSUBSCRIBED=yes
export CHPL_RE2=bundled
cd $CHPL_HOME
make

# Build chpldoc to enable generation of Arkouda docs
make chpldoc

# Add the Chapel and Chapel Doc executables (chpl and chpldoc, respectiveley) to 
# PATH either in ~/.bashrc (single user) or /etc/environment (all users):

#If on Apple Silicon
export PATH=$CHPL_HOME/bin/darwin-arm64:$PATH
#Otherwise
export PATH=$CHPL_HOME/bin/linux64-x86_64/:$PATH

Python Environment ^{^_toc}

While not required, it is highly recommended to install Anaconda to provide a Python3 environment and manage Python dependencies. It is important to note that the install will vary slightly if your Mac is equiped with Apple Silicon.

Anaconda ^{^_toc}

Anaconda Installs - Apple Silicon Compatible

Miniforge arm64

Anaconda Installs - x86 Chipsets

Minforge x86
Anaconda3 x86

Ensure Requirements are Installed:

#Works with all Chipsets (including Apple Silicon)
brew install miniforge
#Add /opt/homebrew/Caskroom/miniforge/base/bin as the first line in /etc/paths

#works with only x86 Architecture (excludes Apple Silicon)
brew install anaconda3
#Add /opt/homebrew/Caskroom/anaconda3/base/bin as the first line in /etc/paths

# User conda env
conda env create -f arkouda-env.yml
conda activate arkouda

# Developer conda env
conda env create -f arkouda-env-dev.yml
conda activate arkouda-dev

#These packages are not required, but nice to have (these are included with Anaconda3)
conda install jupyter

# Install the Arkouda Client Package
pip install -e . --no-deps

Python Only ^{^_toc}

Apple Silicon Compatible - Python3
x86 Compatible Only - Python3

Install the Arkouda Client Package and dependencies

# without developer dependencies
pip install -e .

# With developer dependencies
pip install -e .[dev]

Next Steps: ^{^_toc}

Now that you have Arkouda and its dependencies installed on your machine, you will need to be sure to have the appropriate environment variables configured. A complete list can be found at ENVIRONMENT.md.

Once your environment variables are configured, you are ready to build the server. More information on the build process can be found at BUILD.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

INSTALL.md

INSTALL.md

Arkouda Prerequisite & Client Install Guide

Table of Contents

Overview: ^{^_toc}

Requirements: ^{^_toc}

Linux^{^_toc}

Install Chapel (Ubuntu)^{^_toc}

Install Chapel (RHEL)^{^_toc}

Python Environment - Anaconda (Linux)^{^_toc}

Windows (WSL2) ^{^_toc}

MacOS ^{^_toc}

Homebrew Installation ^{^_toc}

Chapel Installation ^{^_toc}

Python Environment ^{^_toc}

Anaconda (Recommended) ^{^_toc}

Python Only ^{^_toc}

Manual Installation ^{^_toc}

Chapel Installation ^{^_toc}

Python Environment ^{^_toc}

Anaconda ^{^_toc}

Python Only ^{^_toc}

Next Steps: ^{^_toc}

Files

INSTALL.md

Latest commit

History

INSTALL.md

File metadata and controls

Arkouda Prerequisite & Client Install Guide

Table of Contents

Overview: toc

Requirements: toc

Linuxtoc

Install Chapel (Ubuntu)toc

Install Chapel (RHEL)toc

Python Environment - Anaconda (Linux)toc

Windows (WSL2) toc

MacOS toc

Homebrew Installation toc

Chapel Installation toc

Python Environment toc

Anaconda (Recommended) toc

Python Only toc

Manual Installation toc

Chapel Installation toc

Python Environment toc

Anaconda toc

Python Only toc

Next Steps: toc

Overview: ^{^_toc}

Requirements: ^{^_toc}

Linux^{^_toc}

Install Chapel (Ubuntu)^{^_toc}

Install Chapel (RHEL)^{^_toc}

Python Environment - Anaconda (Linux)^{^_toc}

Windows (WSL2) ^{^_toc}

MacOS ^{^_toc}

Homebrew Installation ^{^_toc}

Chapel Installation ^{^_toc}

Python Environment ^{^_toc}

Anaconda (Recommended) ^{^_toc}

Python Only ^{^_toc}

Manual Installation ^{^_toc}

Chapel Installation ^{^_toc}

Python Environment ^{^_toc}

Anaconda ^{^_toc}

Python Only ^{^_toc}

Next Steps: ^{^_toc}