Add support for OmniSciDB 5.2 (#307)

* Updates dependencies for 5.2

* Updates to reflect thrift namespace change (mapd -> omnisci)

* Conda environment to provide all development deps

  * Rather than having the user collect some deps from pip
     and others from conda, let's provide all of the necessary
     development and runtime deps since that is likely what
     the user of the environment will be expecting.

   * Make the supported Python version explicit.

* Adds alternative development env file for GPU

   A super set of environment.yml, this automatically sets up a
   development environment with cudf and others installed.

   A new conda environment is created named `omnisci-gpu-dev`. A benefit of
   this is it makes toggling between GPU and CPU capable environments
   easier.

* Updates docs to reflect usage of new environments.

   This sort of implicitly makes conda the recommended package manager
   since it makes it trivial to switch from GPU development to CPU
   development.

* Prefer the cudf provided interface where possible.

* Ensures cudf installed before performing GPU tests

* Adds workaround to support dicts on GPU with pyarrow 0.15

.gitignore

@@ -120,3 +120,5 @@ mapdql_history.txt
 
 # Folder metadata
 **/.DS_Store
+
+.vscode/

Jenkinsfile

@@ -1,7 +1,7 @@
 def flake8_container_image = "alpine/flake8:3.7.7"
 def flake8_container_name = "pymapd-flake8-$BUILD_NUMBER"
-//def db_container_image = "omnisci/core-os-cuda-dev:master"
-def db_container_image = "omnisci/core-os-cuda"
+def db_container_image = "omnisci/core-os-cuda-dev:master"
+//def db_container_image = "omnisci/core-os-cuda"
 def db_container_name = "pymapd-db-$BUILD_NUMBER"
 def testscript_container_image = "rapidsai/rapidsai:0.8-cuda10.0-runtime-ubuntu18.04-gcc7-py3.6"
 def testscript_container_name = "pymapd-pytest-$BUILD_NUMBER"

ci/install-test-deps-conda.sh

@@ -8,24 +8,27 @@ echo
 echo "[add channels]"
 conda config --add channels conda-forge
 conda config --add channels rapidsai
+conda config --add channels nvidia
 
 conda create -n omnisci-dev python=${PYTHON} \
-thrift=0.11.0 \
+'thrift=0.13.0' \
+'cudf=0.13' \
+'cudatoolkit=10.1' \
+'arrow-cpp=0.15.0' \
+'pyarrow==0.15.0' \
+'pandas>=0.25,<0.26' \
+sqlalchemy \
+numpy \
 numpydoc \
-"pyarrow>=0.12.0,<0.14" \
-"sqlalchemy>=1.3" \
-"numpy>=1.16" \
-"pandas>=1.0,<2.0" \
 coverage \
 flake8 \
-pytest \
 pytest-cov \
 pytest-mock \
 shapely \
-numba \
-cudf \
-cudatoolkit \
-"rbc=0.2.0dev0"
+sphinx  \
+requests \
+sphinx_rtd_theme \
+'rbc==0.2.2'
 
 conda activate omnisci-dev
 

ci/requirements.txt

@@ -1,15 +1,14 @@
-thrift==0.11.0
+thrift==0.13.0
+pyarrow==0.15.0
+pandas>=0.25,<0.26
+sqlalchemy
+numpy
 numpydoc
-pyarrow>=0.12.0,<0.14
-sqlalchemy>=1.3
-numpy>=1.16
-pandas>=1.0,<2.0
 coverage
 flake8
 pytest
 pytest-cov
 pytest-mock
 shapely
 numba
-cudf
-rbc-project==0.2.0dev0
+rbc-project==0.2.2

docs/source/contributing.rst

@@ -16,26 +16,39 @@ Development Environment Setup
 -----------------------------
 
 pymapd is written in plain Python 3 (i.e. no Cython), and as such, doesn't require any specialized development
-environment outside of installing the dependencies. However, we do suggest using a separate conda environment or
-virtualenv to ensure that your changes work without relying on unspecified system-level Python packages.
+environment outside of installing the dependencies. However, we do suggest creating a new conda development enviornment
+with the provided conda `environment.yml` file to ensure that your changes work without relying on unspecified system-level
+Python packages.
 
-To set up a conda environment with python 3.7:
+Two development environment files are provided: one to provide the packages needed to develop on CPU only,
+and the other to provide `GPU` development packages. Only one is required, but you may decide to use both in
+order to run `pytest` against a CPU or GPU environment. 
+
+A `pymapd` development environment can be setup with the following:
+
+*********************
+CPU Environment
+*********************
 
 .. code-block:: shell
+ # clone pymapd repo
+ git clone https://github.com/omnisci/pymapd.git && cd pymapd
+
+ conda env create -f ./environment.yml
 
- # create a conda environment
- conda create -n pymapd_dev
- conda activate pymapd_dev
+ # ensure you have activated the environment
+ conda activate omnisci-dev
 
- # ipython is optional, other packages required to run test suite
- conda install python=3.7 ipython pytest pytest-mock shapely
+*********************
+GPU Environment
+*********************
 
- # get pymapd repo
- git clone https://github.com/omnisci/pymapd.git
+.. code-block:: shell
+ # from the pymapd project root
+ conda env create -f environment_gpu.yml
 
- # install pymapd using pip...make sure you are in the pymapd folder cloned above
- # pip will install dependencies needed to run pymapd
- pip install -e .
+ # ensure you have activated the environment
+ conda activate omnisci-gpu-dev
 
 At this point, you have everything you need to develop pymapd. However, to run the test suite, you need to be running
 an instance of OmniSci on the same machine you are devloping on. OmniSci provides `Docker`_ images that work great for this purpose.
@@ -118,19 +131,21 @@ Updating Apache Thrift Bindings
 -------------------------------
 
 When the upstream `mapd-core`_ project updates its Apache Thrift definition file, the bindings shipped with
-``pymapd`` need to be regenerated.
+``pymapd`` need to be regenerated. Note that the `omniscidb` repository must be cloned locally.
 
 .. code-block:: shell
+ # Clone the omnisci repository
+ git clone https://github.com/omnisci/omniscidb
+
+ # Ensure you are at the root of the omnisci directory.
+ cd ./omniscidb
+
+ # Use Thrift to generate the Python bindings
+ thrift -gen py -r omnisci.thrift
 
- # generate python bindings with Thrift
- thrift -gen py mapd.thrift
- thrift -gen py completion_hints.thrift
- thrift -gen py common.thrift
- thrift -gen py serialized_result_set.thrift
- thrift -gen py extension_functions.thrift
+ # Copy the generated bindings to the pymapd root
+ cp -r ./gen-py/omnisci/* ../pymapd/omnisci/
 
-After the bindings are generated, copy them to their respective folders in the pymapd repo. Each set of Thrift files
-are their own package within the overall pymapd package.
 
 --------------------------
 Updating the Documentation

environment.yml

@@ -3,16 +3,20 @@ channels:
 - conda-forge
 - defaults
 dependencies:
-- thrift=0.11.0
-- numpydoc
-- "pyarrow>=0.12.0,<0.14"
+- thrift=0.13.0
+- arrow-cpp=0.15.0
+- pyarrow==0.15.0
+- pandas>=0.25,<0.26
+- python>=3.6,<3.8
 - sqlalchemy
-- numpy>=1.14
-- "pandas>=1.0,<2.0"
+- numpy
+- numpydoc
 - coverage
 - flake8
-- "pytest>=3.6,<4.0"
 - pytest-cov
 - pytest-mock
-- rbc
 - requests
+- shapely
+- sphinx 
+- sphinx_rtd_theme
+- 'rbc==0.2.2'

environment_gpu.yml

@@ -0,0 +1,26 @@
+name: omnisci-gpu-dev
+channels:
+- conda-forge
+- defaults
+- nvidia
+- rapidsai
+dependencies:
+- thrift=0.13.0
+- cudf=0.13
+- cudatoolkit=10.1
+- arrow-cpp=0.15.0
+- pyarrow==0.15.0
+- pandas>=0.25,<0.26
+- python>=3.6,<3.8
+- sqlalchemy
+- numpy
+- numpydoc
+- coverage
+- flake8
+- pytest-cov
+- pytest-mock
+- shapely
+- sphinx 
+- requests
+- sphinx_rtd_theme
+- 'rbc==0.2.2'