Skip to content

Commit

Permalink
Merge PR #2415: update installation instructions and enable CI on MacOS
Browse files Browse the repository at this point in the history
This PR:
- enables continuous integration testing on MacOS
- updates the developer installation instructions (MacOS users no longer advised to use a system-wide Julia install)
- updates the conda environment (now using `conda-forge::julia>=1.8.5`)
  • Loading branch information
rwest authored May 18, 2023
2 parents 2978f1a + 25cbbc7 commit 99201fe
Show file tree
Hide file tree
Showing 3 changed files with 154 additions and 72 deletions.
35 changes: 27 additions & 8 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,11 @@ concurrency:
cancel-in-progress: true

jobs:
build-and-test-linux:
runs-on: ubuntu-latest
build-and-test-unix:
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
# skip scheduled runs on main from forks
if: ${{ !( github.repository != 'ReactionMechanismGenerator/RMG-Py' && github.event_name == 'schedule' ) }}
env: # update this if needed to match a pull request on the RMG-database
Expand Down Expand Up @@ -73,7 +76,7 @@ jobs:
git clone -b $RMG_DATABASE_BRANCH https://github.com/ReactionMechanismGenerator/RMG-database.git
# modify env variables as directed in the RMG installation instructions
- name: Set PYTHONPATH and PATH
- name: Set Environment Variables
run: |
RUNNER_CWD=$(pwd)
echo "PYTHONPATH=$RUNNER_CWD/RMG-Py:$PYTHONPATH" >> $GITHUB_ENV
Expand Down Expand Up @@ -103,17 +106,18 @@ jobs:

# Regression Testing - Test Execution
- name: Regression Tests - Execution
timeout-minutes: 60
run: |
for regr_test in aromatics liquid_oxidation nitrogen oxidation sulfur superminimal;
do
if timeout 600 python-jl rmg.py test/regression/"$regr_test"/input.py; then
if python-jl rmg.py test/regression/"$regr_test"/input.py; then
echo "$regr_test" "Executed Successfully"
else
echo "$regr_test" "Failed to Execute"
export FAILED=Yes
fi
done
if [[ -v FAILED ]]; then
if [[ ${FAILED} ]]; then
echo "One or more regression tests could not be executed."
echo "Please download the failed results or check the above log to see why."
exit 1
Expand All @@ -131,7 +135,7 @@ jobs:
# Upload Regression Results as Stable if Scheduled or Push to Main
- name: Upload Results as Reference
# upload the results for scheduled CI (on main) and pushes to main
if: github.ref == 'refs/heads/main'
if: github.ref == 'refs/heads/main' && runner.os =='ubuntu'
uses: actions/upload-artifact@v3
with:
name: stable_regression_results
Expand Down Expand Up @@ -215,7 +219,7 @@ jobs:
fi
fi
done
if [[ -v FAILED ]]; then
if [[ ${FAILED} ]]; then
echo "One or more regression tests failed."
echo "Please download the failed results and run the tests locally or check the above log to see why."
exit 1
Expand All @@ -227,14 +231,29 @@ jobs:
mamba install -y -c conda-forge codecov
codecov
# This allows us to have a branch protection rule for tests and deploys with matrix
#
# taken from https://github.com/orgs/community/discussions/4324#discussioncomment-3477871
ci-report-status:
runs-on: ubuntu-latest
needs: build-and-test-unix
if: always()
steps:
- name: Successful CI
if: ${{ !(contains(needs.build-and-test-unix.result, 'failure')) }}
run: exit 0
- name: Failing CI
if: ${{ contains(needs.build-and-test-unix.result, 'failure') }}
run: exit 1

build-and-push-docker:
# after testing and on pushes to main, build and push docker image
# technically we could live without the 'needs' since _in theory_
# nothing will ever be merged into main that fails the tests, but
# who knows ¯\_(ツ)_/¯
#
# taken from https://github.com/docker/build-push-action
needs: build-and-test-linux
needs: build-and-test-unix
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main' && github.repository == 'ReactionMechanismGenerator/RMG-Py'
steps:
Expand Down
69 changes: 34 additions & 35 deletions documentation/source/users/rmg/installation/anacondaDeveloper.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ Installation by Source Using Anaconda Environment for Unix-based Systems: Linux
Note that you should restart your terminal in order for the changes to take effect, as the installer will tell you.

#. There are a few system-level dependencies which are required and should not be installed via Anaconda. These include
`Git <https://git-scm.com/>`_ for version control, `GNU Make <https://www.gnu.org/software/make/>`_, and the C and C++ compilers from the `GNU Compiler Collection (GCC) <https://gcc.gnu.org/>`_ for compiling RMG.
`Git <https://git-scm.com/>`_ for version control, `GNU Make <https://www.gnu.org/software/make/>`_,
and the C and C++ compilers from the `GNU Compiler Collection (GCC) <https://gcc.gnu.org/>`_ for compiling RMG.

For Linux users, you can check whether these are already installed by simply calling them via the command line, which
will let you know if they are missing. To install any missing packages, you should use the appropriate package manager
Expand Down Expand Up @@ -47,17 +48,6 @@ Installation by Source Using Anaconda Environment for Unix-based Systems: Linux
to install this is to simply run one of the commands in the terminal, e.g. ``git``. The terminal will then prompt you on
whether or not you would like to install the Command Line Tools.

For MacOS users only, download and install MacOS Julia 1.8 from here: <https://julialang.org/downloads/>. Then add Julia to PATH by running the following command: ::

echo 'export PATH="/Applications/Julia-1.8.app/Contents/Resources/julia/bin:$PATH"' >> ~/.bash_profile

export PATH="/Applications/Julia-1.8.app/Contents/Resources/julia/bin:$PATH"

If using MacOS Catalina or newer your terminal may be using ``zsh`` by default, in which case you should replace ``.bash_profile`` with ``.zshrc``.

Note that this Julia install will not respect conda environmental boundaries meaning only one conda environment can be linked to it at a time.


#. Install the latest versions of RMG and RMG-database through cloning the source code via Git. Make sure to start in an
appropriate local directory where you want both RMG-Py and RMG-database folders to exist.
Github has deprecated password authentication from the command line, so it
Expand All @@ -79,41 +69,56 @@ Installation by Source Using Anaconda Environment for Unix-based Systems: Linux
conda install -n base conda-libmamba-solver
conda config --set solver libmamba

Now create the conda environment for RMG-Py ::
#. Navigate to the RMG-Py directory ::

cd RMG-Py

#. Apple silicon (M1+) users only: execute the following commands
**instead of** the following `conda env create -f environment.yml` step.
(This will tell conda that we want to the environment to use x86
architecture rather than the native ARM64 architecture) ::

conda create -n rmg_env
conda activate rmg_env
conda config --env --set subdir osx-64
conda env update -f environment.yml

#. Create the conda environment for RMG-Py ::

conda env create -f environment.yml

If either of these commands return an error due to being unable to find the ``conda`` command, try to either close and reopen your terminal to refresh your environment variables or type the following command.
If either of these commands return an error due to being unable to find the ``conda`` command,
try to either close and reopen your terminal to refresh your environment variables
or type the following command.

If on Linux or pre-Catalina MacOS ::
If on Linux or pre-Catalina MacOS (or if you have a bash shell)::

source ~/.bashrc

If on MacOS Catalina or later ::
If on MacOS Catalina or later (or if you have a Z shell)::

source ~/.zshrc

#. Activate conda environment ::

conda activate rmg_env
Note regarding differences between conda versions: Prior to Anaconda 4.4, the command to activate an environment was
``source activate rmg_env``. It has since been changed to ``conda activate rmg_env`` due to underlying changes to
standardize operation across different operating systems. However, a prerequisite to using the new syntax is having
run the ``conda init`` setup routine, which can be done at the end of the install procedure if the user requests.

#. Switch the conda solver to libmamba again, to accelerate any changes you might make to this conda environment in the future::

conda config --set solver libmamba

#. Compile RMG-Py after activating the conda environment ::

make

#. Modify environment variables. Add RMG-Py to the PYTHONPATH to ensure that you can access RMG modules from any folder.
*This is important before the next step in which julia dependencies are installed.*
Also, add your RMG-Py folder to PATH to launch ``rmg.py`` from any folder.

In general, these commands should be placed in the appropriate shell initialization file. For Linux users using
bash (the default on distributions mentioned here), these should be placed in ``~/.bashrc``. For MacOS users using bash (default before MacOS Catalina),
these should be placed in ``~/.bash_profile``, which you should create if it doesn't exist. For MacOS users using zsh
(default beginning in MacOS Catalina), these should be placed in ``~/.zshrc``. ::
In general, these commands should be placed in the appropriate shell initialization file.
For Linux users using bash (the default on distributions mentioned here), these should be placed in ``~/.bashrc``.
For MacOS users using bash (default before MacOS Catalina), these should be placed in ``~/.bash_profile``, which you should create if it doesn't exist.
For MacOS users using zsh (default beginning in MacOS Catalina), these should be placed in ``~/.zshrc``. ::

export PYTHONPATH=YourFolder/RMG-Py/:$PYTHONPATH
export PATH=YourFolder/RMG-Py/:$PATH
Expand All @@ -122,16 +127,12 @@ Installation by Source Using Anaconda Environment for Unix-based Systems: Linux

Be sure to either close and reopen your terminal to refresh your environment variables (``source ~/.bashrc`` or ``source ~/.zshrc``).

#. Install and Link Julia dependencies ::
#. Install and Link Julia dependencies: ::

python -c "import julia; julia.install(); import diffeqpy; diffeqpy.install()"
julia -e 'using Pkg; Pkg.add("PyCall");Pkg.build("PyCall");Pkg.add(PackageSpec(name="ReactionMechanismSimulator",rev="main")); using ReactionMechanismSimulator;'

julia -e 'using Pkg; Pkg.add(PackageSpec(name="ReactionMechanismSimulator",rev="main")); using ReactionMechanismSimulator;'
python -c "import julia; julia.install(); import diffeqpy; diffeqpy.install()"

Note that this links your python to python-jl enabling calls to Julia through pyjulia. Occasionally programs will
interact with python-jl differently than the default python. If this occurs for you we recommend doing that operation
in a different conda environment. However, if convenient you can undo this linking by replacing python-jl with
python3 in the second command above. Just make sure to rerun the linking command once you are done.

#. Finally, you can run RMG from any location by typing the following (given that you have prepared the input file as ``input.py`` in the current folder). ::

Expand All @@ -140,8 +141,6 @@ Installation by Source Using Anaconda Environment for Unix-based Systems: Linux
You may now use RMG-Py, Arkane, as well as any of the :ref:`Standalone Modules <modules>` included in the RMG-Py package.




Test Suite
==========

Expand Down
122 changes: 93 additions & 29 deletions environment.yml
Original file line number Diff line number Diff line change
@@ -1,54 +1,118 @@
# environment.yml - conda environment specification file for RMG-Py
#
# Usage: conda env create --file environment.yml
#
# This file contains all of the software packages needed to run RMG-Py.
# There is a mixture of the following
# - packlages would could be installed at OS level, but we install here
# for better version control
# - python tools
# - external software tools specific to chemistry
# - other software we maintain which RMG depends on
# + some other categories (see below)
#
# Changelog:
# - May 15, 2023 Added this changelog, added inline documentation,
# made depdency list more explicit (@JacksonBurns).
#
name: rmg_env
channels:
- defaults
- rmg
- conda-forge
- cantera
dependencies:
# System-level dependencies - we could install these at the OS level
# but by installing them in the conda environment we get better control
- cairo
- cairocffi
- ffmpeg
- xlrd
- xlwt
- h5py
- graphviz
- markupsafe
- psutil
# conda-forge not default, since default has a version information bug
# (see https://github.com/ReactionMechanismGenerator/RMG-Py/pull/2421)
- conda-forge::ncurses
- conda-forge::suitesparse

# external software tools for chemistry
- coolprop
- cantera::cantera=2.6
- conda-forge::mopac
- conda-forge::cclib >=1.6.3
- rmg::chemprop
- coolprop
- conda-forge::openbabel >= 3

# general-purpose external software tools
- conda-forge::julia>=1.8.5

# Python tools
- python >=3.7
- coverage
- cython >=0.25.2
- rmg::diffeqpy
- ffmpeg
- rmg::gprof2dot
- graphviz
- h5py
- scikit-learn
- scipy
- numpy >=1.10.0
- pydot
- jinja2
- jupyter
- rmg::lpsolve55
- markupsafe
- pymongo
- pyparsing
- pyyaml
- networkx
- nose
- matplotlib >=1.5
- conda-forge::mopac
- mpmath
- pandas

# packages we maintain
- rmg::gprof2dot
- rmg::lpsolve55
- rmg::muq2
- networkx
- nomkl
- nose
- rmg::numdifftools
- numpy >=1.10.0
- conda-forge::openbabel >= 3
- pandas
- psutil
- rmg::pydas >=1.0.3
- pydot
- rmg::pydqed >=1.0.3
- rmg::pyjulia
- pymongo
- pyparsing
- rmg::pyrdl
- rmg::pyrms
- python >=3.7
- pyyaml
- rmg::quantities
- rmg::rdkit >=2020.03.3.0
- scikit-learn
- scipy
- rmg::symmetry
- xlrd
- xlwt
- conda-forge::ncurses

# packages we would like to stop maintaining (and why)
- rmg::diffeqpy
# we should use the official verison https://github.com/SciML/diffeqpy),
# rather than ours (which is only made so that we can get it from conda)
# It is only on pip, so we will need to do something like:
# https://stackoverflow.com/a/35245610

- rmg::chemprop
# Our build of this is version 0.0.1 (!!) and we are using parts
# of the API that are now gone. Need a serious PR to fix this.

- rmg::rdkit >=2020.03.3.0
# We should use the official channel, not sure how difficult this
# change will be.

- rmg::pyjulia
# This is identical to the conda-forge package, except that we run
# a hard to decipher sed command during the build process:
# https://github.com/ReactionMechanismGenerator/conda-recipes/blob/rmg-deps/pyjulia/build.sh#LL15C69-L15C69
#
# We should either remove the need to use this command or add this
# to the installation steps
#
# Both pyrms and diffeqpy depend on this package.

# conda mutex metapackage
- nomkl

# additional packages that are required, but not specified here (and why)
# pydqed, pydas, mopac, and likely others require a fortran compiler (specifically gfortran)
# in the environment. Normally we would add this to the environment file with
# - libgfortran-ng >= 10
# but this exact package is only maintained for Linux, meaning that if we were to add
# it here the environment creation would fail on Mac. The way it ends up working behind
# the scenes is that conda will find a different package for Mac that provides gfortran,
# but because we cannot specify per-platform requirements in this file we simply leave
# it out.

0 comments on commit 99201fe

Please sign in to comment.