es-flow

Atmospheric and marine flow characterisation tools, implementing the 'Attached-Detached Eddy Method' for boundary layer analysis.

View the manual and full documentation at https://es-flow.readthedocs.io/

Developer notes

Commit codes

Each commit has a code to help improve understanding and relevance:

Code	Description
OPS	Related to Devops - release workflow and CI/CD
DOC	Improvements in documentation or descriptive comments in the code
FIX	Fixes a particular bug
IMP	Implements all or part of a particular feature
TST	Implements or improves tests
GIT	Related to Git or GitHub (e.g. submodules, ignores, etc)
REF	Refactoring
STY	Changes to comply with code style guide

Project Structure and Code Style

The folder structure is arranged as:

• ./ CMake file, readme, licensing, requirements (for python) and git configuration files
• ./docs Documentation in .rst form (including autogenerated library docs)
• ./scripts Supporting scripts (eg make_docs.py to autogenerate the library API docs using sphinx and breathe)
• ./source Source code
• ./test Unit tests

CMake builds in its own directories. However, once built, files may be moved / copied to the following locations for inclusion in other projects:

• ./include Header files that expose the public interface to the library
• ./lib Library files (.a, .so, .dylib, .dll etc)
• ./bin Main executable, plus tools and examples

Code style, includes and project structure should follow the Google C++ style guide.

Release Flow

Steps to create a new release:

• Create a new branch, from master, entitled release<version-tag> and push to remote. Version tags are semantic:

git checkout master
git checkout -b release-v0.1.1-alpha.1
git push --set-upstream origin release-v0.1.1-alpha.1

• Once a release is prepared, create a release on GitHub (which tags the repo) then pull that branch into master.
• Master therefore always contains the latest release.
• Log in to ReadTheDocs and add the new version, so it shows up in the version list for the documentation.

Third party dependencies

Currently in use

See documentation for additional info on build and use of these libraries.

Intel MKL to provide FFT and other performance primitives, enhancing Eigen.

ceres-solver, a well supported project by Google, is used for nonlinear least squares optimisation.

Eigen provides a linear algebra library. It isn't as consistent with MATLAB's API as armadillo, but is used extensively in ceres-solver and elsewhere so is our selected library of choice.

NumericalIntegration NB This fork avoids compiling against the GPL licensed MPFRC++ library. Provides a numerical integration module for eigen. It's weird that it's not an unsupported module in eigen (see this thread) given that NumericalDiff is part of eigen.

matio read and write tools for MATLAB .mat format files, including recent v7.3 (HDFS) file formats. Much higher level than writing the HDF5 files ourselves.

cxxopts argument parser for C++11 under the MIT license (NB most "standard" parsers are under GNU!!!).

glog google's asynchronous logging library, used for logging to file.

cpplot for plotting figures

For building documentation

sphinx is the document generator; it'll take the .rst documentation files and turn them into formatted documentation, in either latex or HTML form.

sphinx_rtd_theme gives us the excellent looking ReadTheDocs theme for our HTML documentation.

For consideration and possible future use

We're not yet committed to any of the following, but a range of possibly useful libraries is::

gflags is an optional alternative to cxxopts, which is more widely supported and already used by ceres (i.e. installed already as part of the build chain). We have an issue open to change over (see #18) to gflags.

CppNumericalSolvers provides a directly analagous alternative to MATLAB's fminsearch().

Linterp provides a interpolation of gridded and unstructured data in N dimensions.

Tino Kluge maintains a spline interpolant library with linear extrapolation.

eigen-matio could be useful for reading and writing eigen matrix types to mat files.

Compilation

A cross-platform compilation file is provided using cmake.

MATLAB MEX

DEPRECATION WARNING: The MathWorks really make it extremely difficult to support integrations. MATLAB mex files are almost completely platform- and matlab-version- dependent. So you basically have to recompile for every MATLAB release, on every platform, clearly a nightmare. MATLAB does however allow you to invoke C library functions directly (see loadlibrary in the MATLAB help) so we may consider writing a header exposing C-style library API for use with MATLAB, to allow us to continue supporting MATLAB in a practical way.

The CMake build process used to MATLAB based mex files for library functionality - requiring MATLAB to be installed on the build machine for linking purposes. This is unsupported as of January 2019 and removed in the 0.1.0 release.

Documentation

Documentation resides in the ./docs directory, as *.rst files. The instruction manual includes an automatically generated subfolder of .rst files detailing the API of the library.

Creating the bibliography

For sphinx to create a bibliography, the bibliography.rst file needs to contain, in RestructuredText format, the references used.

However, you may have BibTeX, rather than .rst, references. To convert, you can use bib2reSTcitation, a handy tool for converting .bib files to .rst files.

Building documentation

The build steps are as follows:

• Do any conversions of reference formats that you need to, ensuring bibliography.rst is up to date with all references.
• Use breathe to parse the library and generate doxygen files (in xml form)
• Use exhale to convert doxygen xml files to rst documentation
• Use sphinx (with mathjax to convert AMS LaTeX to rendered equations and sphinx-rtd-theme for prettiness) to convert all the rst files to html.

Build Environment

Most IDEs will give you an approximate preview of RST documentation, so you shouldn't need to build docs on your own machine. On release-a.b.c-alpha.x branches, you can create a pre-release on github to see how they look build on ReadTheDocs.

If you absolutely must build docs on your local machine, see issue 68 for a script which builds docs on OSX using python 2.7. A bit outdated, but you should be able to repeat these or similar on your machine. If you're using pyenv locally and have other versions of sphinx installed, you also need to add a shim to pyenv so that sphinx calls the right python installation. Copy scripts/sphinx-build to the pyenv shim directory, in my case its /Users/thc29/.pyenv/shims/, and edit the last two lines of it to your settings.

Unit Testing

Unit testing is done with the google test framework; test executables should be built with cmake.

Test Environment

The following environment variables are required for the tests (which build signature files) to run:

Name	Example	Description
TEST_DATA_DIR	/Users/me/Source/octue/es-flow/test/test_data	The file path (absolute or relative to the application working directory) of the test data directory, WITHOUT a trailing slash. You must have write access to this directory.