Code for IATI Analytics

Introduction

Code for IATI Analytics displays key numbers and graphs about the data on the IATI registry.

See Analytics in action at https://analytics.codeforiati.org

Analytics is in beta. All contents / URLs / machine-readable downloads are subject to change.

This repository is the code for Analytics frontend. Statistics are generated from the Registry by code in a separate repository - https://github.com/codeforIATI/IATI-Stats

Technology Overview

Analytics is mostly written in Python, with some helper Bash scripts.

Python scripts:

• make_html.py contains is a Flask application that makes use of Frozen Flask to generate some static HTML.
• make_csv.py generates CSV files.
• plots.py generates static images of graphs using matplotlib.

Bash helper scripts:

• The main source of data are the statistics generated by IATI-Stats (about the data on the IATI Registry). get_stats.sh can be used to fetch a recently calculated copy of these stats. (or see calculating your own stats section below)
• Analytics also uses various other data from online sources (including GitHub). These can be fetched using fetch_data.sh.
• git.sh runs all the above commands, see Usage below.

Installation

Requirements:

• Unix based setup (e.g. Linux, Mac OS X) with bash etc.

• wget and curl installed

• Python 3

• Python dev library python-dev

• python-virtualenv (optional)

• Development files for libfreetype, libpng, libxml and libxslt e.g. libfreetype6-dev libpng-dev libxml2-dev libxslt-dev.

  (alternatively, you may be able to install some of the python dependencies in requirements.txt using your package manager)

To install:

## Get the code
git clone https://github.com/codeforIATI/analytics.git
cd analytics

## Set up a virtual environment (recommended)
# Create a virtual environment
virtualenv pyenv
# Activate the virtual environment
# (you need to this every time you open a new terminal session)
source pyenv/bin/activate

## Install python dependencies
## Use pip as described below, or your distro's package manager to install
## the dependcies in requirements.txt
# If you are running a less recent linux distro, you will need to install distribute
easy_install -U distribute
pip install -r requirements.txt

Usage

The following steps are performed routinely:

# Fetch the necessary calculated stats
./get_stats.sh
# Fetch some extra data from github and github gists
./fetch_data.sh

mkdir out
python plots.py
python make_csv.py
python make_html.py

make_html.py will output a MissingURLGeneratorWarning. This is expected, as some of the URLs defined are for the live development server only (see below).

The full list of steps for our deployment can be found in git.sh. (The name of this is now a misnomer as the output is no longer a git repository - previously a commit was pushed to GitHub pages.)

Development

For development, you can use the live Flask development server, instead of Frozen Flask.

python make_html.py --live

Using the live development server is highly recommended, because it displays full bracktraces for 500 errors, whereas frozen flask does not.

Calculating your own statistics

Analytics requires a stats-calculated directory, which can be downloaded using the get_stats.sh shell script as described above, or calculated yourself using http://github.com/codeforIATI/IATI-Stats . stats-calculated corresponds to the gitout directory generated by IATI-Stat's git.sh.

Often you only want to regenerate the current stats, use get_stats.sh to download the pre-calculated historical stats and just replace the stats-calculated/current directory with the out directory produced by running the loop, aggregate and invert commands individually.

License

Copyright (C) 2013-2015 Ben Webb <bjwebb67@googlemail.com>
Copyright (C) 2013-2014 David Carpenter <caprenter@gmail.com>
Copyright (C) 2021 Andy Lulham <a.lulham@gmail.com>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.