diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dc84b1fff..02f29ea5b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,33 +1,37 @@ -If you would like to contribute to the Code for IATI Analytics project, you can.... +# Contributing -* Send us feedback about your user experience. Contact details at: https://github.com/codeforIATI +If you would like to contribute to the IATI Dashboard you can.... + +* Send us feedback about your user experience. You can find our [contact details on the IATI Organisation page on GitHub](https://github.com/IATI) * Report bugs * Request new features * Contribute code or documents to improve the application. See the list of specific tasks below. + ## How to report a bug or request a feature -If you are able to work with GitHub then please [create an issue](https://github.com/codeforIATI/analytics/issues/new/choose). +If you are able to work with GitHub then please [create an issue](https://github.com/IATI/IATI-Dashboard/issues/new/choose). + +Before creating a new issue [check to see if the issue already exists](https://github.com/IATI/IATI-Dashboard/issues/). If not then please do create it. -Before creating a new issue check to see if the issue already exists. If not then please do create it. +If you are not comfortable working with GitHub, but would still like to contribute, then talk to us. You can reach us via the central [IATI Developer Documentation pages](https://iatistandard.org/en/guidance/developer/). -If you are not comfortable working with GitHub, but would still like to contribute, then talk to us. Details at: https://codeforiati.org/get-involved/ ## How to contribute code and documents ### How we use branches in this repository -* `main` represents our main development branch, and is the branch we are currently using for our deployed instance of the code +* `live` represents the branch we are currently using for our deployed instance of the code. +* Eventually a `develop` branch will be for development work that is not yet live. * Other branches represent development work or bug fixes. ### Submitting changes * Fork this repository (if you haven't previously) -* Make sure you're working on top of an up to date copy of the `main` branch - - Create a branch named after the work you're doing (if you're targeting a specific issue, start the branch name with the issue number e.g. `42-feature-name`) -* Do your work - - If your work addresses a specific issue, reference that issue in your commit message by starting the commit message with `[#issue number]` e.g. `[#64]` -* Create a pull request against `main` +* Make sure you're working on top of an up to date copy of the `live` branch +* Create a branch named after the work you're doing (if you're targeting a specific issue, start the branch name with the issue number e.g. `42-feature-name`). +* Do your work, creating atomic commits as you go. If your work addresses a specific issue, reference that issue in your commit message using the full URL to the issue. Please name your commits starting with a one-word description of the commit, e.g., *fix*, *update*, *refactor*. +* Create a pull request against `develop`. ## Specific Tasks: @@ -52,10 +56,6 @@ Do you have other tests/statistics that we could be generating? Can you improve the unit testing to make deployment more robust? -### Fix a Bitesize issue - -We mark some of issues as 'Bitesize'. Generally these will help ease you into the code and help you find your way around. - ## Talk to us -We'd love to hear from you. Details at: https://codeforiati.org/get-involved/ +We'd love to hear from you. You can find our [contact details at the main IATI GitHub page](https://github.com/IATI). diff --git a/README.md b/README.md new file mode 100644 index 000000000..4592fe562 --- /dev/null +++ b/README.md @@ -0,0 +1,124 @@ +IATI Dashboard +============== + +[![Coverage Status](https://coveralls.io/repos/github/IATI/IATI-Dashboard/badge.svg?branch=merge-codeforiati-and-publishingstats)](https://coveralls.io/github/IATI/IATI-Dashboard?branch=merge-codeforiati-and-publishingstats) +[![GPLv3 License](https://img.shields.io/badge/license-GPLv3-blue.svg)](https://github.com/IATI/IATI-Dashboard/blob/merge-codeforiati-and-publishingstats/LICENSE.md) + +## Summary + + Product | IATI Dashboard +--- | --- +Description | A Django web application that provides key numbers, statistics and graphs about the data on the [IATI registry](http://iatiregistry.org/). This repository is currently a development version where the IATI Dashboard/Publishing Statistics and Code for IATI Analytics are being merged. +Website | Development only; see [IATI Dashboard](https://dashboard.iatistandard.org), and [Code for IATI Analytics](https://analytics.codeforiati.org) for live versions. +Related | Repositories for the [live version of the IATI Dashboard](https://github.com/IATI/IATI-Dashboard), [live version of the IATI Publishing Stats](https://github.com/IATI/IATI-Publishing-Statistics), and [Code for IATI Analytics](https://github.com/codeforIATI/analytics). Data is generated from [Code for IATI Stats](https://github.com/codeforIATI/IATI-Stats). +Documentation | Rest of README.md +Technical Issues | See https://github.com/IATI/IATI-Dashboard/issues +Support | https://iatistandard.org/en/guidance/get-support/ + +## High-level requirements + +* Python 3.12 +* Unix-based setup (e.g., Linux, MacOS X) with `bash`, `wget` and `curl` installed. +* Development files for libfreetype, libpng, libxml and libxslt e.g. ``libfreetype6-dev libpng-dev libxml2-dev libxslt-dev``. + +## Running the app locally +### Overview +The IATI Dashboard is mostly written in Python but also has some helper Bash scripts to collect the data that the dashboard uses. Top-level steps required to run the Dashboard are: + +1. Setup Python environment and install dependencies. +2. Fetch the data. +3. Build the static graphs and other data that will be served via the Dashboard. +4. Run the web server. + +Paths to different directories are set in `./src/config.py`. + +### 1. Setup environment + +Assuming that this repository has been cloned and you are in the root directory of the repository. + +``` +# Setup and activate a virtual environment (recommended) - here we use virtualenv +virtualenv ve +source ve/bin/activate +``` + +Now install the dependencies. + +``` +pip install -r requirements.txt +``` + +### 2. Fetching the data + +Bash scripts are used to fetch the data that the Dashboard will present. They will store data in `./data` and `./stats-calculated`. + +``` +# Fetch the necessary calculated stats +./get_stats.sh + +# Fetch some extra data from github and github gists and other sources on the internet +./fetch_data.sh +``` + +### 3. Build static data and graphs and copy to static + +``` +mkdir out +cd src +python make_plots.py +python make_csv.py +python speakers_kit.py +cp ../out/data static/ +cp ../img/aggregate static/ +cp ../img/publishers static/ +``` + +### 4. Run the webserver. + +From `./src/`: + +``` +python manage.py runserver +``` + +The Dashboard will now be accessible from `localhost:8000/`. + + +## Development + +### Calculating your own statistics + +The IATI Dashboard requires a `stats-calculated` directory, which can be downloaded using the `get_stats.sh` shell script as described above. This can also be calculated using [Code for IATI Stats](http://github.com/codeforIATI/IATI-Stats) where `stats-calculated` corresponds to the `gitout` directory generated by [`git.sh` in IATI-Stats](https://github.com/codeforIATI/IATI-Stats#running-for-every-commit-in-the-data-directory). + +Often you only want to regenerate the current stats, use `get_stats.sh` to download the pre-calculated historical stats and just replace the `stats-calculated/current directory` with the `out` directory produced by running the [loop, aggregate and invert commands individually](https://github.com/codeforIATI/IATI-Stats#getting-started), then regenerate graphs and CSV files as per the above. + +### Adding new dependencies + +If a change requires new dependencies then please add to `requirements.in` or `requirements_dev.in` as appropriate and recompile: + +``` +pip-compile requirements_dev.in +pip-compile requirements.in +``` + +### Linting + +Code linting is carried out using [Flake8](https://flake8.pycqa.org/en/latest/) and `setup.cfg` has the configuration. + +## License + Copyright (C) 2013-2015 Ben Webb + Copyright (C) 2013-2014 David Carpenter + Copyright (C) 2021 Andy Lulham + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . diff --git a/README.rst b/README.rst deleted file mode 100644 index 793c29feb..000000000 --- a/README.rst +++ /dev/null @@ -1,136 +0,0 @@ -Code for IATI Analytics -======================= - -.. image:: https://github.com/codeforIATI/analytics/actions/workflows/ci.yml/badge.svg?branch=main - :target: https://github.com/codeforIATI/analytics/actions/workflows/ci.yml -.. image:: https://coveralls.io/repos/github/codeforIATI/analytics/badge.svg?branch=main - :target: https://coveralls.io/github/codeforIATI/analytics?branch=main -.. image:: https://img.shields.io/badge/license-GPLv3-blue.svg - :target: https://github.com/codeforIATI/analytics/blob/main/LICENSE.md - -Introduction ------------- - -Code for IATI Analytics displays key numbers and graphs about the data on the `IATI registry `__. - -See Analytics in action at https://analytics.codeforiati.org - -Analytics is in beta. All contents / URLs / machine-readable downloads are subject to change. - -This repository is the code for Analytics frontend. Statistics are generated from the Registry by code in a separate repository - https://github.com/codeforIATI/IATI-Stats - -Technology Overview -^^^^^^^^^^^^^^^^^^^ - -Analytics is mostly written in Python, with some helper Bash scripts. - -Python scripts: - -* ``make_html.py`` contains is a Flask application that makes use of Frozen Flask to generate some static HTML. -* ``make_csv.py`` generates CSV files. -* ``plots.py`` generates static images of graphs using matplotlib. - -Bash helper scripts: - -* The main source of data are the statistics generated by `IATI-Stats `_ (about the data on the IATI Registry). ``get_stats.sh`` can be used to fetch a recently calculated copy of these stats. (or see calculating your own stats section below) -* Analytics also uses various other data from online sources (including GitHub). These can be fetched using ``fetch_data.sh``. -* ``git.sh`` runs all the above commands, see Usage below. - -Installation -^^^^^^^^^^^^ - -Requirements: - -* Unix based setup (e.g. Linux, Mac OS X) with bash etc. -* wget and curl installed -* Python 3 -* Python dev library ``python-dev`` -* python-virtualenv (optional) -* Development files for libfreetype, libpng, libxml and libxslt e.g. ``libfreetype6-dev libpng-dev libxml2-dev libxslt-dev``. - - (alternatively, you may be able to install some of the python dependencies in - requirements.txt using your package manager) - - -To install: - -.. code-block:: bash - - ## Get the code - git clone https://github.com/codeforIATI/analytics.git - cd analytics - - ## Set up a virtual environment (recommended) - # Create a virtual environment - virtualenv pyenv - # Activate the virtual environment - # (you need to this every time you open a new terminal session) - source pyenv/bin/activate - - ## Install python dependencies - ## Use pip as described below, or your distro's package manager to install - ## the dependcies in requirements.txt - # If you are running a less recent linux distro, you will need to install distribute - easy_install -U distribute - pip install -r requirements.txt - -Usage -^^^^^ - -The following steps are performed routinely: - -.. code-block:: bash - - # Fetch the necessary calculated stats - ./get_stats.sh - # Fetch some extra data from github and github gists - ./fetch_data.sh - - mkdir out - python plots.py - python make_csv.py - python make_html.py - -make_html.py will output a MissingURLGeneratorWarning. This is expected, as some of the URLs defined are for the live development server only (see below). - -The full list of steps for our deployment can be found in ``git.sh``. (The name of this is now a misnomer as the output is no longer a git repository - previously a commit was pushed to GitHub pages.) - -Development -^^^^^^^^^^^ - -For development, you can use the live Flask development server, instead of Frozen Flask. - -.. code-block:: bash - - python make_html.py --live - -Using the live development server is highly recommended, because it displays full bracktraces for 500 errors, whereas frozen flask does not. - -Calculating your own statistics -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Analytics requires a `stats-calculated` directory, which can be downloaded using the `get_stats.sh` shell script as described above, or calculated yourself using http://github.com/codeforIATI/IATI-Stats . `stats-calculated` corresponds to the `gitout` directory generated by `IATI-Stat's git.sh `__. - -Often you only want to regenerate the current stats, use `get_stats.sh` to download the pre-calculated historical stats and just replace the stats-calculated/current directory with the out directory produced by running the `loop, aggregate and invert commands individually `__. - -License -^^^^^^^ - -:: - - Copyright (C) 2013-2015 Ben Webb - Copyright (C) 2013-2014 David Carpenter - Copyright (C) 2021 Andy Lulham - - This program is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program. If not, see .