First part of refactoring the Dashboard to Django

CONTRIBUTING.md

@@ -1,33 +1,37 @@
-If you would like to contribute to the Code for IATI Analytics project, you can....
+# Contributing
 
-* Send us feedback about your user experience. Contact details at: https://github.com/codeforIATI
+If you would like to contribute to the IATI Dashboard you can....
+
+* Send us feedback about your user experience. You can find our [contact details on the IATI Organisation page on GitHub](https://github.com/IATI)
 * Report bugs
 * Request new features
 * Contribute code or documents to improve the application. See the list of specific tasks below.
 
+
 ## How to report a bug or request a feature
 
-If you are able to work with GitHub then please [create an issue](https://github.com/codeforIATI/analytics/issues/new/choose).
+If you are able to work with GitHub then please [create an issue](https://github.com/IATI/IATI-Dashboard/issues/new/choose).
+
+Before creating a new issue [check to see if the issue already exists](https://github.com/IATI/IATI-Dashboard/issues/). If not then please do create it.
 
-Before creating a new issue check to see if the issue already exists. If not then please do create it.
+If you are not comfortable working with GitHub, but would still like to contribute, then talk to us. You can reach us via the central [IATI Developer Documentation pages](https://iatistandard.org/en/guidance/developer/).
 
-If you are not comfortable working with GitHub, but would still like to contribute, then talk to us. Details at: https://codeforiati.org/get-involved/
 
 ## How to contribute code and documents
 
 ### How we use branches in this repository
 
-* `main` represents our main development branch, and is the branch we are currently using for our deployed instance of the code
+* `live` represents the branch we are currently using for our deployed instance of the code.
+* Eventually a `develop` branch will be for development work that is not yet live.
 * Other branches represent development work or bug fixes.
 
 ### Submitting changes
 
 * Fork this repository (if you haven't previously)
-* Make sure you're working on top of an up to date copy of the `main` branch
-    - Create a branch named after the work you're doing (if you're targeting a specific issue, start the branch name with the issue number e.g. `42-feature-name`)
-* Do your work
-    - If your work addresses a specific issue, reference that issue in your commit message by starting the commit message with `[#issue number]` e.g. `[#64]`
-* Create a pull request against `main`
+* Make sure you're working on top of an up to date copy of the `live` branch
+* Create a branch named after the work you're doing (if you're targeting a specific issue, start the branch name with the issue number e.g. `42-feature-name`).
+* Do your work, creating atomic commits as you go.  If your work addresses a specific issue, reference that issue in your commit message using the full URL to the issue.  Please name your commits starting with a one-word description of the commit, e.g., *fix*, *update*, *refactor*.
+* Create a pull request against `develop`.
 
 ## Specific Tasks:
 
@@ -52,10 +56,6 @@ Do you have other tests/statistics that we could be generating?
 
 Can you improve the unit testing to make deployment more robust?
 
-### Fix a Bitesize issue
-
-We mark some of issues as 'Bitesize'. Generally these will help ease you into the code and help you find your way around.
-
 ## Talk to us
 
-We'd love to hear from you. Details at: https://codeforiati.org/get-involved/
+We'd love to hear from you. You can find our [contact details at the main IATI GitHub page](https://github.com/IATI).

README.md

@@ -0,0 +1,124 @@
+IATI Dashboard
+==============
+
+[![Coverage Status](https://coveralls.io/repos/github/IATI/IATI-Dashboard/badge.svg?branch=merge-codeforiati-and-publishingstats)](https://coveralls.io/github/IATI/IATI-Dashboard?branch=merge-codeforiati-and-publishingstats)
+[![GPLv3 License](https://img.shields.io/badge/license-GPLv3-blue.svg)](https://github.com/IATI/IATI-Dashboard/blob/merge-codeforiati-and-publishingstats/LICENSE.md)
+
+## Summary
+
+ Product  |  IATI Dashboard
+--- | ---
+Description | A Django web application that provides key numbers, statistics and graphs about the data on the [IATI registry](http://iatiregistry.org/).  This repository is currently a development version where the IATI Dashboard/Publishing Statistics and Code for IATI Analytics are being merged.
+Website |  Development only; see [IATI Dashboard](https://dashboard.iatistandard.org), and [Code for IATI Analytics](https://analytics.codeforiati.org) for live versions. 
+Related | Repositories for the [live version of the IATI Dashboard](https://github.com/IATI/IATI-Dashboard), [live version of the IATI Publishing Stats](https://github.com/IATI/IATI-Publishing-Statistics), and [Code for IATI Analytics](https://github.com/codeforIATI/analytics).  Data is generated from [Code for IATI Stats](https://github.com/codeforIATI/IATI-Stats).
+Documentation | Rest of README.md
+Technical Issues | See https://github.com/IATI/IATI-Dashboard/issues
+Support | https://iatistandard.org/en/guidance/get-support/
+
+## High-level requirements
+
+* Python 3.12
+* Unix-based setup (e.g., Linux, MacOS X) with `bash`, `wget` and `curl` installed.
+* Development files for libfreetype, libpng, libxml and libxslt e.g. ``libfreetype6-dev libpng-dev libxml2-dev libxslt-dev``.
+
+## Running the app locally
+### Overview
+The IATI Dashboard is mostly written in Python but also has some helper Bash scripts to collect the data that the dashboard uses.  Top-level steps required to run the Dashboard are:
+
+1. Setup Python environment and install dependencies.
+2. Fetch the data.
+3. Build the static graphs and other data that will be served via the Dashboard.
+4. Run the web server.
+
+Paths to different directories are set in `./src/config.py`.
+
+### 1. Setup environment
+
+Assuming that this repository has been cloned and you are in the root directory of the repository.
+
+```
+# Setup and activate a virtual environment (recommended) - here we use virtualenv
+virtualenv ve
+source ve/bin/activate
+```
+
+Now install the dependencies.
+
+```
+pip install -r requirements.txt
+```
+
+### 2. Fetching the data
+
+Bash scripts are used to fetch the data that the Dashboard will present.  They will store data in `./data` and `./stats-calculated`.
+
+```
+# Fetch the necessary calculated stats
+./get_stats.sh
+
+# Fetch some extra data from github and github gists and other sources on the internet
+./fetch_data.sh
+```
+
+### 3. Build static data and graphs and copy to static
+
+```
+mkdir out
+cd src
+python make_plots.py
+python make_csv.py
+python speakers_kit.py
+cp ../out/data static/
+cp ../img/aggregate static/
+cp ../img/publishers static/
+```
+
+### 4. Run the webserver.
+
+From `./src/`:
+
+```
+python manage.py runserver
+```
+
+The Dashboard will now be accessible from `localhost:8000/`.
+
+
+## Development
+
+### Calculating your own statistics
+
+The IATI Dashboard requires a `stats-calculated` directory, which can be downloaded using the `get_stats.sh` shell script as described above.  This can also be calculated using [Code for IATI Stats](http://github.com/codeforIATI/IATI-Stats) where `stats-calculated` corresponds to the `gitout` directory generated by [`git.sh` in IATI-Stats](https://github.com/codeforIATI/IATI-Stats#running-for-every-commit-in-the-data-directory).
+
+Often you only want to regenerate the current stats, use `get_stats.sh` to download the pre-calculated historical stats and just replace the `stats-calculated/current directory` with the `out` directory produced by running the [loop, aggregate and invert commands individually](https://github.com/codeforIATI/IATI-Stats#getting-started), then regenerate graphs and CSV files as per the above.
+
+### Adding new dependencies
+
+If a change requires new dependencies then please add to `requirements.in` or `requirements_dev.in` as appropriate and recompile:
+
+```
+pip-compile requirements_dev.in
+pip-compile requirements.in
+```
+
+### Linting
+
+Code linting is carried out using [Flake8](https://flake8.pycqa.org/en/latest/) and `setup.cfg` has the configuration.
+
+## License
+    Copyright (C) 2013-2015 Ben Webb <bjwebb67@googlemail.com>
+    Copyright (C) 2013-2014 David Carpenter <caprenter@gmail.com>
+    Copyright (C) 2021 Andy Lulham <a.lulham@gmail.com>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.

common.py → dashboard/common.py

@@ -1,11 +1,11 @@
-# Script to define useful functions
-
+"""Load IATI OrganisationType codelist into a global and provide function to get publisher type"""
 import data
 import json
 
+import config
 
 #  Import organisation_type_codelist as a global, then delete when used to save memory
-with open('data/IATI-Codelists-2/out/clv2/json/en/OrganisationType.json') as fh:
+with open(config.join_data_path('IATI-Codelists-2/out/clv2/json/en/OrganisationType.json')) as fh:
     organisation_type_codelist = json.load(fh)
 organisation_type_dict = {c['code']: c['name'] for c in organisation_type_codelist['data']}
 del organisation_type_codelist

dashboard/config.py

@@ -0,0 +1,37 @@
+"""These functions join path fragments to make directories to different input or output files
+
+Note: eventually these functions will probably become redundant or refactored into
+a different module, but for now this helps in refactoring the code.
+"""
+
+import os.path
+
+
+STATS_DIRECTORY = "../stats-calculated"
+DATA_DIRECTORY = "../data"
+BASE_DIRECTORY = "../"
+OUT_DIRECTORY = "../out"
+
+
+def join_stats_path(p: str) -> str:
+    """Make a path to a file or directory within the downloaded stats directory
+    """
+    return os.path.join(STATS_DIRECTORY, p)
+
+
+def join_data_path(p: str) -> str:
+    """Make a path to a file or directory within the downloaded data directory
+    """
+    return os.path.join(DATA_DIRECTORY, p)
+
+
+def join_base_path(p: str) -> str:
+    """Make a path to a file or directory relative to the base of the dashboard directory
+    """
+    return os.path.join(BASE_DIRECTORY, p)
+
+
+def join_out_path(p: str) -> str:
+    """Make a path to a file or directory relative to the base of the out directory
+    """
+    return os.path.join(OUT_DIRECTORY, p)

coverage.py → dashboard/coverage.py

@@ -1,10 +1,12 @@
 # This file converts a range coverage data to variables which can be outputted on the coverage page
 import csv
+
 from data import get_publisher_stats
 from data import get_registry_id_matches
 from data import publisher_name
 from data import publishers_ordered_by_title
 from data import secondary_publishers
+import config
 
 
 def is_number(s):
@@ -178,7 +180,7 @@ def table():
 
 
 # Compile a list of Development finance institutions (DFIs)
-with open('dfi_publishers.csv', 'r') as csv_file:
+with open(config.join_base_path('dfi_publishers.csv'), 'r') as csv_file:
     reader = csv.reader(csv_file, delimiter=',')
     dfi_publishers = []
     for line in reader: