Skip to content

Latest commit

 

History

History
603 lines (442 loc) · 53.6 KB

README.md

File metadata and controls

603 lines (442 loc) · 53.6 KB
Raw

Data Hub API

CircleCI codecov

Data Hub API provides an API into Data Hub for Data Hub clients. Using Data Hub API you can search for entities and manage companies, contacts and interactions.

More guides can be found in the docs folder.

To instantiate the development environment, please follow one of the three following sets of instructions depending on your use case:

Installation with Docker

This project uses Docker compose to setup and run all the necessary components. The docker-compose.yml file provided is meant to be used for running tests and development.

Note for Mac Users: By default, docker on Mac will restrict itself to using just 2GB of memory. This should be increased to at least 4GB to avoid running in to unexpected problems.

  1. Clone the repository:

    git clone https://github.com/uktrade/data-hub-api
cd data-hub-api

  2. Create .env files from sample.env

    cp sample.env .env
cp config/settings/sample.env config/settings/.env

    If you're working with data-hub-frontend and mock-sso, DJANGO_SUPERUSER_SSO_EMAIL_USER_ID should be the same as MOCK_SSO_EMAIL_USER_ID in mock-sso environment definition in data-hub-frontend/docker-compose.frontend.yml

  3. Build and run the necessary containers for the required environment:

    docker-compose up

    or

    make start-dev

    • It will take time for the API container to come up - it will run migrations on both DBs, load initial data, sync opensearch etc. Watch along in the api container's logs.

    • NOTE: If you are using a linux system, the opensearch container may not come up successfully (data-hub-api_es_1) - it might be perpetually restarting. If the logs for that container mention something like max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144], you will need to run the following on your host machine:

      sudo sysctl -w vm.max_map_count=262144

      and append/modify the vm.max_map_count setting in /etc/sysctl.conf (so that this setting persists after restart):

      vm.max_map_count=262144

      For more information, see the opensearch docs on vm.max_map_count.

  4. Optionally, you may want to run a local copy of the data hub frontend. By default, you can run both the API and the frontend under one docker-compose project. See the instructions in the frontend readme to set it up. Alternatively use the make command documented below if you also want to bring up dnb-service

Installation with docker of data-hub-api, data-hub-frontend and dnb-service on same network

There is now a make command to bring up the three environments on a single docker network, allowing the services to talk to each other effortlessly

  1. Clone the repositories

    git clone https://github.com/uktrade/data-hub-api
git clone https://github.com/uktrade/data-hub-frontend
git clone https://github.com/uktrade/dnb-service
cd data-hub-api

  2. Create .env files from sample.env

    cp sample.env .env
cp config/settings/sample.env config/settings/.env

    Ensure DJANGO_SUPERUSER_SSO_EMAIL_USER_ID is the same as MOCK_SSO_EMAIL_USER_ID in mock-sso environment definition in data-hub-frontend/docker-compose.frontend.yml and DJANGO_SUPERUSER_SSO_EMAIL_USER_ID the same as DJANGO_SUPERUSER_EMAIL in data-hub-api .env file otherwise the user may not exist

    The DNB_SERVICE_BASE_URL should match the dnb-service domain you are trying to access (localhost, staging, uat etc). The DNB_SERVICE_TOKEN is a token generated by dnb-service using the django-rest-framework token authentication. For localhost, you will need to generate a token on dnb-service for your user. For staging, uat or prod, the current tokens can be found in vault which are associated with an already existing api user. See the dnb-service README for more information on how to find or generate new tokens.

  3. Run make command

    make start-frontend-api-dnb
    • It will take time for the API container to come up - it will run migrations on both DBs, load initial data, sync opensearch etc. Watch along in the api container's logs.
    • NOTE: If you are using a linux system, the opensearch container may not come up successfully (data-hub-api_es_1) - it might be perpetually restarting. If the logs for that container mention something like max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144], you will need to run the following on your host machine:
    sudo sysctl -w vm.max_map_count=262144

    and append/modify the vm.max_map_count setting in /etc/sysctl.conf (so that this setting persists after restart):

    vm.max_map_count=262144

    For more information, see the opensearch docs on vm.max_map_count.

  4. If you want to stop all the services, run the following make command

    make stop-frontend-api-dnb

Native installation (without Docker)

Dependencies:

  • Python 3.10.x
  • PostgreSQL 12
  • redis 6.x
  • OpenSearch 1.x

  1. Clone the repository:

    git clone https://github.com/uktrade/data-hub-api
cd data-hub-api

  2. Install Python 3.10.

    See this guide for detailed instructions for different platforms.

  3. Install system dependencies:

    On Ubuntu:

    sudo apt install build-essential libpq-dev python3.10-dev python3.10-venv

    On macOS:

    brew install libpq

  4. Install postgres, if not done already, as this is required by psycopg2 in the requirements below

    On Ubuntu:

    sudo apt install postgresql postgresql-contrib

    On macOS:

    brew install postgresql

  5. Create and activate the virtualenv:

    python3.10 -m venv env
source env/bin/activate
pip install -U pip
or
formally to make sure you have the same version as what is used for cloudfoundry, use buildpack to install the same version e.g. https://github.com/cloudfoundry/python-buildpack/releases e.g. 22.1.2
python -m pip install pip==22.1.2

  6. Install the dependencies:

    pip install -r requirements-dev.txt

  7. Create an .env settings file (it’s gitignored by default):

    cp config/settings/sample.env config/settings/.env

  8. Set DOCKER_DEV=False and LOCAL_DEV=True in .env

  9. Make sure you have OpenSearch running locally. If you don't, you can run one in Docker:

    docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "plugins.security.disabled=true" opensearchproject/opensearch:1.2.4

  10. Make sure you have redis running locally and that the REDIS_BASE_URL in your .env is up-to-date.

  11. Populate the databases and initialise OpenSearch:

    ./manage.py migrate
./manage.py migrate_search

./manage.py loadinitialmetadata
./manage.py createinitialrevisions

  12. Optionally, you can load some test data:

    ./manage.py loaddata fixtures/test_data.yaml

    Note that this will queue RQ tasks to index the created records in OpenSearch, and hence the loaded records won‘t be returned by search endpoints until RQ is started and the queued tasks have run.

  13. Create a superuser:

    ./manage.py createsuperuser

    (You can enter any valid email address as the username and SSO email user ID.)

  14. Start the server:

    ./manage.py runserver

  15. Start RQ (Redis Queue):

    python rq/rq-worker.py

    Note that in production the cron-scheduler:1/1, short-running-working:4/4, long-running-worker:4/4 are run in separate instances .

API documentation

Automatically-generated API documentation is served at /docs (requires admin site credentials).

Local development

If using Docker, prefix these commands with docker-compose run api.

To run the tests:

./tests.sh

To run the tests in parallel, pass -n <number of processes> to ./tests.sh. For example, for four processes:

./tests.sh -n 4

By default all tests are executed. To skip tests that should not run under certain conditions ie. are dependent on external services, use a marker excluded or a marker that start with excluded_.

import pytest

@pytest.mark.excluded
class TestClass:
    def test_1(self):
        ...

@pytest.mark.excluded
def test_1():
    ...

@pytest.mark.excluded_dependent_on_redis
def test_2():
    ...

To exclude such tests from running use one of the options below:

pytest --skip-excluded
pytest -m excluded_dependent_on_redis
pytest -m "excluded_x or excluded_y"
pytest -m "not (excluded_x or excluded_y)"

NOTE: When testing, using the --reuse-db can speed up test runs by excluding migrations

To run the linter:

flake8

There is also a pre-commit hook for Flake8. To install this, run:

make setup-flake8-hook

Obtaining an API access token

You can obtain an access token for local development work in one of two ways:

  • by running ./manage.py add_access_token <SSO email user ID> with the SSO email user ID of an existing adviser (run ./manage.py add_access_token --help for a list of options)
  • using the form on http://localhost:8000/admin/add-access-token/

(If you’re using Docker, an access token will be created automatically if certain environment variables are set. See sample.env for more details.)

This access token can be used with most endpoints by setting an Authorization header value of Bearer <access token>.

Note that machine-to-machine endpoints (such as those under /v4/metadata/) instead use Hawk authentication and request signing.

Granting access to the front end

The internal front end uses single sign-on. You should configure the API as follows to use with the front end:

  • SSO_ENABLED: True
  • STAFF_SSO_BASE_URL: URL of a Staff SSO or Mock SSO instance. This should be the same server the front end is configured to use.
  • STAFF_SSO_AUTH_TOKEN: Access token for Staff SSO.

Granting access to machine-to-machine clients

Pure machine-to-machine clients use Hawk authentication with separate credentials for each client.

There are separate views for such clients as these views don’t expect request.user to be set.

Hawk credentials for each client are defined in settings below and each client is assigned scopes in config/settings/common.py.

These scopes define which views each client can access.

Deployment

Data Hub API can run on any Heroku-style platform. Configuration is performed via the following environment variables:

Variable name Required Description
ACTIVITY_STREAM_ACCESS_KEY_ID No A non-secret access key ID, corresponding to ACTIVITY_STREAM_SECRET_ACCESS_KEY. The holder of the secret key can access the activity stream endpoint by Hawk authentication.
ACTIVITY_STREAM_SECRET_ACCESS_KEY If ACTIVITY_STREAM_ACCESS_KEY_ID is set A secret key, corresponding to ACTIVITY_STREAM_ACCESS_KEY_ID. The holder of this key can access the activity stream endpoint by Hawk authentication.
ACTIVITY_STREAM_OUTGOING_URL No The URL used to read from activity stream
ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID No A non-secret access key ID, corresponding to ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY. This is used when reading from the activity stream at ACTIVITY_STREAM_OUTGOING_URL.
ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY No A secret key, corresponding to ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID. This is used when reading from the activity stream at ACTIVITY_STREAM_OUTGOING_URL.
ADMIN_OAUTH2_ENABLED Yes Enables Django Admin SSO login when is True.
ADMIN_OAUTH2_BASE_URL If ADMIN_OAUTH2_ENABLED is set A base URL of OAuth provider.
ADMIN_OAUTH2_TOKEN_FETCH_PATH If ADMIN_OAUTH2_ENABLED is set OAuth fetch token path for Django Admin SSO login.
ADMIN_OAUTH2_USER_PROFILE_PATH If ADMIN_OAUTH2_ENABLED is set OAuth user profile path for Django Admin SSO login.
ADMIN_OAUTH2_AUTH_PATH If ADMIN_OAUTH2_ENABLED is set OAuth auth path for Django Admin SSO login.
ADMIN_OAUTH2_CLIENT_ID If ADMIN_OAUTH2_ENABLED is set OAuth client ID for Django Admin SSO login.
ADMIN_OAUTH2_CLIENT_SECRET If ADMIN_OAUTH2_ENABLED is set OAuth client secret for Django Admin SSO login.
AV_V2_SERVICE_URL Yes URL for ClamAV V2 service. If not configured, virus scanning will fail.
DEFAULT_BUCKET_AWS_ACCESS_KEY_ID No Used as part of boto3 auto-configuration.
DEFAULT_BUCKET_AWS_DEFAULT_REGION No Default region used by boto3.
DEFAULT_BUCKET_AWS_SECRET_ACCESS_KEY No Used as part of boto3 auto-configuration.
CONSENT_SERVICE_BASE_URL No The base url of the consent service, to post email consent preferences to (default=None).
CONSENT_SERVICE_HAWK_ID No The hawk id to use when making a request to the consent service (default=None).
CONSENT_SERVICE_HAWK_KEY No The hawk key to use when making a request to the consent service (default=None).
CSRF_COOKIE_HTTPONLY No Whether to use HttpOnly flag on the CSRF cookie (default=False).
CSRF_COOKIE_SECURE No Whether to use a secure cookie for the CSRF cookie (default=False).
DATA_FLOW_API_ACCESS_KEY_ID No A non-secret access key ID, corresponding to DATA_FLOW_API_SECRET_ACCESS_KEY. The holder of the secret key can access the omis-dataset endpoint by Hawk authentication.
DATA_FLOW_API_SECRET_ACCESS_KEY If DATA_FLOW_API_ACCESS_KEY_ID is set A secret key, corresponding to DATA_FLOW_API_ACCESS_KEY_ID. The holder of this key can access the omis-dataset endpoint by Hawk authentication.
DATABASE_CONN_MAX_AGE No Maximum database connection age (in seconds).
DATABASE_CREDENTIALS Yes PostgreSQL data base credentials contained in JSON structure: {"username": application_user", "password": "p4s5w0rd!", "engine": "postgres", "port": 5432, "dbname": "main", "host": "hostname.rds.amazonaws.com", "dbInstanceIdentifier", "db-instance"}
DATAHUB_FRONTEND_BASE_URL Yes
DATAHUB_NOTIFICATION_API_KEY No The GOVUK notify API key to use for the datahub.notification django app.
DATAHUB_SUPPORT_EMAIL_ADDRESS No Email address for DataHub support team.
DATA_HUB_ENQUIRY_MGMT_HAWK_ID No A non-secret access key ID, corresponding to DATA_HUB_ENQUIRY_MGMT_HAWK_SECRET_KEY. The holder of the secret key can access the metadata endpoints by Hawk authentication.
DATA_HUB_ENQUIRY_MGMT_HAWK_SECRET_KEY If DATA_HUB_ENQUIRY_MGMT_HAWK_ID is set A secret key, corresponding to METADATA_ACCESS_KEY_ID. The holder of this key can access the metadata endpoints by Hawk authentication.
DATA_HUB_FRONTEND_ACCESS_KEY_ID No A non-secret access key ID, corresponding to DATA_HUB_FRONTEND_SECRET_ACCESS_KEY. The holder of the secret key can access the metadata endpoints by Hawk authentication.
DATA_HUB_FRONTEND_SECRET_ACCESS_KEY If DATA_HUB_FRONTEND_ACCESS_KEY_ID is set A secret key, corresponding to METADATA_ACCESS_KEY_ID. The holder of this key can access the metadata endpoints by Hawk authentication.
DEBUG Yes Whether Django's debug mode should be enabled.
DIT_EMAIL_DOMAIN_* No An allowable DIT email domain for email ingestion along with it's allowed email authentication methods. Django-environ dict format e.g. example.com=dmarc:pass implementation either spf:pass or dkim:pass
DIT_EMAIL_INGEST_BLOCKLIST No A list of emails for which email ingestion is prohibited.
DJANGO_SECRET_KEY Yes
DJANGO_SENTRY_DSN Yes
DJANGO_SETTINGS_MODULE Yes
DNB_AUTOMATIC_UPDATE_LIMIT No Integer of the maximum number of updates the DNB automatic update task should ingest before exiting. This is unlimited if this setting is not set.
DNB_SERVICE_BASE_URL No The base URL of the DNB service.
DNB_SERVICE_TOKEN No Access token provided by dnb-service through it's django-rest-framework token authentication.
DEFAULT_BUCKET Yes S3 bucket for object storage.
DISABLE_PAAS_IP_CHECK No Disable PaaS IP check for Hawk endpoints (default=False).
ENABLE_ADMIN_ADD_ACCESS_TOKEN_VIEW No Whether to enable the add access token page for superusers in the admin site (default=True).
ENABLE_AUTOMATIC_REMINDER_ITA_USER_MIGRATIONS No Whether to enable automatic migration of ITA users to receive notification reminders
ENABLE_AUTOMATIC_REMINDER_POST_USER_MIGRATIONS No Whether to enable automatic migration of POST users to receive notification reminders
ENABLE_DAILY_OPENSEARCH_SYNC No Whether to enable the daily OpenSearch sync (default=False).
ENABLE_INVESTMENT_NOTIFICATION No True or False. Whether or not to activate the RQ task for sending investment notifications
ENABLE_ESTIMATED_LAND_DATE_REMINDERS No True or False. Whether or not to activate the RQ task for sending investment notifications
ENABLE_ESTIMATED_LAND_DATE_REMINDERS_EMAIL_DELIVERY_STATUS No True or False. Whether or not to activate the RQ task for checking delivery status
ENABLE_NEW_EXPORT_INTERACTION_REMINDERS No True or False. Whether or not to activate the RQ task for sending new export interaction notifications
ENABLE_NEW_EXPORT_INTERACTION_REMINDERS_EMAIL_DELIVERY_STATUS No True or False. Whether or not to activate the RQ task for checking delivery status
ENABLE_NO_RECENT_EXPORT_INTERACTION_REMINDERS No True or False. Whether or not to activate the RQ task for sending no recent interaction notifications
ENABLE_NO_RECENT_EXPORT_INTERACTION_REMINDERS_EMAIL_DELIVERY_STATUS No True or False. Whether or not to activate the RQ task for checking delivery status
ENABLE_NO_RECENT_INTERACTION_REMINDERS No True or False. Whether or not to activate the RQ task for sending investment notifications
ENABLE_MAILBOX_PROCESSING No True or False. Whether or not to activate the RQ task for mailbox processing
ENABLE_SLACK_MESSAGING No If present and truthy, enable the transmission of messages to Slack. Necessitates the specification of the other env vars SLACK_API_TOKEN and SLACK_MESSAGE_CHANNEL
ENABLE_SPI_REPORT_GENERATION No Whether to enable daily SPI report (default=False).
ES_APM_ENABLED Yes Enables Elasticsearch APM agent when is True.
ES_APM_SERVICE_NAME Yes, if ES_APM_ENABLED A name of the running service. Must match following regexp: ^[a-zA-Z0-9 _-]+$.
ES_APM_SECRET_TOKEN Yes, if ES_APM_ENABLED A secret token used to authorise requests to the APM server.
ES_APM_SERVER_TIMEOUT No A timeout for requests to the Elasticsearch APM server in duration format (default=20s).
ES_APM_SERVER_URL Yes, if ES_APM_ENABLED The URL of the Elasticsearch APM server.
ES_APM_ENVIRONMENT Yes, if ES_APM_ENABLED A name of the environment the service is running, for example: develop.
EXPORT_WINS_SERVICE_BASE_URL No The base url of the Export Wins API (default=None).
EXPORT_WINS_HAWK_ID No The hawk id to use when making a request to the Export Wins API (default=None).
EXPORT_WINS_HAWK_KEY No The hawk key to use when making a request to the Export Wins API (default=None).
EXTRA_DJANGO_APPS Yes Additional Django apps to load (comma-separated). Can be used to reverse the migrations of a removed third-party app (see comment in config/settings/common.py for more detail).
INTERACTION_ADMIN_CSV_IMPORT_MAX_SIZE No Maximum file size in bytes for interaction admin CSV uploads (default=2MB).
INTERACTION_NOTIFICATION_API_KEY Yes
INVESTMENT_DOCUMENT_AWS_ACCESS_KEY_ID No Same use as DEFAULT_BUCKET_AWS_ACCESS_KEY_ID, but for investment project documents.
INVESTMENT_DOCUMENT_AWS_SECRET_ACCESS_KEY No Same use as DEFAULT_BUCKET_AWS_SECRET_ACCESS_KEY, but for investment project documents.
INVESTMENT_DOCUMENT_AWS_REGION No Same use as DEFAULT_BUCKET_AWS_DEFAULT_REGION, but for investment project documents.
INVESTMENT_DOCUMENT_BUCKET No S3 bucket for investment project documents storage.
INVESTMENT_NOTIFICATION_ADMIN_EMAIL Yes
INVESTMENT_NOTIFICATION_API_KEY Yes
INVESTMENT_NOTIFICATION_ESTIMATED_LAND_DATE_TEMPLATE_ID Yes An ID of Notify Template for Estimated Land Date notifications
INVESTMENT_NOTIFICATION_ESTIMATED_LAND_DATE_SUMMARY_TEMPLATE_ID Yes An ID of Notify Template for Estimated Land Date summary notifications
INVESTMENT_NOTIFICATION_NO_RECENT_INTERACTION_TEMPLATE_ID Yes An ID of Notify Template for No Recent Investment Interaction notifications
EXPORT_NOTIFICATION_NO_RECENT_INTERACTION_TEMPLATE_ID Yes An ID of Notify Template for No Recent Export company Interaction notifications
EXPORT_NOTIFICATION_NO_INTERACTION_TEMPLATE_ID Yes An ID of Notify Second template for No Recent Export company Interaction notifications
EXPORT_NOTIFICATION_NO_RECENT_INTERACTION_TEMPLATE_ID Yes An ID of Notify Template for No Recent Export company Interaction notifications
EXPORT_WIN_NOTIFICATION_API_KEY Yes
EXPORT_WIN_CLIENT_RECEIPT_TEMPLATE_ID Yes An ID of Notify Template for Export Win Client Email Receipt notifications
EXPORT_WIN_LEAD_OFFICER_APPROVED_TEMPLATE_ID Yes An ID of Notify Template for Export Win Lead Officer Approved notifications
EXPORT_WIN_LEAD_OFFICER_REJECTED_TEMPLATE_ID Yes An ID of Notify Template for Export Win Lead Officer Rejected notifications
TASK_REMINDER_EMAIL_TEMPLATE_ID No An ID of Notify Template for the generic Task reminder notifications
TASK_NOTIFICATION_FROM_OTHERS_TEMPLATE_ID Yes An ID of Notify Template for Task assigned by others notifications
MAILBOX_AWS_ACCESS_KEY_ID No Same use as AWS_ACCESS_KEY_ID, but for mailbox.
MAILBOX_AWS_SECRET_ACCESS_KEY No Same use as AWS_SECRET_ACCESS_KEY, but for mailbox.
MAILBOX_AWS_REGION No Same use as AWS_DEFAULT_REGION, but for mailbox.
MAILBOX_BUCKET No S3 bucket for mailbox storage.
MAILBOX_INGESTION_SUCCESS_TEMPLATE_ID No An ID of Notify Template for mailbox ingestion success
MAILBOX_INGESTION_FAILURE_TEMPLATE_ID No An ID of Notify Template for mailbox ingestion failure
MARKET_ACCESS_ACCESS_KEY_ID No A non-secret access key ID used by the Market Access service to access Hawk-authenticated public company endpoints.
MARKET_ACCESS_SECRET_ACCESS_KEY If MARKET_ACCESS_ACCESS_KEY_ID is set A secret key used by the Market Access service to access Hawk-authenticated public company endpoints.
NOTIFICATION_SUMMARY_THRESHOLD No Number of notification items that trigger sending a summary email. (default=5)
OMIS_PUBLIC_ACCESS_KEY_ID No A non-secret access key ID, corresponding to OMIS_PUBLIC_SECRET_ACCESS_KEY. The holder of the secret key can access the OMIS public endpoints by Hawk authentication.
OMIS_NOTIFICATION_ADMIN_EMAIL Yes
OMIS_NOTIFICATION_API_KEY Yes
OMIS_NOTIFICATION_OVERRIDE_RECIPIENT_EMAIL No
OMIS_PUBLIC_BASE_URL Yes
OMIS_PUBLIC_SECRET_ACCESS_KEY If OMIS_PUBLIC_ACCESS_KEY_ID is set A secret key, corresponding to OMIS_PUBLIC_ACCESS_KEY_ID. The holder of this key can access the OMIS public endpoints by Hawk authentication.
OPENSEARCH_INDEX_PREFIX Yes Prefix to use for indices and aliases
OPENSEARCH_SEARCH_REQUEST_TIMEOUT No Timeout (in seconds) for searches (default=20).
OPENSEARCH_SEARCH_REQUEST_WARNING_THRESHOLD No Threshold (in seconds) for emitting warnings about slow searches (default=10).
OPENSEARCH_VERIFY_CERTS No
OPENSEARCH_POOL_MAXSIZE No The OpenSearch Python client max connection pool
(default=10).
PAAS_IP_ALLOWLIST No IP addresses (comma-separated) that can access the Hawk-authenticated endpoints.
REDIS_BASE_URL No redis base URL without the db
REDIS_CACHE_DB No redis db for django cache (default 0)
REPORT_AWS_ACCESS_KEY_ID No Same use as DEFAULT_BUCKET_AWS_ACCESS_KEY_ID, but for reports.
REPORT_AWS_SECRET_ACCESS_KEY No Same use as DEFAULT_BUCKET_AWS_SECRET_ACCESS_KEY, but for reports.
REPORT_AWS_REGION No Same use as DEFAULT_BUCKET_AWS_DEFAULT_REGION, but for reports.
REPORT_BUCKET No S3 bucket for report storage.
SECTOR_ENVIRONMENT No Only set to "production", in production environment. Otherwise it can be left empty.
SENTRY_ENVIRONMENT Yes Value for the environment tag in Sentry.
SKIP_OPENSEARCH_MAPPING_MIGRATIONS No If non-empty, skip applying OpenSearch mapping type migrations on deployment.
SLACK_API_TOKEN No (Required if ENABLE_SLACK_MESSAGING is truthy) Auth token for connection to Slack API for purposes of sending messages through the datahub.core.realtime_messaging module
SLACK_MESSAGE_CHANNEL No (Required if ENABLE_SLACK_MESSAGING is truthy) Name (or preferably ID) of the channel into which datahub.core.realtime_messaging should send messages
SSO_ENABLED Yes Whether single sign-on via RFC 7662 token introspection is enabled
STAFF_SSO_AUTH_TOKEN If SSO enabled Access token for the Staff SSO API.
STAFF_SSO_BASE_URL If SSO enabled The base URL for the Staff SSO API.
STAFF_SSO_REQUEST_TIMEOUT No Staff SSO API request timeout in seconds (default=5).
STATSD_HOST No StatsD host url.
STATSD_PORT No StatsD port number.
STATSD_PREFIX No Prefix for metrics being pushed to StatsD.
VCAP_SERVICES No Set by GOV.UK PaaS when using their backing services. Contains connection details for OpenSearch and Redis.

Management commands

If using Docker, remember to run these commands inside your container by prefixing them with docker-compose run api.

Database

Apply migrations

For the default database
./manage.py migrate

Create django-reversion initial revisions

If the database is freshly built or a new versioned model is added run:

./manage.py createinitialrevisions

Load initial metadata

These commands are generally only intended to be used on a blank database.

./manage.py loadinitialmetadata

OpenSearch

Update indexes and mapping types

To create missing OpenSearch indexes and migrate modified mapping types:

./manage.py migrate_search

This will also resync data (using RQ) for any newly-created indexes.

See docs/OpenSearch migrations.md for more detail about how the command works.

Resync all OpenSearch records

To resync all records using RQ:

./manage.py sync_search

To resync all records synchronously (without RQ running):

./manage.py sync_search --foreground

You can resync only specific models by using the --model= argument.

./manage.py sync_search --model=company --model=contact

For more details including all the available choices:

./manage.py sync_search --help

Dependencies

See Managing dependencies for information about installing, adding and upgrading dependencies.

Activity Stream

The /v3/activity-stream/* endpoints are protected by two mechanisms:

  • IP address allowlisting via the X-Forwarded-For header, with a comma separated list of allowlisted IPs in the environment variable PAAS_IP_ALLOWLIST.

  • Hawk authentication via the Authorization header, with the credentials in the environment variables ACTIVITY_STREAM_ACCESS_KEY_ID and ACTIVITY_STREAM_SECRET_ACCESS_KEY.

IP address allowlisted

The authentication blocks requests that do not have a allowlisteded IP in the second-from-the-end IP in X-Forwarded-For header. In general, this cannot be trusted. However, in PaaS, this can be, and this is the only production environment. Ideally, this would be done at a lower level than HTTP, but this is not possible with the current architecture.

If making requests to this endpoint locally, you must manually add this header or disable the check using DISABLE_PAAS_IP_CHECK environment variable.

Hawk authentication

In general, Hawk authentication hashing the HTTP payload and Content-Type header, and using a nonce, are both optional. Here, as with the Activity Stream endpoints in other DIT projects, both are required. Content-Type may be the empty string, and if there is no payload, then it should be treated as the empty string.

Emails

Email templates are created in GOV UK Notify. You can create a personal account for testing new templates or template changes locally. To test locally you need to update the API Keys and template ID environment variables.

Updating emails

Only update the live email templates once your changes are pushed live. Extra parameters passed to email templates are ignored but not sending parameters required by an email template will cause the emails to error.

Example of environment variables for updating the interaction notification email template:

INTERACTION_NOTIFICATION_API_KEY=<noftify-api-key>
EXPORT_NOTIFICATION_NEW_INTERACTION_TEMPLATE_ID=<template-id>