Digitaal Loket Data warehouse

Started from the mu-project template.

Running and maintaining

General information on running, maintaining and installing the stack.

How to start the stack

Prerequisites

• docker, docker-compose and git are installed on your system
• you've cloned the repository

Start the system:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up

docker-compose up CLI reference.

Wait for everything to boot to ensure clean caches. You may choose to monitor the migration service in a separate terminal and wait for the overview of all migrations to appear:

docker-compose logs -f --tail=100 migrations

docker-compose logs CLI reference.

With environement variables

You might find the above docker-compose up command tedious. To simplify it's usage we can define the COMPOSE_FILE variable in our environment.

Create an .env file in the root of the project with the following contence:

COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml

docker-compose CLI env. vars. reference

Start the system:

docker-compose up

Sync data from digitaal loket (Mandaten en Leidingevenden)

Set up a sync to load Mandaten en Leidingevenden from a Digitaal Loket instance to Digitaal Loket Data Warehouse with the delta-consumer.

An example docker-compose.override.yml configuration. See delta-consumer for more info.

version: '3.7'

services:
  mandatendatabank-consumer:
    environment:
      DCR_SYNC_BASE_URL: 'https://my-digital-loket-instance.net'
      DCR_DISABLE_INITIAL_SYNC: 'false'
      BATCH_SIZE: 1000
  leidinggevenden-consumer:
    environment:
      DCR_SYNC_BASE_URL: 'https://my-digital-loket-instance.net'
      DCR_DISABLE_INITIAL_SYNC: 'false'
      BATCH_SIZE: 1000
  persons-sensitive-consumer:
    environment:
      DCR_SECRET_KEY: 'shared key with producer stack'
      DCR_SYNC_BASE_URL: 'https://my-digital-loket-instance.net'
      DCR_SYNC_LOGIN_ENDPOINT: 'http://my-digital-loket-instance.net/sync/persons-sensitive/login'
      DCR_DISABLE_INITIAL_SYNC: 'false'