Merge pull request #425 from maykinmedia/feature/envvar-documentation

 📝 [open-zaak/open-zaak#1649] Document envvars

.github/workflows/ci.yml

@@ -63,6 +63,37 @@ jobs:
       - name: Publish coverage report
         uses: codecov/codecov-action@v3
 
+  docs:
+    runs-on: ubuntu-latest
+    name: Documentation build
+
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+          cache-dependency-path: 'requirements/*.txt'
+      - name: Install system packages
+        run: |
+          sudo apt-get update \
+          && sudo apt-get install -y --no-install-recommends \
+            libgdal-dev \
+            gdal-bin
+      - name: Install dependencies
+        run: pip install -r requirements/ci.txt pytest
+      - name: Generate environment variable documentation using OAf and check if it was updated
+        run: |
+          bin/generate_envvar_docs.sh
+          changes=$(git diff docs/installation/config.rst)
+          if [ ! -z "$changes" ]; then
+              echo $changes
+              echo "Please update the environment documentation by running \`bin/generate_envvar_docs.sh\`"
+              exit 1
+          fi
+        env:
+          DJANGO_SETTINGS_MODULE: objects.conf.ci
+
   docker:
     needs: tests
 

bin/generate_envvar_docs.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+# Generates the documentation for environment variables
+src/manage.py generate_envvar_docs --file docs/installation/config.rst

docs/installation/config.rst

@@ -1,109 +1,100 @@
-.. _installation_environment_config:
+.. _installation_env_config:
 
 ===================================
 Environment configuration reference
 ===================================
 
-The Objects and Objecttypes APIs can be run both as a Docker container or a VPS / dedicated server. It relies on other services, such as
-database and cache backends, which can be configured through environment
-variables.
 
-Available environment variables
-===============================
-
-Required settings
------------------
-
-* ``DJANGO_SETTINGS_MODULE``: which environment settings to use. Available options:
-
-  - ``objects.conf.docker`` or ``objecttypes.conf.docker``
-  - ``objects.conf.dev`` or ``objecttypes.conf.dev``
-  - ``objects.conf.ci`` or ``objecttypes.conf.ci``
-
-* ``SECRET_KEY``: secret key that's used for certain cryptographic utilities. You
-  should generate one via
-  `miniwebtool <https://www.miniwebtool.com/django-secret-key-generator/>`_
+The Objects and Objecttypes APIs can be run both as a Docker container or a VPS / dedicated server.
+It relies on other services, such as database and cache backends, which can be configured through environment variables.
 
-* ``ALLOWED_HOSTS``: A comma separated (without spaces!) list of domains that
-  serve the installation. Used to protect against ``Host`` header attacks.
-  Defaults to ``*`` for the ``docker`` environment and defaults to
-  ``127.0.0.1,localhost`` for the ``dev`` environment.
 
-Common settings
----------------
-
-* ``DB_HOST``: Hostname of the PostgreSQL database. Defaults to ``db`` for the
-  ``docker`` environment, otherwise defaults to ``localhost``.
-
-* ``DB_USER``: Username of the database user. Defaults to ``objects`` or ``objecttypes``,
+Available environment variables
+===============================
 
-* ``DB_PASSWORD``: Password of the database user. Defaults to ``objects`` or ``objecttypes``,
 
-* ``DB_NAME``: Name of the PostgreSQL database. Defaults to ``objects`` or ``objecttypes``,
+Required
+--------
 
-* ``DB_PORT``: Port number of the database. Defaults to ``5432``.
+* ``SECRET_KEY``: Secret key that's used for certain cryptographic utilities. You should generate one via `miniwebtool <https://www.miniwebtool.com/django-secret-key-generator>`_.
+* ``ALLOWED_HOSTS``: a comma separated (without spaces!) list of domains that serve the installation. Used to protect against Host header attacks. Defaults to: ``(empty string)``.
 
-* ``CELERY_BROKER_URL``: URL for the Redis task broker for Celery. Defaults
-  to ``redis://127.0.0.1:6379/1``.
 
-* ``CELERY_RESULT_BACKEND``: URL for the Redis result broker for Celery.
-  Defaults to ``redis://127.0.0.1:6379/1``.
+Database
+--------
 
-Elastic APM settings
---------------------
+* ``DB_NAME``: name of the PostgreSQL database. Defaults to: ``objects``.
+* ``DB_USER``: username of the database user. Defaults to: ``objects``.
+* ``DB_PASSWORD``: password of the database user. Defaults to: ``objects``.
+* ``DB_HOST``: hostname of the PostgreSQL database. Defaults to ``db`` for the docker environment, otherwise defaults to ``localhost``.
+* ``DB_PORT``: port number of the database. Defaults to: ``5432``.
 
-An integration with `Elastic APM <https://www.elastic.co/observability/application-performance-monitoring>`_
-can be configured by setting the following environment variables
 
-* ``ELASTIC_APM_SERVICE_NAME``: the name of the service in APM, i.e. "Objects API - staging"
+Cross-Origin-Resource-Sharing
+-----------------------------
 
-* ``ELASTIC_APM_SERVER_URL``: the URL of the APM instance to connect with
+* ``CORS_ALLOW_ALL_ORIGINS``: allow cross-domain access from any client. Defaults to: ``False``.
+* ``CORS_ALLOWED_ORIGINS``: explicitly list the allowed origins for cross-domain requests. Example: http://localhost:3000,https://some-app.gemeente.nl. Defaults to: ``[]``.
+* ``CORS_ALLOWED_ORIGIN_REGEXES``: same as ``CORS_ALLOWED_ORIGINS``, but supports regular expressions. Defaults to: ``[]``.
+* ``CORS_EXTRA_ALLOW_HEADERS``: headers that are allowed to be sent as part of the cross-domain request. By default, Authorization, Accept-Crs and Content-Crs are already included. The value of this variable is added to these already included headers. Defaults to: ``[]``.
 
-* ``ELASTIC_APM_SECRET_TOKEN``: the token that is required to communicate with the APM instance
 
-Other settings
---------------
-* ``ENVIRONMENT``: An identifier for the environment, displayed in the admin depending on
-  the settings module used and included in the error monitoring (see ``SENTRY_DSN``).
-  The default is set according to ``DJANGO_SETTINGS_MODULE``. Good examples values are:
+Celery
+------
 
-  * ``production``
-  * ``test``
+* ``CELERY_RESULT_BACKEND``: the URL of the backend/broker that will be used by Celery to send the notifications. Defaults to: ``redis://localhost:6379/1``.
+* ``CELERY_TASK_HARD_TIME_LIMIT``: Task hard time limit in seconds. The worker processing the task will be killed and replaced with a new one when this is exceeded. Defaults to: ``900``.
 
-* ``SITE_ID``: The database ID of the site object. Defaults to ``1``.
 
-* ``DEBUG``: Used for more traceback information on development environment.
-  Various other security settings are derived from this setting! Defaults to
-  ``True`` for the ``dev`` environment, otherwise defaults to ``False``.
+Elastic APM
+-----------
 
-* ``IS_HTTPS``: Used to construct absolute URLs and controls a variety of
-  security settings. Defaults to the inverse of ``DEBUG``.
+* ``ELASTIC_APM_SERVER_URL``: URL where Elastic APM is hosted. Defaults to: ``None``.
+* ``ELASTIC_APM_SERVICE_NAME``: Name of the service for this application in Elastic APM. Defaults to ``objects - <environment>``.
+* ``ELASTIC_APM_SECRET_TOKEN``: Token used to communicate with Elastic APM. Defaults to: ``default``.
+* ``ELASTIC_APM_TRANSACTION_SAMPLE_RATE``: By default, the agent will sample every transaction (e.g. request to your service). To reduce overhead and storage requirements, set the sample rate to a value between 0.0 and 1.0. Defaults to: ``0.1``.
 
-* ``SUBPATH``: If hosted on a subpath, provide the value here. If you provide
-  ``/gateway``, the component assumes its running at the base URL:
-  ``https://somedomain/gateway/``. Defaults to an empty string.
 
-* ``SENTRY_DSN``: URL of the sentry project to send error reports to. Defaults
-  to an empty string (ie. no monitoring).
+Optional
+--------
 
-* ``NOTIFICATIONS_DISABLED``: indicates whether or not notifications should be
-  sent to the Notificaties API for operations on the Object endpoint.
-  Defaults to ``True`` for the ``dev`` environment, otherwise defaults to ``False``.
+* ``SITE_ID``: The database ID of the site object. You usually won't have to touch this. Defaults to: ``1``.
+* ``DEBUG``: Only set this to ``True`` on a local development environment. Various other security settings are derived from this setting!. Defaults to: ``False``.
+* ``USE_X_FORWARDED_HOST``: whether to grab the domain/host from the X-Forwarded-Host header or not. This header is typically set by reverse proxies (such as nginx, traefik, Apache...). Note: this is a header that can be spoofed and you need to ensure you control it before enabling this. Defaults to: ``False``.
+* ``IS_HTTPS``: Used to construct absolute URLs and controls a variety of security settings. Defaults to the inverse of ``DEBUG``.
+* ``CACHE_DEFAULT``: redis cache address for the default cache. Defaults to: ``localhost:6379/0``.
+* ``CACHE_AXES``: redis cache address for the brute force login protection cache. Defaults to: ``localhost:6379/0``.
+* ``EMAIL_HOST``: hostname for the outgoing e-mail server. Defaults to: ``localhost``.
+* ``EMAIL_PORT``: port number of the outgoing e-mail server. Note that if you're on Google Cloud, sending e-mail via port 25 is completely blocked and you should use 487 for TLS. Defaults to: ``25``.
+* ``EMAIL_HOST_USER``: username to connect to the mail server. Defaults to: ``(empty string)``.
+* ``EMAIL_HOST_PASSWORD``: password to connect to the mail server. Defaults to: ``(empty string)``.
+* ``EMAIL_USE_TLS``: whether to use TLS or not to connect to the mail server. Should be True if you're changing the ``EMAIL_PORT`` to 487. Defaults to: ``False``.
+* ``DEFAULT_FROM_EMAIL``: The default email address from which emails are sent. Defaults to: ``objects@example.com``.
+* ``LOG_STDOUT``: whether to log to stdout or not. Defaults to: ``False``.
+* ``LOG_LEVEL``: control the verbosity of logging output. Available values are ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO`` and ``DEBUG``. Defaults to: ``WARNING``.
+* ``LOG_QUERIES``: enable (query) logging at the database backend level. Note that you must also set ``DEBUG=1``, which should be done very sparingly!. Defaults to: ``False``.
+* ``LOG_REQUESTS``: enable logging of the outgoing requests. Defaults to: ``False``.
+* ``SESSION_COOKIE_SAMESITE``: The value of the SameSite flag on the session cookie. This flag prevents the cookie from being sent in cross-site requests thus preventing CSRF attacks and making some methods of stealing session cookie impossible. Defaults to: ``Strict``.
+* ``CSRF_COOKIE_SAMESITE``: The value of the SameSite flag on the CSRF cookie. This flag prevents the cookie from being sent in cross-site requests. Defaults to: ``Strict``.
+* ``ENVIRONMENT``: An identifier for the environment, displayed in the admin depending on the settings module used and included in the error monitoring (see ``SENTRY_DSN``). The default is set according to ``DJANGO_SETTINGS_MODULE``.
+* ``SUBPATH``: If hosted on a subpath, provide the value here. If you provide ``/gateway``, the component assumes its running at the base URL: ``https://somedomain/gateway/``. Defaults to an empty string. Defaults to: ``None``.
+* ``RELEASE``: The version number or commit hash of the application (this is also sent to Sentry).
+* ``NUM_PROXIES``: the number of reverse proxies in front of the application, as an integer. This is used to determine the actual client IP adres. On Kubernetes with an ingress you typically want to set this to 2. Defaults to: ``1``.
+* ``CSRF_TRUSTED_ORIGINS``: A list of trusted origins for unsafe requests (e.g. POST). Defaults to: ``[]``.
+* ``NOTIFICATIONS_DISABLED``: indicates whether or not notifications should be sent to the Notificaties API for operations on the API endpoints. Defaults to ``True`` for the ``dev`` environment, otherwise defaults to ``False``.
+* ``DISABLE_2FA``: Whether or not two factor authentication should be disabled. Defaults to: ``False``.
+* ``LOG_OUTGOING_REQUESTS_EMIT_BODY``: Whether or not outgoing request bodies should be logged. Defaults to: ``True``.
+* ``LOG_OUTGOING_REQUESTS_DB_SAVE``: Whether or not outgoing request logs should be saved to the database. Defaults to: ``False``.
+* ``LOG_OUTGOING_REQUESTS_DB_SAVE_BODY``: Whether or not outgoing request bodies should be saved to the database. Defaults to: ``True``.
+* ``LOG_OUTGOING_REQUESTS_MAX_AGE``: The amount of time after which request logs should be deleted from the database. Defaults to: ``7``.
+* ``SENTRY_DSN``: URL of the sentry project to send error reports to. Default empty, i.e. -> no monitoring set up. Highly recommended to configure this.
 
-* ``USE_X_FORWARDED_HOST``: whether to grab the domain/host from the ``X-Forwarded-Host``
-  header or not. This header is typically set by reverse proxies (such as nginx,
-  traefik, Apache...). Default ``False`` - this is a header that can be spoofed and you
-  need to ensure you control it before enabling this.
 
-* ``DISABLE_2FA``: whether to disable two-factor authentication. Defaults to ``False``.
-  If set to ``False``, 2FA will be required if not using OIDC.
 
-* ``LOG_REQUESTS``: whether to enable logging of outgoing HTTP requests. Defaults to ``False``.
 
-* ``LOG_OUTGOING_REQUESTS_DB_SAVE``: whether to save logged requests to the database. Defaults to ``False``.
 
-Initial superuser creation
---------------------------
+Initial superuser creation (Docker only)
+----------------------------------------
 
 A clean installation of Objects API comes without pre-installed or pre-configured admin
 user by default.
@@ -127,6 +118,7 @@ which allows configuration via environment variables.
 All these environment variables are described at :ref:`command line <installation_config_cli>`.
 
 
+
 Specifying the environment variables
 =====================================
 

requirements/base.txt

@@ -235,7 +235,7 @@ notifications-api-common==0.2.2
     # via
     #   -r requirements/base.in
     #   commonground-api-common
-open-api-framework==0.6.1
+open-api-framework==0.7.1
     # via -r requirements/base.in
 orderedmultidict==1.0.1
     # via furl

requirements/ci.txt

@@ -234,6 +234,7 @@ django-solo==2.2.0
 django-two-factor-auth[phonenumberslite,webauthn]==1.17.0
     # via
     #   -r requirements/base.txt
+    #   django-two-factor-auth
     #   maykin-2fa
 django-webtest==1.9.7
     # via -r requirements/test-tools.in
@@ -375,7 +376,7 @@ notifications-api-common==0.2.2
     # via
     #   -r requirements/base.txt
     #   commonground-api-common
-open-api-framework==0.6.1
+open-api-framework==0.7.1
     # via -r requirements/base.txt
 orderedmultidict==1.0.1
     # via

requirements/dev.txt

@@ -260,6 +260,7 @@ django-solo==2.2.0
 django-two-factor-auth[phonenumberslite,webauthn]==1.17.0
     # via
     #   -r requirements/ci.txt
+    #   django-two-factor-auth
     #   maykin-2fa
 django-webtest==1.9.7
     # via -r requirements/ci.txt
@@ -424,7 +425,7 @@ notifications-api-common==0.2.2
     # via
     #   -r requirements/ci.txt
     #   commonground-api-common
-open-api-framework==0.6.1
+open-api-framework==0.7.1
     # via -r requirements/ci.txt
 orderedmultidict==1.0.1
     # via

src/objects/conf/base.py

@@ -5,17 +5,7 @@
 
 init_sentry()
 
-
-DATABASES = {
-    "default": {
-        "ENGINE": "django.contrib.gis.db.backends.postgis",
-        "NAME": config("DB_NAME", "objects"),
-        "USER": config("DB_USER", "objects"),
-        "PASSWORD": config("DB_PASSWORD", "objects"),
-        "HOST": config("DB_HOST", "localhost"),
-        "PORT": config("DB_PORT", 5432),
-    }
-}
+DATABASES["default"]["ENGINE"] = "django.contrib.gis.db.backends.postgis"
 
 
 # Application definition
@@ -82,14 +72,17 @@
 
 # settings for sending notifications
 NOTIFICATIONS_KANAAL = "objecten"
-NOTIFICATIONS_DISABLED = config("NOTIFICATIONS_DISABLED", False)
 
-# TODO should this be moved to open-api-framework?
 # Add (by default) 5 (soft), 15 (hard) minute timeouts to all Celery tasks.
-CELERY_TASK_TIME_LIMIT = config("CELERY_TASK_HARD_TIME_LIMIT", default=15 * 60)  # hard
-CELERY_TASK_SOFT_TIME_LIMIT = config(
-    "CELERY_TASK_SOFT_TIME_LIMIT", default=5 * 60
-)  # soft
+CELERY_TASK_TIME_LIMIT = config(
+    "CELERY_TASK_HARD_TIME_LIMIT",
+    default=15 * 60,
+    help_text=(
+        "Task hard time limit in seconds. The worker processing the task will be "
+        "killed and replaced with a new one when this is exceeded."
+    ),
+    group="Celery",
+)  # hard
 
 #
 # Django setup configuration
@@ -107,22 +100,24 @@
 
 # setup_configuration command
 # sites config
-SITES_CONFIG_ENABLE = config("SITES_CONFIG_ENABLE", default=True)
-OBJECTS_DOMAIN = config("OBJECTS_DOMAIN", "")
-OBJECTS_ORGANIZATION = config("OBJECTS_ORGANIZATION", "")
+SITES_CONFIG_ENABLE = config("SITES_CONFIG_ENABLE", default=True, add_to_docs=False)
+OBJECTS_DOMAIN = config("OBJECTS_DOMAIN", "", add_to_docs=False)
+OBJECTS_ORGANIZATION = config("OBJECTS_ORGANIZATION", "", add_to_docs=False)
 # objecttypes config
 OBJECTS_OBJECTTYPES_CONFIG_ENABLE = config(
-    "OBJECTS_OBJECTTYPES_CONFIG_ENABLE", default=True
+    "OBJECTS_OBJECTTYPES_CONFIG_ENABLE", default=True, add_to_docs=False
 )
-OBJECTTYPES_API_ROOT = config("OBJECTTYPES_API_ROOT", "")
-if OBJECTTYPES_API_ROOT and not OBJECTTYPES_API_ROOT.endswith("/"):
+OBJECTTYPES_API_ROOT = config("OBJECTTYPES_API_ROOT", "", add_to_docs=False)
+if OBJECTTYPES_API_ROOT and not OBJECTTYPES_API_ROOT.endswith("/", add_to_docs=False):
     OBJECTTYPES_API_ROOT = f"{OBJECTTYPES_API_ROOT.strip()}/"
 OBJECTTYPES_API_OAS = config(
-    "OBJECTTYPES_API_OAS", default=f"{OBJECTTYPES_API_ROOT}schema/openapi.yaml"
+    "OBJECTTYPES_API_OAS",
+    default=f"{OBJECTTYPES_API_ROOT}schema/openapi.yaml",
+    add_to_docs=False,
 )
-OBJECTS_OBJECTTYPES_TOKEN = config("OBJECTS_OBJECTTYPES_TOKEN", "")
+OBJECTS_OBJECTTYPES_TOKEN = config("OBJECTS_OBJECTTYPES_TOKEN", "", add_to_docs=False)
 # Demo User Configuration
-DEMO_CONFIG_ENABLE = config("DEMO_CONFIG_ENABLE", default=DEBUG)
-DEMO_TOKEN = config("DEMO_TOKEN", "")
-DEMO_PERSON = config("DEMO_PERSON", "")
-DEMO_EMAIL = config("DEMO_EMAIL", "")
+DEMO_CONFIG_ENABLE = config("DEMO_CONFIG_ENABLE", default=DEBUG, add_to_docs=False)
+DEMO_TOKEN = config("DEMO_TOKEN", "", add_to_docs=False)
+DEMO_PERSON = config("DEMO_PERSON", "", add_to_docs=False)
+DEMO_EMAIL = config("DEMO_EMAIL", "", add_to_docs=False)

src/objects/conf/dev.py

@@ -10,6 +10,7 @@
 os.environ.setdefault("IS_HTTPS", "no")
 os.environ.setdefault("RELEASE", "dev")
 os.environ.setdefault("ENVIRONMENT", "development")
+os.environ.setdefault("DISABLE_2FA", "True")
 
 os.environ.setdefault("DB_NAME", "objects"),
 os.environ.setdefault("DB_USER", "objects"),
@@ -104,10 +105,6 @@
 if "test" in sys.argv:
     NOTIFICATIONS_DISABLED = True
 
-# None of the authentication backends require two-factor authentication.
-if config("DISABLE_2FA", default=True):  # pragma: no cover
-    MAYKIN_2FA_ALLOW_MFA_BYPASS_BACKENDS = AUTHENTICATION_BACKENDS
-
 # Override settings with local settings.
 try:
     from .local import *  # noqa