Reload configuration file

[JakobMiksch]

I want to add a new service to my configuration file, but I don't know how to make it work on the server.
Is there a way to update the running instance of pygeoapi without restarting it?
Or do I have to stop and start pygeoapi every time I make a change?

(I found this discussion that is related to my question: #1074)

[matthesrieke]

At the moment a restart is required. I have just proposed a feature request (#1351) that is a first step into the direction of dynamic resources.

How would your workflow look like? Would you want to update the config and reload it at runtime? Or would you want to manage the resources somewhere else (e.g. in a database)?

[JakobMiksch]

A typical scenario is that another program (let's call it the manager program) creates a PostGIS table and wants to directly publish it using pygeoapi. Possible options that I currently see are:

• add the layer to pygeoapi via an HTTP call (similar to GeoServer, that can be controlled by a REST API)
• the manager program edits the pygeoapi config file and reloads pygeoapi. There I see two options:
  • either pygeoapi has a "hot-reload" functionality that automatically detects a change in the config file and reloads it (as long the change is valid)
  • or there is a reload functionality that can be triggered externally by an HTTP call

these were just some ideas.

[justb4]

As a suggestion for lightweight reload on changes in (multiple files): for a similar scenario in MapProxy Mapnik Source I needed a hot-reload of the Mapnik-config, which is spread over multiple files, for the Mapnik plugin (not MapProxy itself) as to speed-up development edit-render cycle. I used the Python package Watchdog with FileSystemEventHandler. This package works well, even over multiple MP Processes with each Mapnik instance itself already having multiprocessing/threading. Still in process PR-ing to MapProxy as it depends on another MP PR. See details here.

Complex issue IMO, pygeoapi hot-reloading: it may run in multiple server-tech contexts with long-running requests in progress plus the OAS/Swagger file is currently being generated offline using the whole of pygeoapi config. Some kind of life-cycle management (LCM) is needed. A solution direction is to have finer-grained LCM at the plugin-level such that only affected plugins (Providers) are (re)started. Using files in this context may be brittle as well. In a Kubernetes environment these kind of scenarios (restart with new config) are already supported without downtime. Testing is a challenge as well..

[JakobMiksch]

maybe other software outside of the geo-world can give inspiration as well. For example Apache HTTP Server or NGINX also have their solutions how to handle config change while they are running

[matthesrieke]

@justb4 great insight, thanks!

I wrote a proof-of-concept that allows adding resources at runtime and also re-creates the OpenAPI file after the change is established (very dirty, will not show it ;-) ). So I think it will work in general. Again, see my FR that aims at creating the first step into that direction: #1351