A tool to load a lot of FHIR resources into a "naked" FHIR server.
It is intended to quickly load a package of FHIR Profiles (StructureDefinitions) and associated artefacts (such as CodeSystem, ValueSet, ConceptMap) into a FHIR server that has just been spun up.
This tool was developed in the context of the Core Dataset (KDS) of the Medical Informatics Initiative (MII) in Germany as well as the German Corona Consensus Dataset (GECCO) developed by the Network University Medicine.
The script is written in Python 3 and available on PyPI.
As the package is available on the Python package index, you can install it quickly into a Virtual Environment. First, you may need to create a folder for FHIR populator and a Virtual Environment (all commands are for Unix-based OS and may need tweaking on Windows):
mkdir fhir-populator
cd fhir-populator
python -m venv .venv
source .venv/bin/activateOn Windows without Windows Subsystem for Linux, you will need to change the last command to .venv\bin\activate.bat.
These commands will create a new directory, visit it, create the virtual environment, and activate it.
Next, load the package from PyPI:
python -m pip install fhir-populatorYou can now start it as a Python module:
python -m fhir_populator --helpand the help will be printed:
usage: fhir_populator [-h] --endpoint ENDPOINT [--authorization-header AUTHORIZATION_HEADER] [--log-file LOG_FILE]
[--get-dependencies] [--non-interactive] [--include-examples]
[--log-level {INFO,WARNING,DEBUG,ERROR}] [--rewrite-versions] [--only-put] [--versioned-ids]
[--exclude-resource-type [EXCLUDE_RESOURCE_TYPE ...] | --only [ONLY ...]]
[--registry-url REGISTRY_URL] [--package PACKAGES [PACKAGES ...]]
[--persist] [--from-persistence] [--persistence-dir PERSISTANCE_DIR]
optional arguments:
-h, --help show this help message and exit
--endpoint ENDPOINT The FHIR server REST endpoint (default: None)
--authorization-header AUTHORIZATION_HEADER
an authorization header to use for uploading. If none, nothing will be sent. (default: None)
--log-file LOG_FILE A log file path (default: None)
--get-dependencies if provided, dependencies will be retrieved from the FHIR registry. (default: False)
--non-interactive In case of errors returned by this FHIR server, the error will be ignored with only a log
message being written out. Might be helpful when integrating this module into a script.
(default: False)
--include-examples If provided, the resources in the 'examples' folder of the packages will be uploaded.
(default: False)
--log-level {INFO,WARNING,DEBUG,ERROR}
The level to log at (default: INFO)
--rewrite-versions If provided, all versions of FHIR resources will be modified to be consistent with the
package version. Otherwise, the version is used as-is! (default: False)
--only-put if provided, IDs will be generated for all resources that lack one. This can be combined with
--versioned-ids. (default: False)
--versioned-ids if provided, all resource IDs will be prefixed with the package version. (default: False)
--exclude-resource-type [EXCLUDE_RESOURCE_TYPE ...]
Specify resource types to ignore! (default: None)
--only [ONLY ...] Only upload the resource types provided here, e.g. only StructureDefinitions, CodeSystems and
ValueSets (default: None)
--registry-url REGISTRY_URL
The FHIR registry url, Simplifier by default (default: https://packages.simplifier.net)
--package PACKAGES [PACKAGES ...]
Specification for the package to download and push to the FHIR server. You can specify more
than one package. Use the syntax 'package@version', or leave out the version to use the
latest package available on the registry. (default: None)
--persist if provided the package will be persisted in the persist-dir
--persistence-dir directory where the persisted packages will be stored or loaded from
--from-persistence if provided the package will be loaded from the persistence-dir
There are a lot of command line options that can be used to customize the behaviour of the program.
To try out the program, you can spin up a FHIR server, such as HAPI FHIR JPA Server Starter on your local machine, e.g. using Docker. Assuming the endpoint of the server is http://localhost:8080/fhir, you can upload the latest version of the GECCO package, including dependencies (e.g. the MII KDS modules used by that package), thus:
python -m fhir_populator --endpoint http://localhost:8080/fhir --get-dependencies --package de.geccoAs this example does not specify a version of the de.gecco package, the latest version of the package will first be determined from the Simplifier API. You can also specify a version using the syntax package@version:
python -m fhir_populator --endpoint http://localhost:8080/fhir --get-dependencies --package de.gecco@1.0.3Also, you can specify as many packages as you like, and mix-and-match versioned references with unversioned ones:
python -m fhir_populator --endpoint http://localhost:8080/fhir --get-dependencies --package de.gecco@1.0.3 de.medizininformatikinitiative.kerndatensatz.personThe script is broken into multiple steps:
- All unversioned package references are converted to versioned references, by retrieving the package metadata from the NPM registry.
- The packages are downloaded as Tarballs into a temporary directory (under
/tmpfor Unix systems), and extracted there - After each package is downloaded, the
package.jsonis examined, and dependencies are added to the download queue, if desired. During this download, a dependency graph is built from the downloaded packages, to make sure that every package is uploaded after its dependencies - The packages are uploaded, file-by-file, to the FHIR server. This uses the topological sort of the directed dependency graph, to maintain consistency. Also, the files are uploaded in logical versions (e.g.
CodeSystembeforeValueSetbeforeStructureDefinitionbeforePatientetc.) - If the FHIR server returns an error, the user is prompted interactively for input.
- When all resources are uploaded (or if the user aborts execution with CTRL-C), the temporary directory is recursively deleted.
There are a number of configuration options, which are (hopefully) mostly self-explanatory. Some of the more obscure ones are explained below:
--authorization-header: USe if your server is configured for Authentication. You can enter something like `--authorization-header "Bearer asdf" here, which will be presented to the server for each request.--exclude-resource-type: You can skip resource types, e.g.--exclude-resource-type CodeSystem ValueSet ConceptMap. This is not case-sensitive, the lower-case version of the resource type will be matched against the lower-case parameter list.--include-examples: Examples in FHIR packages are great, but often not consistent across packages. For example, anObservationexample might referencePatient/example, and this patient is nowhere to be found in the package, or its dependencies. Some FHIR servers (such as HAPI JPA Server) validate references on CREATE and return errors for missing references. Hence, examples (files in theexamplesfolder of the package, as per the spec) are ignored by default.--non-interactive: If provided, errors returned by the FHIR server will be ignored, and only a warning will be printed out.--only-put: FHIR requires that IDs are present for all resources that are uploaded via HTTP PUT. Hence, if IDs are missing, an HTTP POST request is used by the script. This does not generate stable, or nice, IDs by default. You can provide this parameter to make the script generate IDs from the file name of the resource, which should be stable across reruns. This uses a "slugified" version of the filename without unsafe characters, and restricted to 64 characters, as per the specification.--registry-url: While the script was only tested using the Simplifier registry, it should be compatible to other implementations of the FHIR NPM Package Spec, which is implemented by the Simplifier software. You can provide the endpoint of an alternative registry hence.--rewrite-versions: If provided, allversionattributes of the resources will be rewritten to match the version in thepackage.json, to separate these definitions from previous versions. You will need to think about the versions numbers you use when communicating with others, who might not use the same versions -⚠️ use with caution!⚠️ --versioned-ids: To separate versions of the resources on the same FHIR server, you can override the IDs provided in the resources, by including the slugified version of the package in the ID. If combined with the--only-putswitch, this will work the same, versioning existing IDs, and slugifying + versioning the filename of resources without IDs.--persist: If provided, the downloaded packages will be persisted in the--persistence-dirdirectory.--persistence-dir: The directory where the persisted packages will be stored or loaded from.--from-persistence: If provided, the package will be loaded from the--persistence-dirdirectory.
--http-proxy: URL of your HTTP proxy, may optionally include credentials (c.f. the Requests documentation)--https-proxy: URL for HTTPS requests, if not provided, the HTTP proxy is used instead--proxy-for-fhir: If provided, the proxy is also used for requests to your FHIR server--proxy-verify: If provided, this public key (-chain) on your disk is used for validating the re-encrypted traffic to your proxy--proxy-for-fhir: If provided, the proxy is also used for FHIR requests, not only for NPM requests`
cd fhir-populator
source venv/bin/activate
python -m pip install --upgrade fhir-populatorIf you want to customize the program, you should:
- create a fork in GitHub, and clone it.
- create a new virtual environment in your fork:
python -m venv .venv;source .venv/bin/activate - Install the package locally, using
pip install . - Customize the script. Re-run step 3 if you change the script.
python -m fhir_populator, as before.- Create an issue and pull request in the GitHub Repo! We welcome contributions!
| Version | Date | Changes |
|---|---|---|
| v1.0.10 | 2021-06-03 | Initial release |
| v1.1.0 | 2021-06-08 | - handle Unicode filenames, especially on BSD/macOS (#1) - do not serialize null ID for POST (#2) - include option for only certain resource types(#6) - fix XML handling (#6) - add LICENSE |
| v1.1.1 | 2021-06-09 | - explicitly open files with UTF-8 encoding (#12) - ignore pycharm and vscode (#11) |
| v1.2.0 | 2022-12-13 | - support HTTP/HTTPS proxies |
| v1.3.0 | 2023-04-03 | - support using a persistence directory |