AI runtime configuration scheme

[nikos-livathinos]

Objective

• We want to introduce input parameters that control the AI runtime configuration which is used to run the Docling AI models.
• The AI runtime parameters should include the "device" and "number of threads".
• The user should be able to set these parameters either as envvars or as API/CLI parameters.

Introduce the logic to do the resolution of input parameters and decide the AI runtime configuration based on:

• The values of envvars (if set).
• The values of API/CLI parameters (if set).
• The default values of each parameter.
• The availability of AI execution devices on the system.
• The input parameters required by the models.

Proposal

A. Introduce input parameters in docling package:

• API: Extend the PipelineOptions to introduce the fields: device, num_threads.
  • The device parameter can be an Enum with values: [AUTO, CUDA, CPU, MPS]. The default value is AUTO.
  • The num_threads parameter is an integer. The default value is 4.
• Envvars: Standardize names of envvars that set the device and number of threads.
  • For example [DOCLING_DEVICE, DOCLING_NUM_THREADS] or [DOCLING_DEVICE, OMP_NUM_THREADS].
• CLI: Introduce corresponding CLI parameters device and num_threads.

B. Introduce configuration resolution logic as a utility function in docling-ibm-models package:

1. Define device precedence: [CUDA, MPS, CPU].
2. Use the user-provided API/CLI parameters, if they have been explicitly set by the user.
3. Use the envvars, if they are set.
4. Use the default values.
   5a. If device == AUTO, use the first available device according to the precedence.
   5b. If device is explicitly set by the user but it is not available in the system, replace it with the next available device.

C. Usage:

1. The user either relies on the default values or explicitly sets the input parameters.
   • The API parameters are set individually for each pipeline.
   • In case of the docling CLI, the parameters apply for all pipelines.
2. Each model receives the input runtime parameters as part of the pipeline_options.
3. The utility function from docling-ibm-models is used to resolve the runtime configuration.
   • This step can be done either by the "model predictors" inside the docling-ibm-models package or by the "models" inside the docling package.
   • The resolved runtime configuration can be further customized to feed the input parameters of each AI model.

[dolfim-ibm]

In terms of resolution priority, we should do (from higher priority to lower):

1. device set in the pipeline options (or directly in the models init)
2. CLI argument
3. ENV variable
4. Code defaults

[vagenas]

To streamline this config mgmt part, let's try to leverage Pydantic Settings as much as possible, as it already covers many of the pain points.

[vagenas]

FYI we can more precisely get inspired by my implementation of profiles in deepsearch-toolkit, which is essentially a utility for persisted, swappable configurations using Pydantic Settings and platformdirs.