Skip to content

Dev: Duffy Components

Jump to bottom
Nils Philippsen edited this page Jun 23, 2022 · 1 revision

Duffy contains several “runnable“ components with differing sets of dependencies, which are encoded as package extras of the duffy Python package.

Core

The Duffy core contains the code to run the CLI program duffy and to read and validate YAML-encoded configuration files.

Duffy packages and modules
  • duffy.api_models: Pydantic models specifying API valid requests and responses
  • duffy.cli: The duffy CLI program
  • duffy.configuration: Code to read and validate YAML configuration files
  • duffy.exceptions: Custom exception types
  • duffy.misc: Custom Pydantic types used in configuration and the API
  • duffy.util: Miscellaneous helper code
  • duffy.version: Extract the Duffy version from Python package metadata
Dependencies
  • Python 3.8 or later
  • PyYAML: To read YAML configuration files
  • click: To define the CLI commands and options/arguments
  • pydantic: To specify the configuration file and API structures

API Web App

The main purpose of the Duffy API web app is to manage node sessions. Tenants request what types and numbers of nodes they need in a session, the web app looks up in the database if the request can be served – i.e. if enough nodes of the requested type(s) are available – and if so, allocate the nodes into the session, installs the tenant’s SSH key onto the nodes and return information about the nodes to the tenant so they can use them. When tenants are done using the nodes, they retire the session, and the web app tells the backend task workers to deprovision the nodes.

Duffy packages and modules
  • duffy.app: The web app
  • duffy.database: The SQLAlchemy database model
  • duffy.nodes.context: To (de)contextualize nodes, i.e. install tenants’ SSH keys onto them and remove them again
  • duffy.nodes.pools: To provide information about configured node pools
  • duffy.tasks: The backend tasks called from the web app
Dependencies
  • SQLAlchemy: To actually talk to the database
  • alembic: To migrate database schemas and contents
  • bcrypt: To store only hashes of API keys and validate them
  • fastapi: High performance asynchronous web framework
  • uvicorn: ASGI web server
  • celery: To communicate with backend task workers
  • aiodns *: To asynchronously look up host names from DNS
  • ansible-runner *: To run Ansible playbooks
  • Jinja2 *: To expand templates in configuration
  • jmespath *: To enable the json_query filter in Jinja templates
  • pottery *: To lock critical sections of backend tasks

*: This dependency is pulled in by importing duffy.tasks which is needed to communicate with backend task workers.

Legacy API Metaclient Web App

This web app sits between legacy API clients and the current API, serving the former and acting as a client to the latter. Its purpose is to translate back and forth between the legacy and current protocols.

Duffy packages and modules
  • duffy.legacy: Pydantic models specifying API valid requests and responses
Dependencies
  • httpx: To asynchronously communicate with the current API over HTTP
  • Jinja2: To enable Jinja macros and filters in pool mapping configuration

Task Workers

The Celery task workers perform background work that a) potentially takes longer (like provisioning and deprovisioning nodes) and/or b) needs to be done on a regular basis (like expiring sessions past their configured lifetime).

Duffy packages and modules
  • duffy.tasks: The backend tasks called from the web app
  • duffy.nodes.pools, duffy.nodes.mechanisms: To maintain configured fill levels of nodes pools by provisioning new nodes into them when already provisioned ones are put into use
Dependencies
  • celery: To define remotely callable tasks
  • aiodns: To asynchronously look up host names from DNS
  • ansible-runner: To run Ansible playbooks
  • Jinja2: To expand templates in configuration
  • jmespath: To enable the json_query filter in Jinja templates
  • pottery: To lock critical sections of backend tasks

Client

The duffy client CLI application lets users run commonly used functions of the API from the command line. It can format output as JSON, YAML or “flat“, which lends itself well for processing from shell scripts.

Duffy packages and modules
  • duffy.client: The DuffyClient class wrapping access to the API
  • duffy.client.formatters: Classes implementing different result formatting options
Dependencies
  • httpx: To communicate with the Duffy API over HTTP
Clone this wiki locally