This repository is a monorepo for managing the configuration and deployment of services managed by MIT Open Learning engineering. It uses a combination of Pulumi and PyInfra to build a pure-python deployment stack to enable more developer-friendly access to creating and modifying the systems that power the applications that we build and serve.
All infrastructure provisioning performed via Pulumi is located under src/ol-infrastructure/ and all configuration management written in PyInfra lives under src/bilder.
This is a Pulumi project, so the first step is to install the Pulumi CLI along with the relevant language protocol. Instructions here.
To install and manage the relevant dependencies this project uses Poetry. After installing
the Poetry CLI simply run poetry install.
We use the S3 state backend for Pulumi, so after installing the Pulumi CLI run pulumi login s3://mitol-pulumi-state.
This is a monorepo of the Pulumi code that we use to manage the infrastructure that powers MIT Open Learning
Component resources are located under src/ol-infrastructure/components/ with a descriptive name that makes it evident what the intent of the
component is.
Concrete implementations of infrastructure are located under src/ol-infrastructure/infrastructure/ with a descriptive name that makes it
evident what is contained in that project.
Management of resources that rely on foundational infrastructure being provisioned, but
which supports the operation of applications is located under
src/ol-infrastructure/substructure. An Example of this includes Consul prepared queries.
Provisioning of all of the resources needed to support and deploy a specific application
is located under src/ol-infrastructure/applications/ with a name that indicates the
application being managed (e.g. concourse).
Each component or concrete infrastructure that is more complex than a single resource will include a diagram.py file
that uses the diagrams package to illustrate the system structure that it creates.
Pulumi organizes code into Projects which represent a deployable unit. Within a project they have a concept of
Stacks which are often used as a mapping for different environments. Each module underneath src/ol-infrastructure/infrastructure/ and
src/ol-infrastructure/applications/ is its own Project, meaning that it will have a Pulumi.yaml definition of that project. Each stack
has its own yaml file in which the configuration for that stack is defined.
Stack names should be a dot-separated namespaced representation of the project path, suffixed with an environment
specifier in the form of QA or Production. The capitalization is important as it will be used directly to interpolate
into tag objects. It is easier to start with QA and Production and then call .lower() than it is to build a dictionary
mapping the lowercase versions to their properly capitalized representation.
The dotted namespace allows for peaceful coexistence of multiple projects within a single state backend, as well as allowing for use of stack references between projects.
The infrastructure components should be properly namespaced to match the stack names. For example, the project for
managing VPC networking in AWS is located at src/ol-infrastructure/infrastructure/aws/network/ and the corresponding stacks are defined as
aws.network.QA and aws.network.Production.
In order to run a deployment, you need to specify the project and the stack that you would like to deploy. From the root
of the repository, you can run pulumi -C src/ol_infrastructure/path/to/module/ up. If you haven't already selected the
stack, it will ask you to interactively select the stack which you are deploying.
For each deployable unit of work we need to have a Pulumi project defined. The Pulumi CLI has a new command, but that
introduces extra files that we don't want. The minimum necessary work to signal that a given directory is a project is
the presence of a Pulumi.yaml file and a __main__.py where the deployment code is located. The contents of the
Pulumi.yaml file should follow this structure:
name: ol-infrastructure-dagster-application # Change the name here to be descriptive of the purpose of this project
runtime: python # not necessary to change
description: Pulumi project for deploying the stack of services needed by the Dagster ETL framework # update the description accordingly
backend:
url: s3://mitol-pulumi-state/ # should not be changed
Create the directory path according to the conventions detailed above and then create these files. Once that is done you
will need to create the stack definitions (again according to the above conventions). To do this, cd to the target
directory and run the command pulumi stack init --secrets-provider=awskms://alias/infrastructure-secrets-qa <your.dotted.stack.name.QA>
Now you're ready to start writing the code that will define the target deployment.
In order to streamline the management of infrastructure required to support the services run by MIT Open Learning we have implemented our configuration management logic using PyInfra. This has the benefit of being pure Python, allowing us to take advantage of the broad ecosystem that it provides including linting, testing, and integrations.
Any given service is unlikely to be composed of a single service, instead requiring a
variety of components to be used together. In order to make the composition of a running
system easier to reason about, we decompose the responsibilities for any given piece of
technology into a self-contained component. These components live under the
src/bilder/components/ directory. Each component provide at minimum the logic needed to
install and configure the associated service. In addition, each component should also
provide the logic needed to upgrade an instance of the service, the SystemD unit files
needed to register the service, and any logic required to perform maintenance on the
associated technology.
A common requirement across all configuration management systems is a method to provide settings at build/runtime to populate config files, determine which logical paths to take, etc. A problem that often comes from managing these settings is uncertainty around the constraints for each value, and the ability to easily merge default values with user supplied information. In order to reduce the complexity of this situation and provide useful constraints we use Pydantic models with relevant validators to contain the values needed by each component.
Each model defined for a given component will inherit from
ol_configuration_management.lib.model_helpers.OLBaseSettings. This class inherits from
the Pydantic BaseSettings class and sets the case_sensitive model setting to
False. This allows us to easily use environment variables to override attributes of a
model at build time.
When faced with a decision of where to locate any given files or directories we default
to following the Filesystem Hierarchy
Standard. This typically
means that applications or services which are deployed by downloading an archive or
cloning a software repository should go under /opt, configuration files under /etc/,
and data files under /var/lib.