pb_deploy is a Terraform configuration for a
production deployment of PolicyBrain, as run by the
Open Source Policy Center, on Amazon Web Services infrastructure.
Terraform is a declarative "infrastructure as code" open-source application, meaning that it can deploy a new set of infrastructure components with ease given account information, and it can keep it up-to-date with a programmatically specified configuration, performing changes only as needed.
This is possible because Terraform keeps track of the state. In the case of OSPC, we use AWS S3 and DynamoDB to share and lock state information among different OSPC collaborators.
The main component of this infrastructure is Amazon ECS Fargate, which is a
technology for deploying Docker containers without having to manage the
underlying servers. ECS is used to run relatively few Internet-facing Flask
servers, as well as relatively many Celery workers. Rather than being run on
ECS, Redis is managed through Amazon ElastiCache, and stores the task queue.
Amazon Route 53 and Amazon ELB are used to provide a permanent hostname,
such as staging.ospcapi.org, that will always point to the correct ECS task
IP(s). (Currently, the alternative, Route53 Service Discovery for ECS, only
supports private IPs.)
The networking is handled using a Virtual Public Cloud and a single public subnet, and all instances are given public IPs. This is to avoid the expense of a NAT gateway required with private subnets, but AWS security groups are used to define permissible Internet access as restrictively as possible.
Because of Terraform's declarative nature, the steps for initial setup and for
modification of the infrastructure should be the same. You should specify
AWS authentication credentials either through
environment variables or through the ~/.aws/credentials file. The
configuration also assumes that the following resources have already been
created in the region given in the terraform.tfvars file:
- An Amazon ECS task execution role
- An Amazon Route 53 zone with the name given in
terraform.tfvars; this can be any domain or subdomain for which you need to configure the nameservers - An S3 bucket and a DynamoDB table with the names given in
main.tfin order to store Terraform state
This configuration makes extensive use of Terraform workspaces in order to
separate state and resources for different deployment environments. The
default workspace can be used if this distinction is not needed, but should
not be used for OSPC deployments, which will make use of the production
and staging environments.
An example deployment workflow would be:
cd ~/pb_deploy
terraform init
terraform workspace select production
terraform plan -out=production.tfplan
terraform apply production.tfplanA deployment environment could be torn down as follows:
cd ~/pb_deploy
terraform init
terraform workspace select staging
terraform destroy