Environments

Environments

Established Environments

Established environments support BFD's path-to-production and include test, prod-sbx, and prod.

test

PHI/PII	Reliability	Access
No	Non-Production	Internal

Historically, the test environment has been used as a common integration testing environment. The process for deploying changes to test supports on-demand deployments of individual contributors' feature branches and pull requests. This environment is also used in determining production-readiness of a given change set. Generally, software changes that are based off the trunk or master branch of the beneficiary-fhir-data repository can be promoted to a production deployment after the changes yield a successful deployment in test.

As of mid 2022, the test environment was expanded to include additional resources to be more production-like, including both resource scaling-out and scaling-up to the RDS cluster and BFD FHIR API Server AutoScaling Group. Along with more production-like resources, BFD has also loaded an additional 10,000 synthetic beneficiaries and associated claims data to facilitate performance and load testing. As a result, test can also be used as a performance testing environment.

prod-sbx

PHI/PII	Reliability	Access
No	Production	External

prod

PHI/PII	Reliability	Access
Yes	Production	Internal

Ephemeral Environments

Constraints

Ephemeral environments leverage one of the established environments as the baseline to evaluate updates. Ephemeral environments are created via terraform workspaces. For instance, if Jira ticket 1001 needed to evaluate the impact of a core infrastructure change for BFD-Server, starting from prod-sbx, the process to create a corresponding ephemeral environment would be the following, if the current working branch was BFD-1001:

cd ops/terraform/services/base
terraform workspace new 1001-prod-sbx
terraform workspace select 1001-prod-sbx
terraform plan && terraform apply
cd ../common
terraform workspace new 1001-prod-sbx
terraform workspace select 1001-prod-sbx
terraform plan && terraform apply
cd ../server
terraform workspace new 1001-prod-sbx
terraform workspace select 1001-prod-sbx
terraform plan && terraform apply

For destroying the same Ephemeral environment, the abbreviated process would be executed in reverse:

cd ops/terraform/services/server
terraform workspace select 1001-prod-sbx
terraform destroy
terraform workspace select test
terraform workspace delete 1001-prod-sbx
cd ../common
terraform workspace select 1001-prod-sbx
terraform destroy
terraform workspace select test
terraform workspace delete 1001-prod-sbx
cd ../base
terraform workspace select 1001-prod-sbx
terraform destroy
terraform workspace select test
terraform workspace delete 1001-prod-sbx

Exceptional Cleanup

If fundamental IaC changes are implemented beneath longer-lived ephemeral environments, it is inevitable that latest changes will introduce incompatibilities and prevent terraform's three-way diff to function correctly.

For any problem in an ephemeral terraservice where starting over from the latest, good configuration is the best approach:

1. Set working directory inside of the terraservice directory of interest, with the desired workspace selected
2. git checkout master (or similar, e.g. rebase, merge, etc) to use the latest changes from the trunk
3. rm *.tf to prepare a signal to terraform that you're no longer managing any resources stored in the remote state
4. git checkout terraform.tf to restore the provider configurations, etc.
5. delete default_tags values from any/all provider blocks in the terraform.tf file
6. terraform apply to destroy all resources tracked by the remote state; unlike executing terraform destroy with the IaC present, this method allows one to avoid some of checks that can fail under certain types of changes during the three-way reconciliation between the desired configuration (IaC), the remote state, and the infrastructure.
7. git restore . to restore the IaC for the current terraservice
8. terraform apply to apply the latest changes from IaC