diff --git a/cloud/README.md b/cloud/README.md index 2ef79a72..912d4105 100644 --- a/cloud/README.md +++ b/cloud/README.md @@ -18,43 +18,73 @@ This should be much easier, as it is natively supported by API Gateway, but it s ## Commands +The pipeline only needs to be synthesized and deployed once, after which it will self-update whenever triggered. It is +currently configured to run on merging to `main` branch, so adjust that if you want to trigger on a different branch. + `npm run cdk:synth` -Generates the CloudFormation templates for a build pipeline - dev stage. +Generates the CloudFormation templates for a "dev stage" build pipeline. `npm run cdk:synth:prod` -Generates the CloudFormation templates for a build pipeline - production stage. +Generates the CloudFormation templates for a "production stage" build pipeline. `npm run cdk:synth -- --context STAGE={STAGENAME}` -Generates the CloudFormation templates for a build pipeline - stage given by `{STAGENAME}`. +Generates the CloudFormation templates for a build pipeline, stage given by `{STAGENAME}`. `npm run cdk:deploy:all` Deploys the synthesized pipeline to an AWS Environment, as defined by your active AWS config profile. -Note that the pipeline only needs to be deployed once, after which it self-updates whenever it is triggered. It is -currently configured to run on merging to `main` branch, so adapt to your needs. - `npm run cdk:destroy:all` Destroys the deployed pipeline in your remote AWS Environment. -Destroying the pipeline stack does not destroy the application stacks, so those will need to be deleted manually in the -AWS Console. +Note that destroying the pipeline stack does not destroy the application stacks deployed by the pipeline, so those would +need to be deleted manually in the AWS Console. + +## Testing stack changes + +As the pipeline deploys the application stacks, it is wise to test any changes to those stacks before committing them. +You can do this by synthesizing just the application stacks locally, and deploying to AWS as `dev` stage. +There is one small task to complete before you begin. In AWS Secrets Manager, you will find a secret storing API key and +secret values for the `prod` stage, which the server needs for successful startup. You must create a new secret for the +dev stage, with the same OPENAI_API_KEY value and any random string for SESSION_SECRET. Once that is in place, you can +synthesize and deploy the stacks for testing. Once you have finished, please delete the secret to avoid unnecessary +costs. -Additionally, +`npm run cdk:test:synth` - synthesizes just the application stacks (i.e. not the pipeline) + +`npm run cdk:test:deploy` - deploys these stacks to AWS as "dev" stage + +All being successful, you should see the application login screen at `https://dev.spylogic.ai`. Log into the AWS Console to add a +user to the dev Cognito userpool, then log into the UI to test app deployment was successful. + +`npm run cdk:test:destroy` - Remember to destroy the stacks after testing, else you will rack up costs! --- +## A note on costs + +At the time of writing, current infrastructure costs us around $60 per month, with just two AZs for the load balancer, +deployed into `eu-north-1`. This is one of the [greenest AWS regions](https://app.electricitymaps.com/map), but costs +are about average. The vast majority of the bill is for the VPC, Load Balancer and NAT EC2 Instance. We have tasks on +our todo list to reduce these costs (such as removing the NAT Instance in favour of IPv6 egress), but those are +work-in-progress. + +The bottom line: remember to destroy your stacks when no longer needed! + ## First-time admin tasks -If you are setting up the CDK project for the first time, there are a few setup tasks you must complete. +When setting up the CDK project for the first time, there are a few one-time tasks you must complete. ### Bootstrapping the CDK using a Developer Policy In order to deploy AWS resources to a remote environment using CDK, you must first [bootstrap the CDK](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html). For this project, as per -[CDK guidelines](https://aws.amazon.com/blogs/devops/secure-cdk-deployments-with-iam-permission-boundaries/), we use a -lightweight permissions boundary to restrict permissions, to prevent creation of new users or roles with elevated -permissions. See `cdk-developer-policy.yaml` for details. +[CDK guidelines](https://aws.amazon.com/blogs/devops/secure-cdk-deployments-with-iam-permission-boundaries/), we are +using a lightweight permissions boundary to restrict permissions, to prevent creation of new users or roles with +elevated permissions. See `cdk-developer-policy.yaml` for details. + +Note that once the pipeline is deployed, it is a good idea to restrict permissions further, so that only the pipeline +can make changes to the stacks. Create the permissions boundary CloudFormation stack: @@ -75,9 +105,20 @@ npm install npx cdk bootstrap --custom-permissions-boundary cdk-developer-policy ``` -Unless your default region is `us-east-1`, you will also need to bootstrap that region, as certificates for CloudFront -currently need to be deployed into that region: +Unless your default region is `us-east-1`, you will also need to bootstrap this region, as certificates for CloudFront +currently need to be deployed there: ``` npx cdk bootstrap --custom-permissions-boundary cdk-developer-policy aws://YOUR_ACCOUNT_NUMBER/us-east-1 ``` + +### Server secrets + +The Node Express server needs a couple of secret values, which are injected into the container environment during +deployment via AWS Secrets Manager. + +You will need to create a secret in your chosen region named `{stagename}/SpyLogic/ApiKey`, where stagename matches the +name used during Synth (for example, "dev" or "prod"). Within this secret, you will need values for OPENAI_API_KEY and +SESSION_SECRET. + +Refer to the [main README](../README.md) for further details on these. diff --git a/cloud/bin/application.ts b/cloud/bin/application.ts new file mode 100644 index 00000000..91b10bff --- /dev/null +++ b/cloud/bin/application.ts @@ -0,0 +1,89 @@ +#!/usr/bin/env node +import { App, Environment } from 'aws-cdk-lib/core'; +import 'source-map-support/register'; + +import { + appName, + resourceDescription, + resourceId, + stackName, + stageName, + ApiStack, + AuthStack, + CertificateStack, + HostedZoneStack, + UiStack, +} from '../lib'; + +/* +This application can be used to test stack changes before they are deployed by +the production pipeline. See the cloud README for details. +*/ + +const app = new App(); +const generateStackName = stackName(app); +const generateDescription = resourceDescription(app); + +/* Common stack resources */ + +// NOTE: Need DOMAIN_NAME and HOSTED_ZONE_ID env vars, see "cloud/.env.example" +// If you have access to SpyLogic AWS account, these are viewable in Parameter Store. +const env: Environment = { + account: process.env.CDK_DEFAULT_ACCOUNT, + region: process.env.CDK_DEFAULT_REGION, +}; + +const tags = { + owner: appName, + stage: stageName(app), +}; + +/* Stack constructs */ + +const hostedZoneStack = new HostedZoneStack(app, generateStackName('hostedzone'), { + description: generateDescription('Hosted Zone stack'), + env, + tags, +}); + +const certificateStack = new CertificateStack(app, generateStackName('certificate'), { + description: generateDescription('Certificate stack'), + env, + tags, + domainName: hostedZoneStack.topLevelDomain.value as string, + hostedZone: hostedZoneStack.hostedZone, +}); + +const authStack = new AuthStack(app, generateStackName('auth'), { + description: generateDescription('Auth stack'), + env, + tags, + domainName: hostedZoneStack.topLevelDomain.value as string, +}); + +new ApiStack(app, generateStackName('api'), { + description: generateDescription('API stack'), + env, + tags, + apiDomainName: certificateStack.apiDomainName, + certificate: certificateStack.loadBalancerCert, + customAuthHeaderName: authStack.customAuthHeaderName, + customAuthHeaderValue: authStack.customAuthHeaderValue, + domainName: hostedZoneStack.topLevelDomain.value as string, + hostedZone: hostedZoneStack.hostedZone, +}); + +new UiStack(app, generateStackName('ui'), { + description: generateDescription('UI stack'), + env, + tags, + apiDomainName: certificateStack.apiDomainName, + certificate: certificateStack.cloudFrontCert, + customAuthHeaderName: authStack.customAuthHeaderName, + customAuthHeaderValue: authStack.customAuthHeaderValue, + domainName: hostedZoneStack.topLevelDomain.value as string, + hostedZone: hostedZoneStack.hostedZone, + hostBucketName: resourceId(app)('host-bucket'), + parameterNameUserPoolClient: authStack.parameterNameUserPoolClient, + parameterNameUserPoolId: authStack.parameterNameUserPoolId, +}); diff --git a/cloud/bin/cloud.ts b/cloud/bin/pipeline.ts similarity index 100% rename from cloud/bin/cloud.ts rename to cloud/bin/pipeline.ts diff --git a/cloud/cdk.json b/cloud/cdk.json index a5c0ece0..ffea123a 100644 --- a/cloud/cdk.json +++ b/cloud/cdk.json @@ -1,5 +1,5 @@ { - "app": "npx ts-node --prefer-ts-exts -r dotenv/config bin/cloud.ts", + "app": "npx ts-node --prefer-ts-exts -r dotenv/config bin/pipeline.ts", "watch": { "include": ["**"], "exclude": [ diff --git a/cloud/lib/index.ts b/cloud/lib/index.ts index af3098cd..181acff3 100644 --- a/cloud/lib/index.ts +++ b/cloud/lib/index.ts @@ -1,3 +1,10 @@ export * from './resourceNamingUtils'; + +export { ApiStack } from './api-stack'; +export { AuthStack } from './auth-stack'; +export { CertificateStack } from './certificate-stack'; +export { HostedZoneStack } from './hostedzone-stack'; +export { UiStack } from './ui-stack'; + export { PipelineAssistUsEast1Stack } from './pipeline-assist-useast1-stack'; export { PipelineStack } from './pipeline-stack'; diff --git a/cloud/package.json b/cloud/package.json index e498732e..cf64a73d 100644 --- a/cloud/package.json +++ b/cloud/package.json @@ -2,7 +2,7 @@ "name": "cloud", "version": "1.0.3", "bin": { - "cloud": "bin/cloud.js" + "cloud": "bin/pipeline.js" }, "scripts": { "cdk:synth": "cdk synth -q \"*\"", @@ -13,6 +13,10 @@ "cdk:destroy": "cdk destroy --app cdk.out", "cdk:destroy:all": "cdk destroy --app cdk.out --all", "cdk:clean": "rimraf cdk.out", + "cdk:test:synth": "cdk synth -o cdk.test.out -a \"npx ts-node --prefer-ts-exts -r dotenv/config bin/application.ts\"", + "cdk:test:deploy": "cdk deploy --app cdk.test.out --all", + "cdk:test:destroy": "cdk destroy --app cdk.test.out --all", + "cdk:test:clean": "rimraf cdk.test.out", "codecheck": "concurrently \"npm run lint:check\" \"npm run format:check\"", "format": "prettier . --write", "format:check": "prettier . --check",