Skip to content

Commit

Permalink
Merge branch 'main' into wml-consolodate-contract-types
Browse files Browse the repository at this point in the history
  • Loading branch information
macrael committed Aug 14, 2023
2 parents dde765b + 1ecfb3e commit ea49a2d
Show file tree
Hide file tree
Showing 137 changed files with 502 additions and 488 deletions.
50 changes: 35 additions & 15 deletions docs/technical-design/howto-migrations.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,69 @@
# How to complete migrations

## Background

Migrations are a controlled way to change existing records in the database or else change the shape of tables/relationships. They are necessary when fields are added, removed, or when the meaning behind fields and their relationships change.

## General Guidance
- Prioritize testing in environments that have data similar to production. If you are testing on a review app, you should pre-populate data through using the deployed app as a user or via Cypress runs to have a diversity of test data available.
- Run in all deployed environments and validate for each environment.
- Write migrations that can be tested repeatedly. Include logic to skip or handle data that has already been changed or doesn't need to be migrated, so repeated runs of the lambda don't result in bad data.
- Avoid making migrations directly dependent on a specific feature being available in production.

- Prioritize testing in environments that have data similar to production. If you are testing on a review app, you should pre-populate data through using the deployed app as a user or via Cypress runs to have a diversity of test data available.
- Run in all deployed environments and validate for each environment.
- Write migrations that can be tested repeatedly. Include logic to skip or handle data that has already been changed or doesn't need to be migrated, so repeated runs of the lambda don't result in bad data.
- Avoid making migrations directly dependent on a specific feature being available in production.

## Data Migrations
Currently data migrations change existing records in the database. They are run as standalone lambdas that developers must manually trigger by environment.

### Steps
Currently data migrations change existing records in the database. They are run as standalone lambdas that developers must manually trigger by environment. They can be prototyped for fast feedback in the local environment, and then tested more thoroughly in lower environments.

### Local prototyping

1. Log into the AWS console (you'll need to be connected to the CMS VPN).
1. Go to the Dev environment dropdown: Cloud Access Roles --> Developer Admin --> Short-term Access Keys.
1. Choose option 1, and copy those values to the clipboard.
1. Paste the values into your .envrc.local file. Be sure that you're overwriting any previous values, and don't have duplicates.
1. If the app is running anywhere, stop it.
1. Run `DIRENV ALLOW` in the terminal in the directory where you'll start the app.
1. Start the app.
1. When the app is running, connect to the local database so that you can inspect the changes you're making. Use the DATABASE_URL from your .envrc.local file to connect to the database interface of your choice. You can connect via the terminal, or an app like TablePlus, dBeaver, Postico, DataGrip, etc.
1. Run some Cypress tests to populate the database. The CMSWorkflow tests are a good choice, since they create two entries which contain rates, one of which has a revision.
1. Verify that in the local database you can see entries in the HealthPlanPackageTable and HealthPlanRevisionTable.
1. Now you're ready to run your lambda. In a different terminal instance, run `DIRENV ALLOW` again.
1. Navigate to `services/app-api`.
1. Run `npx serverless invoke local --function `name_of_your_lambda``. You should see it build with webpack, then run. Any log statements you've put in your code will display in this window.
1. You can inspect the database again and should see any changes or additions you've made.

### Running in a lower environment

1. Prepare for manual testing in lower environments.
- Build a PR review app off `main` to start out. Try to populate with submissions similar to what is in production (For example, populate submissions via Cypress and if there are specific submissions types you know you will need to test, make sure add them).
1. Write the migration with verbose logs.
- The migration will be written as a [Node lambda](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html) and live in `app-api/src/handlers`. Make sure you use ES6 async await.
- The migration will be written as a [Node lambda](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html) and live in `app-api/src/handlers`. Make sure you use ES6 async await.
- Console statements are essential because there are line numbers in Cloudwatch when the migration fails or does not apply correctly. There is little additional context about why a run execution paused or failed.
- Here is an example of verbose debug console statements wrapping a loop:
![debug consoles in data migration script](../../.images/verbose-logs-example.png)
![debug consoles in data migration script](../../.images/verbose-logs-example.png)
1. Write unit tests.
- Consider including a test for `migration can be run repeatedly without data loss or unexpected results". Using repeatable migrations (when possible) makes it easier on developers and easier to unwind work.
1. Manually test the migration in your review app.
- Log into [AWS Lambda Console](https://console.aws.amazon.com/lambda/home) and find the lambda. This means choosing `app-api-` lambda with your branch name and migration name included.
- Click the Test tab and the `Test` button. Use the generic hello world event.
- All output will appear inline on the same page as the lambda run. All consoles appear there as well. You can also click into a link from there into Cloudwatch to be able to see the entire output of logs around the lambda execution.
- If you need to debug quickly, without waiting for redeploy in review app use the severless CLI script.
- `serverless deploy function --function name_of_migration --stage my_branch && echo "DEPLOYED" && serverless logs --function name_of_migration --stage my_branch --tail` can be run in your `app-api` directory. This script deploys your local handler code directly to AWS, without going through our normal CI build process. `my_branch`is the name of your deployed Github branch and `name_of_migration` is the name of the migration file mind the file path.
- If you need to debug quickly, without waiting for redeploy in review app use the severless CLI script.
- `serverless deploy function --function name_of_migration --stage my_branch && echo "DEPLOYED" && serverless logs --function name_of_migration --stage my_branch --tail` can be run in your `app-api` directory. This script deploys your local handler code directly to AWS, without going through our normal CI build process. `my_branch`is the name of your deployed Github branch and `name_of_migration` is the name of the migration file mind the file path.
- If this is to be run in DEV (`main`) you will have to reshape your `envrc.local` to imitate configuration on that stage. See AWS console for Lambda to reference the shape of config.
![lambda configuration panel in aws](../../.images/aws-console-lambda-config.png)
![lambda configuration panel in aws](../../.images/aws-console-lambda-config.png)
- Command line serverless scripts should not be used on PROD. Ideally do not use in VAL either. Running a migration via the AWS lambda console web interface is preferred because it is guaranteed to build off merged code and use similar configuration to how our application is deployed in CI.
1. Define clear acceptance criteria, this will be re-used.
- How do you know migration applied? What will be checked either in the app, via api request, or in the reports CSV?
- Developers will have to verify that the change worked multiple times, basically each time the migration is run on a new environment since the data could be quite different.
- If verification involves looking into reports and comparing fields, consider using a CSV tool like [CSVKit](https://csvkit.readthedocs.io/en/latest/index.html) if the comparisons seem involved.
1. After the PR merges and promotes,run in higher environments.
- Start with DEV. Follow similar steps Step #4 but now using the `main` lambda for your migration.
- Run the migration DEV > VAL > PROD in order, verifying in the application (or via reporting output) after each run.

- Run the migration DEV > VAL > PROD in order, verifying in the application (or via reporting output) after each run.

## Schema Migrations

We use Prisma to perform schema migrations. Right now the developer process for schema migrations is described a bit [here](../../README.md#updating-the-database). More detailed steps forthcoming.

## Proto migrations
The [proto migrations approach](../../services/app-proto/README.md#adding-a-new-migration) is no longer recommended - we are moving off protos.


The [proto migrations approach](../../services/app-proto/README.md#adding-a-new-migration) is no longer recommended - we are moving off protos.
2 changes: 2 additions & 0 deletions services/app-api/.eslintrc
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
"jest/no-identical-title": "error",
"@typescript-eslint/no-floating-promises": "error",
"no-unused-vars": "off",
"@typescript-eslint/consistent-type-exports": "error",
"@typescript-eslint/consistent-type-imports": "error",
"@typescript-eslint/no-unused-vars": [
"warn",
{ "argsIgnorePattern": "^_*" }
Expand Down
6 changes: 3 additions & 3 deletions services/app-api/src/authn/authn.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { Result } from 'neverthrow'
import { UserType } from '../domain-models'
import { Store } from '../postgres'
import type { Result } from 'neverthrow'
import type { UserType } from '../domain-models'
import type { Store } from '../postgres'

export type userFromAuthProvider = (
authProvider: string,
Expand Down
5 changes: 3 additions & 2 deletions services/app-api/src/authn/cognitoAuthn.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { Result, ok, err } from 'neverthrow'
import type { Result } from 'neverthrow'
import { ok, err } from 'neverthrow'
import { parseAuthProvider, userTypeFromAttributes } from './cognitoAuthn'
import { UserType } from '../domain-models'
import type { UserType } from '../domain-models'

describe('cognitoAuthn', () => {
describe('parseAuthProvider', () => {
Expand Down
10 changes: 6 additions & 4 deletions services/app-api/src/authn/cognitoAuthn.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
import { Result, ok, err } from 'neverthrow'
import type { Result } from 'neverthrow'
import { ok, err } from 'neverthrow'
import type { UserType as CognitoUserType } from '@aws-sdk/client-cognito-identity-provider'
import {
CognitoIdentityProviderClient,
ListUsersCommand,
UserType as CognitoUserType,
} from '@aws-sdk/client-cognito-identity-provider'

import { UserType } from '../domain-models'
import type { UserType } from '../domain-models'
import { performance } from 'perf_hooks'
import { Store, InsertUserArgsType, isStoreError } from '../postgres'
import type { Store, InsertUserArgsType } from '../postgres'
import { isStoreError } from '../postgres'
import { isValidCmsDivison } from '../domain-models'

export function parseAuthProvider(
Expand Down
2 changes: 1 addition & 1 deletion services/app-api/src/authn/index.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
export { userFromAuthProvider } from './authn'
export type { userFromAuthProvider } from './authn'

export { userFromCognitoAuthProvider, lookupUserAurora } from './cognitoAuthn'

Expand Down
8 changes: 5 additions & 3 deletions services/app-api/src/authn/localAuthn.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import { Result, ok, err } from 'neverthrow'
import { UserType } from '../domain-models/index'
import { Store, InsertUserArgsType, isStoreError } from '../postgres'
import type { Result } from 'neverthrow'
import { ok, err } from 'neverthrow'
import type { UserType } from '../domain-models/index'
import type { Store, InsertUserArgsType } from '../postgres'
import { isStoreError } from '../postgres'
import { lookupUserAurora } from './cognitoAuthn'

export async function userFromLocalAuthProvider(
Expand Down
4 changes: 2 additions & 2 deletions services/app-api/src/domain-models/HealthPlanPackageType.d.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { IndexQuestionsPayload } from './QuestionsType'
import type { IndexQuestionsPayload } from './QuestionsType'

type HealthPlanPackageStatusType =
| 'DRAFT'
Expand Down Expand Up @@ -27,7 +27,7 @@ type HealthPlanRevisionType = {
formDataProto: Uint8Array
}

export {
export type {
HealthPlanPackageStatusType,
HealthPlanPackageType,
UpdateInfoType,
Expand Down
2 changes: 1 addition & 1 deletion services/app-api/src/domain-models/QuestionResponseType.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { StateUserType } from './UserType'
import type { StateUserType } from './UserType'

type QuestionResponseDocument = {
name: string
Expand Down
6 changes: 3 additions & 3 deletions services/app-api/src/domain-models/QuestionsType.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { CMSUserType } from './UserType'
import { QuestionResponseType } from './QuestionResponseType'
import { DivisionType } from './DivisionType'
import type { CMSUserType } from './UserType'
import type { QuestionResponseType } from './QuestionResponseType'
import type { DivisionType } from './DivisionType'

type Document = {
name: string
Expand Down
4 changes: 2 additions & 2 deletions services/app-api/src/domain-models/UserType.d.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { StateType } from './StateType'
import { DivisionType } from './DivisionType'
import type { StateType } from './StateType'
import type { DivisionType } from './DivisionType'

type UserType = StateUserType | CMSUserType | AdminUserType | HelpdeskUserType

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,4 @@ export type {
ContractFormDataType,
} from './contractTypes'

export { ContractStatusType, UpdateInfoType } from './updateInfoType'
export type { ContractStatusType, UpdateInfoType } from './updateInfoType'
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { z } from 'zod'
import { contractSchema } from './contractTypes'
import type { contractSchema } from './contractTypes'

const updateInfoSchema = z.object({
updatedAt: z.date(),
Expand Down
2 changes: 1 addition & 1 deletion services/app-api/src/domain-models/division.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { DivisionType } from './DivisionType'
import type { DivisionType } from './DivisionType'

function isValidCmsDivison(division: string): division is DivisionType {
return ['DMCO', 'DMCP', 'OACT'].includes(division)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import {
packageSubmittedAt,
packageSubmitters,
} from './healthPlanPackage'
import {
import type {
HealthPlanPackageStatusType,
HealthPlanPackageType,
} from './HealthPlanPackageType'
Expand Down
6 changes: 3 additions & 3 deletions services/app-api/src/domain-models/healthPlanPackage.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
import {
import type {
HealthPlanRevisionType,
HealthPlanPackageStatusType,
HealthPlanPackageType,
} from './HealthPlanPackageType'
import { pruneDuplicateEmails } from '../emailer/formatters'
import { ContractType } from './contractAndRates'
import {
import type { ContractType } from './contractAndRates'
import type {
SubmissionDocument,
UnlockedHealthPlanFormDataType,
} from '../../../app-web/src/common-code/healthPlanFormDataType'
Expand Down
4 changes: 2 additions & 2 deletions services/app-api/src/domain-models/user.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
import {
import type {
StateUserType,
CMSUserType,
UserType,
AdminUserType,
HelpdeskUserType,
} from './UserType'
import { User as PrismaUser } from '@prisma/client'
import type { User as PrismaUser } from '@prisma/client'

function isUser(user: unknown): user is UserType {
if (user && typeof user === 'object') {
Expand Down
7 changes: 3 additions & 4 deletions services/app-api/src/emailer/awsSES.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
import {
SESClient,
import type {
SESServiceException,
SendEmailRequest,
SendEmailResponse,
SendEmailCommand,
} from '@aws-sdk/client-ses'
import { EmailData } from './'
import { SESClient, SendEmailCommand } from '@aws-sdk/client-ses'
import type { EmailData } from './'

const ses = new SESClient({ region: 'us-east-1' })

Expand Down
4 changes: 2 additions & 2 deletions services/app-api/src/emailer/emailer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ import {
resubmitPackageStateEmail,
resubmitPackageCMSEmail,
} from './'
import {
import type {
LockedHealthPlanFormDataType,
UnlockedHealthPlanFormDataType,
} from '../../../app-web/src/common-code/healthPlanFormDataType'
import { UpdateInfoType, ProgramType } from '../domain-models'
import type { UpdateInfoType, ProgramType } from '../domain-models'
import { SESServiceException } from '@aws-sdk/client-ses'

// See more discussion of configuration in docs/Configuration.md
Expand Down
20 changes: 10 additions & 10 deletions services/app-api/src/emailer/emails/newPackageCMSEmail.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ import {
mockMNState,
mockMSState,
} from '../../testHelpers/emailerHelpers'
import type { LockedHealthPlanFormDataType } from '../../../../app-web/src/common-code/healthPlanFormDataType'
import {
generateRateName,
LockedHealthPlanFormDataType,
packageName,
} from '../../../../app-web/src/common-code/healthPlanFormDataType'
import { newPackageCMSEmail } from './index'
Expand Down Expand Up @@ -331,7 +331,7 @@ test('includes expected data summary for a multi-rate contract and rates submiss
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateCertified: new Date('10/17/2022'),
rateProgramIDs: ['3fd36500-bf2c-47bc-80e8-e7aa417184c5'],
rateCertificationName:
Expand Down Expand Up @@ -359,7 +359,7 @@ test('includes expected data summary for a multi-rate contract and rates submiss
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateCertified: new Date('10/17/2022'),
rateProgramIDs: ['abbdf9b0-c49e-4c4c-bb6f-040cb7b51cce'],
rateCertificationName:
Expand Down Expand Up @@ -387,7 +387,7 @@ test('includes expected data summary for a multi-rate contract and rates submiss
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateCertified: new Date('10/17/2022'),
rateProgramIDs: [
'ea16a6c0-5fc6-4df8-adac-c627e76660ab',
Expand Down Expand Up @@ -510,7 +510,7 @@ test('includes expected data summary for a contract amendment submission', async
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateCertified: new Date('10/17/2022'),
rateProgramIDs: ['3fd36500-bf2c-47bc-80e8-e7aa417184c5'],
rateCertificationName:
Expand Down Expand Up @@ -591,7 +591,7 @@ test('includes expected data summary for a rate amendment submission CMS email',
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateCertified: new Date('10/17/2022'),
rateProgramIDs: ['3fd36500-bf2c-47bc-80e8-e7aa417184c5'],
rateCertificationName:
Expand Down Expand Up @@ -855,7 +855,7 @@ test('CHIP contract and rate submission does include state specific analysts ema
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateStart: new Date(),
rateDateEnd: new Date(),
rateDateCertified: new Date(),
Expand Down Expand Up @@ -940,7 +940,7 @@ test('CHIP contract and rate submission does not include oactEmails', async () =
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateStart: new Date('2021-02-02'),
rateDateEnd: new Date('2021-11-31'),
rateDateCertified: new Date('2020-12-01'),
Expand Down Expand Up @@ -1019,7 +1019,7 @@ test('renders overall email as expected', async () => {
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateCertified: new Date('01/02/2021'),
rateProgramIDs: ['3fd36500-bf2c-47bc-80e8-e7aa417184c5'],
rateCertificationName:
Expand Down Expand Up @@ -1047,7 +1047,7 @@ test('renders overall email as expected', async () => {
documentCategories: ['RATES' as const],
},
],
supportingDocuments: [],
supportingDocuments: [],
rateDateCertified: new Date('02/02/2022'),
rateProgramIDs: ['abbdf9b0-c49e-4c4c-bb6f-040cb7b51cce'],
rateCertificationName:
Expand Down
Loading

0 comments on commit ea49a2d

Please sign in to comment.