Merge branch 'main' into wml-consolodate-contract-types

docs/technical-design/howto-migrations.md

@@ -1,49 +1,69 @@
 # How to complete migrations
 
 ## Background
+
 Migrations are a controlled way to change existing records in the database or else change the shape of tables/relationships. They are necessary when fields are added, removed, or when the meaning behind fields and their relationships change.
 
 ## General Guidance
-- Prioritize testing in environments that have data similar to production. If you are testing on a review app, you should pre-populate data through using the deployed app as a user or via Cypress runs to have a diversity of test data available.
-- Run in all deployed environments and validate for each environment.
-- Write migrations that can be tested repeatedly.  Include logic to skip or handle data that has already been changed or doesn't need to be migrated, so repeated runs of the lambda don't result in bad data.
-- Avoid making migrations directly dependent on a specific feature being available in production.
+
+-   Prioritize testing in environments that have data similar to production. If you are testing on a review app, you should pre-populate data through using the deployed app as a user or via Cypress runs to have a diversity of test data available.
+-   Run in all deployed environments and validate for each environment.
+-   Write migrations that can be tested repeatedly. Include logic to skip or handle data that has already been changed or doesn't need to be migrated, so repeated runs of the lambda don't result in bad data.
+-   Avoid making migrations directly dependent on a specific feature being available in production.
 
 ## Data Migrations
-Currently data migrations change existing records in the database. They are run as standalone lambdas that developers must manually trigger by environment.
 
-### Steps
+Currently data migrations change existing records in the database. They are run as standalone lambdas that developers must manually trigger by environment. They can be prototyped for fast feedback in the local environment, and then tested more thoroughly in lower environments.
+
+### Local prototyping
+
+1. Log into the AWS console (you'll need to be connected to the CMS VPN).
+1. Go to the Dev environment dropdown: Cloud Access Roles --> Developer Admin --> Short-term Access Keys.
+1. Choose option 1, and copy those values to the clipboard.
+1. Paste the values into your .envrc.local file. Be sure that you're overwriting any previous values, and don't have duplicates.
+1. If the app is running anywhere, stop it.
+1. Run `DIRENV ALLOW` in the terminal in the directory where you'll start the app.
+1. Start the app.
+1. When the app is running, connect to the local database so that you can inspect the changes you're making. Use the DATABASE_URL from your .envrc.local file to connect to the database interface of your choice. You can connect via the terminal, or an app like TablePlus, dBeaver, Postico, DataGrip, etc.
+1. Run some Cypress tests to populate the database. The CMSWorkflow tests are a good choice, since they create two entries which contain rates, one of which has a revision.
+1. Verify that in the local database you can see entries in the HealthPlanPackageTable and HealthPlanRevisionTable.
+1. Now you're ready to run your lambda. In a different terminal instance, run `DIRENV ALLOW` again.
+1. Navigate to `services/app-api`.
+1. Run `npx serverless invoke local --function `name_of_your_lambda``. You should see it build with webpack, then run. Any log statements you've put in your code will display in this window.
+1. You can inspect the database again and should see any changes or additions you've made.
+
+### Running in a lower environment
+
 1. Prepare for manual testing in lower environments.
     - Build a PR review app off `main` to start out. Try to populate with submissions similar to what is in production (For example, populate submissions via Cypress and if there are specific submissions types you know you will need to test, make sure add them).
 1. Write the migration with verbose logs.
-    - The migration will be written as a [Node lambda](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html) and live in `app-api/src/handlers`.  Make sure you use ES6 async await.
+    - The migration will be written as a [Node lambda](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html) and live in `app-api/src/handlers`. Make sure you use ES6 async await.
     - Console statements are essential because there are line numbers in Cloudwatch when the migration fails or does not apply correctly. There is little additional context about why a run execution paused or failed.
     - Here is an example of verbose debug console statements wrapping a loop:
-    ![debug consoles in data migration script](../../.images/verbose-logs-example.png)
+      ![debug consoles in data migration script](../../.images/verbose-logs-example.png)
 1. Write unit tests.
     - Consider including a test for `migration can be run repeatedly without data loss or unexpected results". Using repeatable migrations (when possible) makes it easier on developers and easier to unwind work.
 1. Manually test the migration in your review app.
     - Log into [AWS Lambda Console](https://console.aws.amazon.com/lambda/home) and find the lambda. This means choosing `app-api-` lambda with your branch name and migration name included.
     - Click the Test tab and the `Test` button. Use the generic hello world event.
         - All output will appear inline on the same page as the lambda run. All consoles appear there as well. You can also click into a link from there into Cloudwatch to be able to see the entire output of logs around the lambda execution.
-     - If you need to debug quickly, without waiting for redeploy in review app use the severless CLI script.
-        -  `serverless deploy function --function name_of_migration --stage my_branch && echo "DEPLOYED" && serverless logs --function name_of_migration  --stage my_branch --tail` can be run in your `app-api` directory. This script deploys your local handler code directly to AWS, without going through our normal CI build process.  `my_branch`is the name of your deployed Github branch and `name_of_migration` is the name of the migration file mind the file path.
+    - If you need to debug quickly, without waiting for redeploy in review app use the severless CLI script.
+        - `serverless deploy function --function name_of_migration --stage my_branch && echo "DEPLOYED" && serverless logs --function name_of_migration  --stage my_branch --tail` can be run in your `app-api` directory. This script deploys your local handler code directly to AWS, without going through our normal CI build process. `my_branch`is the name of your deployed Github branch and `name_of_migration` is the name of the migration file mind the file path.
         - If this is to be run in DEV (`main`) you will have to reshape your `envrc.local` to imitate configuration on that stage. See AWS console for Lambda to reference the shape of config.
-        ![lambda configuration panel in aws](../../.images/aws-console-lambda-config.png)
+          ![lambda configuration panel in aws](../../.images/aws-console-lambda-config.png)
         - Command line serverless scripts should not be used on PROD. Ideally do not use in VAL either. Running a migration via the AWS lambda console web interface is preferred because it is guaranteed to build off merged code and use similar configuration to how our application is deployed in CI.
 1. Define clear acceptance criteria, this will be re-used.
     - How do you know migration applied? What will be checked either in the app, via api request, or in the reports CSV?
     - Developers will have to verify that the change worked multiple times, basically each time the migration is run on a new environment since the data could be quite different.
     - If verification involves looking into reports and comparing fields, consider using a CSV tool like [CSVKit](https://csvkit.readthedocs.io/en/latest/index.html) if the comparisons seem involved.
 1. After the PR merges and promotes,run in higher environments.
     - Start with DEV. Follow similar steps Step #4 but now using the `main` lambda for your migration.
-    - Run the migration  DEV > VAL > PROD in order, verifying in the application (or via reporting output) after each run.
-
+    - Run the migration DEV > VAL > PROD in order, verifying in the application (or via reporting output) after each run.
 
 ## Schema Migrations
+
 We use Prisma to perform schema migrations. Right now the developer process for schema migrations is described a bit [here](../../README.md#updating-the-database). More detailed steps forthcoming.
 
 ## Proto migrations
-The [proto migrations approach](../../services/app-proto/README.md#adding-a-new-migration) is no longer recommended - we are moving off protos.
-
 
+The [proto migrations approach](../../services/app-proto/README.md#adding-a-new-migration) is no longer recommended - we are moving off protos.

services/app-api/.eslintrc

@@ -24,6 +24,8 @@
         "jest/no-identical-title": "error",
         "@typescript-eslint/no-floating-promises": "error",
         "no-unused-vars": "off",
+        "@typescript-eslint/consistent-type-exports": "error",
+        "@typescript-eslint/consistent-type-imports": "error",
         "@typescript-eslint/no-unused-vars": [
             "warn",
             { "argsIgnorePattern": "^_*" }

services/app-api/src/authn/authn.ts

@@ -1,6 +1,6 @@
-import { Result } from 'neverthrow'
-import { UserType } from '../domain-models'
-import { Store } from '../postgres'
+import type { Result } from 'neverthrow'
+import type { UserType } from '../domain-models'
+import type { Store } from '../postgres'
 
 export type userFromAuthProvider = (
     authProvider: string,

services/app-api/src/authn/cognitoAuthn.test.ts

@@ -1,6 +1,7 @@
-import { Result, ok, err } from 'neverthrow'
+import type { Result } from 'neverthrow'
+import { ok, err } from 'neverthrow'
 import { parseAuthProvider, userTypeFromAttributes } from './cognitoAuthn'
-import { UserType } from '../domain-models'
+import type { UserType } from '../domain-models'
 
 describe('cognitoAuthn', () => {
     describe('parseAuthProvider', () => {

services/app-api/src/authn/cognitoAuthn.ts

@@ -1,13 +1,15 @@
-import { Result, ok, err } from 'neverthrow'
+import type { Result } from 'neverthrow'
+import { ok, err } from 'neverthrow'
+import type { UserType as CognitoUserType } from '@aws-sdk/client-cognito-identity-provider'
 import {
     CognitoIdentityProviderClient,
     ListUsersCommand,
-    UserType as CognitoUserType,
 } from '@aws-sdk/client-cognito-identity-provider'
 
-import { UserType } from '../domain-models'
+import type { UserType } from '../domain-models'
 import { performance } from 'perf_hooks'
-import { Store, InsertUserArgsType, isStoreError } from '../postgres'
+import type { Store, InsertUserArgsType } from '../postgres'
+import { isStoreError } from '../postgres'
 import { isValidCmsDivison } from '../domain-models'
 
 export function parseAuthProvider(

services/app-api/src/authn/index.ts

@@ -1,4 +1,4 @@
-export { userFromAuthProvider } from './authn'
+export type { userFromAuthProvider } from './authn'
 
 export { userFromCognitoAuthProvider, lookupUserAurora } from './cognitoAuthn'
 

services/app-api/src/authn/localAuthn.ts

@@ -1,6 +1,8 @@
-import { Result, ok, err } from 'neverthrow'
-import { UserType } from '../domain-models/index'
-import { Store, InsertUserArgsType, isStoreError } from '../postgres'
+import type { Result } from 'neverthrow'
+import { ok, err } from 'neverthrow'
+import type { UserType } from '../domain-models/index'
+import type { Store, InsertUserArgsType } from '../postgres'
+import { isStoreError } from '../postgres'
 import { lookupUserAurora } from './cognitoAuthn'
 
 export async function userFromLocalAuthProvider(

services/app-api/src/domain-models/HealthPlanPackageType.d.ts

@@ -1,4 +1,4 @@
-import { IndexQuestionsPayload } from './QuestionsType'
+import type { IndexQuestionsPayload } from './QuestionsType'
 
 type HealthPlanPackageStatusType =
     | 'DRAFT'
@@ -27,7 +27,7 @@ type HealthPlanRevisionType = {
     formDataProto: Uint8Array
 }
 
-export {
+export type {
     HealthPlanPackageStatusType,
     HealthPlanPackageType,
     UpdateInfoType,

services/app-api/src/domain-models/QuestionResponseType.ts

@@ -1,4 +1,4 @@
-import { StateUserType } from './UserType'
+import type { StateUserType } from './UserType'
 
 type QuestionResponseDocument = {
     name: string

services/app-api/src/domain-models/QuestionsType.ts

@@ -1,6 +1,6 @@
-import { CMSUserType } from './UserType'
-import { QuestionResponseType } from './QuestionResponseType'
-import { DivisionType } from './DivisionType'
+import type { CMSUserType } from './UserType'
+import type { QuestionResponseType } from './QuestionResponseType'
+import type { DivisionType } from './DivisionType'
 
 type Document = {
     name: string

services/app-api/src/domain-models/UserType.d.ts

@@ -1,5 +1,5 @@
-import { StateType } from './StateType'
-import { DivisionType } from './DivisionType'
+import type { StateType } from './StateType'
+import type { DivisionType } from './DivisionType'
 
 type UserType = StateUserType | CMSUserType | AdminUserType | HelpdeskUserType
 

services/app-api/src/domain-models/contractAndRates/index.ts

@@ -19,4 +19,4 @@ export type {
     ContractFormDataType,
 } from './contractTypes'
 
-export { ContractStatusType, UpdateInfoType } from './updateInfoType'
+export type { ContractStatusType, UpdateInfoType } from './updateInfoType'

services/app-api/src/domain-models/contractAndRates/updateInfoType.ts

@@ -1,5 +1,5 @@
 import { z } from 'zod'
-import { contractSchema } from './contractTypes'
+import type { contractSchema } from './contractTypes'
 
 const updateInfoSchema = z.object({
     updatedAt: z.date(),

services/app-api/src/domain-models/division.ts

@@ -1,4 +1,4 @@
-import { DivisionType } from './DivisionType'
+import type { DivisionType } from './DivisionType'
 
 function isValidCmsDivison(division: string): division is DivisionType {
     return ['DMCO', 'DMCP', 'OACT'].includes(division)

services/app-api/src/domain-models/healthPlanPackage.test.ts

@@ -3,7 +3,7 @@ import {
     packageSubmittedAt,
     packageSubmitters,
 } from './healthPlanPackage'
-import {
+import type {
     HealthPlanPackageStatusType,
     HealthPlanPackageType,
 } from './HealthPlanPackageType'

services/app-api/src/domain-models/healthPlanPackage.ts

@@ -1,11 +1,11 @@
-import {
+import type {
     HealthPlanRevisionType,
     HealthPlanPackageStatusType,
     HealthPlanPackageType,
 } from './HealthPlanPackageType'
 import { pruneDuplicateEmails } from '../emailer/formatters'
-import { ContractType } from './contractAndRates'
-import {
+import type { ContractType } from './contractAndRates'
+import type {
     SubmissionDocument,
     UnlockedHealthPlanFormDataType,
 } from '../../../app-web/src/common-code/healthPlanFormDataType'

services/app-api/src/domain-models/user.ts

@@ -1,11 +1,11 @@
-import {
+import type {
     StateUserType,
     CMSUserType,
     UserType,
     AdminUserType,
     HelpdeskUserType,
 } from './UserType'
-import { User as PrismaUser } from '@prisma/client'
+import type { User as PrismaUser } from '@prisma/client'
 
 function isUser(user: unknown): user is UserType {
     if (user && typeof user === 'object') {

services/app-api/src/emailer/awsSES.ts

@@ -1,11 +1,10 @@
-import {
-    SESClient,
+import type {
     SESServiceException,
     SendEmailRequest,
     SendEmailResponse,
-    SendEmailCommand,
 } from '@aws-sdk/client-ses'
-import { EmailData } from './'
+import { SESClient, SendEmailCommand } from '@aws-sdk/client-ses'
+import type { EmailData } from './'
 
 const ses = new SESClient({ region: 'us-east-1' })
 

services/app-api/src/emailer/emailer.ts

@@ -9,11 +9,11 @@ import {
     resubmitPackageStateEmail,
     resubmitPackageCMSEmail,
 } from './'
-import {
+import type {
     LockedHealthPlanFormDataType,
     UnlockedHealthPlanFormDataType,
 } from '../../../app-web/src/common-code/healthPlanFormDataType'
-import { UpdateInfoType, ProgramType } from '../domain-models'
+import type { UpdateInfoType, ProgramType } from '../domain-models'
 import { SESServiceException } from '@aws-sdk/client-ses'
 
 // See more discussion of configuration in docs/Configuration.md

services/app-api/src/emailer/emails/newPackageCMSEmail.test.ts

@@ -9,9 +9,9 @@ import {
     mockMNState,
     mockMSState,
 } from '../../testHelpers/emailerHelpers'
+import type { LockedHealthPlanFormDataType } from '../../../../app-web/src/common-code/healthPlanFormDataType'
 import {
     generateRateName,
-    LockedHealthPlanFormDataType,
     packageName,
 } from '../../../../app-web/src/common-code/healthPlanFormDataType'
 import { newPackageCMSEmail } from './index'
@@ -331,7 +331,7 @@ test('includes expected data summary for a multi-rate contract and rates submiss
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateCertified: new Date('10/17/2022'),
                 rateProgramIDs: ['3fd36500-bf2c-47bc-80e8-e7aa417184c5'],
                 rateCertificationName:
@@ -359,7 +359,7 @@ test('includes expected data summary for a multi-rate contract and rates submiss
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateCertified: new Date('10/17/2022'),
                 rateProgramIDs: ['abbdf9b0-c49e-4c4c-bb6f-040cb7b51cce'],
                 rateCertificationName:
@@ -387,7 +387,7 @@ test('includes expected data summary for a multi-rate contract and rates submiss
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateCertified: new Date('10/17/2022'),
                 rateProgramIDs: [
                     'ea16a6c0-5fc6-4df8-adac-c627e76660ab',
@@ -510,7 +510,7 @@ test('includes expected data summary for a contract amendment submission', async
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateCertified: new Date('10/17/2022'),
                 rateProgramIDs: ['3fd36500-bf2c-47bc-80e8-e7aa417184c5'],
                 rateCertificationName:
@@ -591,7 +591,7 @@ test('includes expected data summary for a rate amendment submission CMS email',
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateCertified: new Date('10/17/2022'),
                 rateProgramIDs: ['3fd36500-bf2c-47bc-80e8-e7aa417184c5'],
                 rateCertificationName:
@@ -855,7 +855,7 @@ test('CHIP contract and rate submission does include state specific analysts ema
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateStart: new Date(),
                 rateDateEnd: new Date(),
                 rateDateCertified: new Date(),
@@ -940,7 +940,7 @@ test('CHIP contract and rate submission does not include oactEmails', async () =
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateStart: new Date('2021-02-02'),
                 rateDateEnd: new Date('2021-11-31'),
                 rateDateCertified: new Date('2020-12-01'),
@@ -1019,7 +1019,7 @@ test('renders overall email as expected', async () => {
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateCertified: new Date('01/02/2021'),
                 rateProgramIDs: ['3fd36500-bf2c-47bc-80e8-e7aa417184c5'],
                 rateCertificationName:
@@ -1047,7 +1047,7 @@ test('renders overall email as expected', async () => {
                         documentCategories: ['RATES' as const],
                     },
                 ],
-                  supportingDocuments: [],
+                supportingDocuments: [],
                 rateDateCertified: new Date('02/02/2022'),
                 rateProgramIDs: ['abbdf9b0-c49e-4c4c-bb6f-040cb7b51cce'],
                 rateCertificationName: