BRS-237 - Migration for Historical Data 2000-2016

[davidclaveau]

Ticket:

BRS-237

Ticket URL:

#237

Description:

• Copy over and update /migrations-data/legacy-data/2017-2019 into new migrations folder for SAM
• Small tweaks to remove Keycloak params and subsequent role creation functionality
• Update to use AWS SDK v3
• Small logic tweaks to avoid importing empty cells in .xlsx file

[cameronpettit]

I'm going to trust this is correct since I wrote it originally, we've discussed it together and you agree it's sane, and you've run it successfully locally. We can run this against our envs to validate before it hits prod, so I think its good to go.

One small consideration, which I'm not sure has been accounted for - the original isLegacy tag was used in the event that we needed to back out the migration entirely. Wondering if we need to differentiate the new records you're going to be migrating in from the ones that I imported (2017-2019). If we need to back out your migration, we wont be able to differentiate your migrated records from mine. Perhaps it would be useful to add another field like legacyMigrationVersion: 2 on your records so we have a little more fidelity on our imported records?

[davidclaveau]

One small consideration, which I'm not sure has been accounted for - the original isLegacy tag was used in the event that we needed to back out the migration entirely. Wondering if we need to differentiate the new records you're going to be migrating in from the ones that I imported (2017-2019)...

I was initially going to just snapshot the database and save it in S3 and reimport if need be, but it's probably best to use the code if it exists and save time that way. I've added the purgeLegacy.js function and updated to use SDK v3 as well. I've used your recommendation and added legacyMigrationVersion: 2 for all these records and will purge legacy records using that attribute. Good call!

Also added some scan constraints to the Export Report option for when the user selects Export all - opted to not scan past 2017 (so not scanning the 150,000+ records added).