Implement viz_publish_service lambda auto republish via geoprocessing service

[EdisonOrellana-NOAA]

This PR is essentially the same as #630 but with a few changes and loose ends to tie up. I believe mostly everything is captured in the list below but I will add additional information as a PR comment as it comes up.

Background

At this point, all services have been moved off of the Viz EC2 instance, and it only performs one task: converting ArcGIS Pro Projects (which we save as MapX documents) into the ArcGIS service definition (SD) files that viz pipelines need to automatically publish services to ArcGIS server, including updating the data sources with the correct database host and authentication information.

The Viz EC2 instance does this by running the \aws_loosa\deploy\update_data_stores_and_sd_files.py script as part of the Core\EC2\viz\scripts\viz_ec2_setup.ps1 deployment script, and this is the number one cause of issues during all deployments, as it takes quite a bit of work to dig into the logs and troubleshoot issues on the Viz EC2 machine. Because ESRI is greedy and won't fully allow their Python API to update these map document parameters (or allow use of a limited version of ArcPy in a container that can do it), we've had to get creative on another way to automatically update these map service definitions as part of our automatic pipelines - credit to Justin for initially coming up with the idea of using a geoprocessing service.

With Bob's help, I've successfully tested this geoprocessing service in TI, and while questions remain around how this will actually work operationally with the regular pipelines, it seems worth a shot. That said, because it is ESRI, there is a pretty convoluted workflow required to make this happen.

Steps required to finish

• Convert the update_data_stores_and_sd_files python script into a standalone script that can be published as a ArcGIS Geoprcessing Service (in this PR).
• Test the geoprocessing service via REST - you can find the test here.
• Allow the ESRISSMDeploy IAM Role to view the EGIS RDS user secret (this allows the geoprocessing server to fetch the RDS password securely from AWS without the password being passed around in text in the REST calls). The Terraform for this is included in MapX to SD File - ArcGIS Geoprocessing Service Script #630
• @nickchadwick-noaa Add IaC for copying *.mapx files in the subdirectories of Core/LAMBDA/viz_functions/viz_publish_service/services to the expected location on S3. Currently in ti this is s3://hydrovis-ti-deployment-us-east-1/viz/pro_projects/. This value is currently hard coded in the lambda_function.py but should be set from an environment variable.
• Add additional lambda enviornment variables in terraform:

EGIS_DB_DATABASE	hydrovis
EGIS_DB_HOST	rds-egis.hydrovis.internal
EGIS_DB_PASSWORD_SECRET_NAME	hv-vpp-ti-egis-pg-rds-secret
EGIS_DB_USERNAME	hydrovis
ENVIRONMENT	ti

• Configure ArcGIS Server to use a custom python environment with the AWS Boto3 library installed. Bob did this in TI following the instructions of this ESRI doc. I believe this will need to be baked into the AMI, so this task should be assigned to whoever owns that process. I'm not sure which server of the ArcGIS stack this should be installed on, probably the same one as the print service? After futilely attempting to make a custom python environment work, @shawncrawley suggested avoiding the problem by making a subprocess call to the aws command to avoid needing boto3. This worked well. See this comment
• Publish the geoprocessing service to each ArcGIS Server stack - This is unfortunately a manual process that must be done with the admin credentials. See these instructions which are more or less still the same. Exception is regarding default values and which values are required.
• Update the viz_publish_service lambda function to call this geoprocessing service first via API when it needs to republish a service. (changes included in this PR) This can be done now in TI, using the test geoprocessing service I created, to test this out with the operational TI pipelines.
• Delete the Viz EC2 entirely (it should be manually stopped after a deployment for the time being, as it doesn't serve any purpose after the initial deployment steps).

Testing

1. Additional unit tests per service?

[shawncrawley]

What is the purpose of this?

[EdisonOrellana-NOAA]

Remnant from creating a markdown file on Windows. (rich text format)
This readme is a placeholder meant to hold documentation for the GP service.

[EdisonOrellana-NOAA]

Thanks for the help getting this on UAT!
Further steps:

• Update the URL to call the service in UAT. This gets updated in the Lambda
• Fix the terraform to configure the proper location for the mapx files

The merge-base changed after approval.

[EdisonOrellana-NOAA]

• Change the service name on maps-testing to match what is named in UAT
• Delete the parts that are currently creating SD files since this service deprecates that functionality
• Update read me to match staging
• Follow up with Shawn about examining success on UAT
• I left the service with the old name alongside it. So nothing is broken currently. After successful deployment to TI, it should start using the new service and then we can delete the old service.

[EdisonOrellana-NOAA]

UAT test requested for geoprocessing service

The geoprocessing service is meant to convert a mapx file to a service definition (.sd) file and place the file in the necessary location on S3. The viz_publish_service lambda publishes the service using that .sd file.

Test procedure

1. Rename/move a service definition file. Example, ana_high_flow_magnitude_ak_noaa.sd
   s3://hydrovis-uat-deployment-us-east-1/viz_sd_files/ana_high_flow_magnitude_ak_noaa.sd
2. Rename/move the corresponding publish flag file. Example, ana_high_flow_magnitude_ak_noaa
   s3://hydrovis-uat-fim-us-east-1/published_flags/server/nwm/ana_high_flow_magnitude_ak_noaa/ana_high_flow_magnitude_ak_noaa
3. Either trigger a pipeline run in UAT that includes ana_high_flow_magnitude_ak_noaa (preferred) OR trigger the GP service manually at the REST URL
4. Observe Cloudwatch logs for the Lambda: hv-vpp-uat-viz-publish-service and look for the log entry: ---> ana_high_flow_magnitude_ak_noaa does not currently exist. Attempting to publish via GP service.
5. Verify successful recreation of the service definition (.sd) file at s3://hydrovis-uat-deployment-us-east-1/viz_sd_files/ana_high_flow_magnitude_ak_noaa.sd
6. Verify successful recreation of publish flag at s3://hydrovis-uat-fim-us-east-1/published_flags/server/nwm/ana_high_flow_magnitude_ak_noaa/ana_high_flow_magnitude_ak_noaa
7. Verify the tested service is functional in UAT

Success

.sd and publish flag are successfully recreated and service is successfully republished and working

Failure

.sd and publish flag are NOT successfully recreated and service is successfully republished and working

Recovery/rollback

Manually restore the .sd file and publish flag in their respective locations in S3