Setup new manual workflow file

.gitignore

@@ -50,4 +50,7 @@ terraform.rc
 **/buildplan
 **/destroyplan
 
+# Ignore all jupyter checkpoint folders
+.ipynb_checkpoints
+
 Dev/env.yaml

Core/Manual_Workflows/11. RASTER PRODUCT DEV WORKFLOW.ipynb

@@ -0,0 +1,228 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "4f3d72f8-df5f-4335-b81a-9a3996e27348",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "<h3>Step 1: Create the raster product processing file in the raster_product_development/products folder</h3>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64369f01-e501-4a87-8d73-b94cb03432aa",
+   "metadata": {},
+   "source": [
+    "<h3>Step 2: Tweak the following variable per the service you are developing</h3>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "07050a1f-6b4a-499c-9086-eae485ae463b",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "SERVICE_NAME = 'mrf_gfs_10day_snow_water_equivalent_change'\n",
+    "PRODUCT_FILE_NAME = SERVICE_NAME  # The name of the file in the raster_product_development/products folder. It's not always the same as the service name.\n",
+    "FILE_FORMAT = 'common/data/model/com/nwm/prod/nwm.{{datetime:%Y%m%d}}/medium_range_mem1/nwm.t{{datetime:%H}}z.medium_range.land_1.f003.conus.nc'\n",
+    "FILE_STEP = '24H'\n",
+    "FILE_WINDOW = 'P10D'\n",
+    "REFERENCE_TIME = '2024-06-13 00:00:00'\n",
+    "RUN_OPTIMIZE_RASTERS_STEP = True  # Set this to true when you are ready to create Pro Project for service"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "26bf0bfe-a3ee-4a73-81a3-9bff14338dd2",
+   "metadata": {},
+   "source": [
+    "<h3>Step 3: Click the \"Restart Kernel and Run All Cells...\" icon above (i.e. the double play icon) to test/execute your script</h3>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3937b328-6575-4bc7-8a07-b4cc4d6f0e96",
+   "metadata": {},
+   "source": [
+    "When processed successfully, you will see \"ALL DONE!\" printed below and your raster outputs will be found at s3://hydrovis-ti-fim-us-east-1/processing_outputs/<SERVICE_NAME>/YYYYMMDD/HH/workspace/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7d4967e1-755b-47e9-be49-4d18f22a05dd",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "REFERENCE_TIME_DATE = ''.join(REFERENCE_TIME.split(' ')[0].split('-'))\n",
+    "REFERENCE_TIME_HOUR = REFERENCE_TIME.split(' ')[1].split(':')[0]\n",
+    "\n",
+    "# The name of the product file as found in the raster_product_development/products folder (often the same as the product/service name)\n",
+    "RASTER_PROCESSING_EVENT_METADATA = {\n",
+    "    \"reference_time\": REFERENCE_TIME,\n",
+    "    \"product\": {\n",
+    "        \"product\": SERVICE_NAME,\n",
+    "        \"raster_input_files\": {\n",
+    "            \"file_format\": FILE_FORMAT,\n",
+    "            \"file_step\": FILE_STEP,\n",
+    "            \"file_window\": FILE_WINDOW,\n",
+    "            \"product_file\": PRODUCT_FILE_NAME,\n",
+    "            \"bucket\": \"nws-shared-data-226711853580-us-east-1\"\n",
+    "        },\n",
+    "        \"raster_outputs\": {\n",
+    "            \"output_bucket\": \"hydrovis-ti-fim-us-east-1\",\n",
+    "            \"output_raster_workspaces\": [\n",
+    "                {\n",
+    "                    f\"{SERVICE_NAME}\": f\"processing_outputs/{SERVICE_NAME}/{REFERENCE_TIME_DATE}/{REFERENCE_TIME_HOUR}/workspace\"\n",
+    "                }\n",
+    "            ]\n",
+    "        }\n",
+    "    }\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0ea46c3f-5279-4f2d-9b93-08428f1fd0d0",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!pip install isodate\n",
+    "from raster_product_development.lambda_function import lambda_handler as run_processing\n",
+    "\n",
+    "print('Running processing...')\n",
+    "output = run_processing(RASTER_PROCESSING_EVENT_METADATA, None)\n",
+    "print('... Done!')\n",
+    "if RUN_OPTIMIZE_RASTERS_STEP:\n",
+    "    import boto3\n",
+    "    import botocore\n",
+    "    import json\n",
+    "    boto_config = botocore.client.Config(max_pool_connections=1, connect_timeout=60, read_timeout=600)\n",
+    "    lambda_client = boto3.client('lambda', config=boto_config)\n",
+    "    print('Optimizing output rasters...')\n",
+    "    for output_raster in output['output_rasters']['output_rasters']:\n",
+    "        lambda_client.invoke(\n",
+    "            FunctionName='hv-vpp-ti-viz-optimize-rasters',\n",
+    "            InvocationType='Event',\n",
+    "            Payload=json.dumps({\n",
+    "                \"output_raster\": output_raster,\n",
+    "                \"output_bucket\": output['output_rasters']['output_bucket']\n",
+    "            })\n",
+    "        )\n",
+    "    print('... Done!')\n",
+    "print('ALL DONE!')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e70f2d13-7e65-4a35-b572-79b32637ed5c",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "<h3>Step 4: Create a Pro Projecto to symbolize the newly created raster data</h3>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e756266a-a5b2-48fb-bc8a-241d19768cc1",
+   "metadata": {},
+   "source": [
+    "Instructions on connecting your ArcGIS Pro to the raster data in S3 data can be found here: https://docs.google.com/presentation/d/1i7-QZhW7DUg3AV9davUy_m4pRutH0nD1iWCMILBm31I/edit#slide=id.p"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "692671de-829f-44e5-9e71-4467cf1b5f12",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "<h3>Step 5: Save .mapx file of your ArcGIS Pro Project and also create service configuration .yml file</h3>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f163ad73-e9aa-4e58-8ce8-9517e63740f0",
+   "metadata": {},
+   "source": [
+    "Commit these two files to your branch of the repo by creating a subfolder for your service here: https://github.com/NOAA-OWP/hydrovis/tree/ti/Core/LAMBDA/viz_functions/viz_publish_service/services"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "09cb2e12-fdbf-4fde-b4ef-5996da7d395a",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "<h3>Step 6: Commit your product script to your branch of the repo</h3>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "43f39478-72ac-422c-94ad-7da5e35ac612",
+   "metadata": {},
+   "source": [
+    "That script should go here: https://github.com/NOAA-OWP/hydrovis/tree/ti/Core/LAMBDA/viz_functions/image_based/viz_raster_processing/products"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1db967d3-977f-49f7-922b-a9be6234c14c",
+   "metadata": {},
+   "source": [
+    "<h3>Step 7: Create a pipeilne product_config .yml file and commit it to your branch of the repo</h3>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "91081787-99df-40fc-bada-8c7b8cfa60d5",
+   "metadata": {},
+   "source": [
+    "That script should go here: https://github.com/NOAA-OWP/hydrovis/tree/ti/Core/LAMBDA/viz_functions/viz_initialize_pipeline/product_configs"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "325e8340-29ba-4946-a59b-976257076551",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "<h3>Step 8: Push your branch of the repo and create/submit a Pull Request</h3>"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "conda_python3",
+   "language": "python",
+   "name": "conda_python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}