\n",
+ "
HydroVIS Visualization Workflow
\n",
+ "
![](\"attachment:IntroDiagram.JPG\")
\n",
+ "
\n",
+ " Viz EGIS services (except inundation) are now stored in a PostgreSQL database. While a version of the on-prem loosa viz pipeline does still run on a Windows virtual machine in HydroVIS, we're moving in the direction of a serverless pipeline based on a central RDS database with processing taking place in on-demand lambda functions. In order to maintain the stability of the DB and lambda functions in the pipeline, we have developed a workflow to minimize the need to work directly with either the database or the automated lambda functions. Instead, services can be developed with structured query language (SQL, and more specifically PostgreSQL) inside of a Jupyter notebook like this one. A good tutorial resource can be found here to learn more about writing SQL. \n",
+ "
\n",
+ "
\n",
+ " In other words, this new framework replaces the process.py and product python files of the on-prem loosa library with SQL files that do the same work in less code (a SQL file is just an executable text file that stores SQL code with a .sql extension, similar to a .py file). That said, the entire pipeline is still python-based, and while we're encouraging the use of SQL wherever possible due to the reduction of code and built-in data validation / QA-QC... we do still have the ability to use Python for post-processing as well, and can continue to evaluate that on a service-by-service basis, depending on requirements.\n",
+ "
\n",
+ "
\n",
+ " There are 3 main categories of SQL files in the viz workflow (explained in detail through the rest of this notebook):\n",
+ "
\n",
+ " - Parent SQL Files (pre-processing, e.g. max flows)
\n",
+ " - Service SQL Files (processing, e.g. loosa product files)
\n",
+ " - Final SQL Files (post-processing, e.g. hucs hotspots)
\n",
+ "
\n",
+ " \n",
+ "
Almost all development of vector services can/should be done in these SQL files (primarily #2 Serivce SQL Files and occasionaly #3 Final SQL Files, such as HUC hotspot layers). We're still working out the right approach for raster services, and will be collaborating on that approach in the coming weeks.
\n",
+ "
\n",
+ "
Getting to Know The Database
\n",
+ "
The VizProcessing PostgreSQL database now serves as our primary source of authoritative and pipeline input data. Similar to the folder structure of the on-prem EGIS pipelines, we now use database schemas to categorize available database tables.
\n",
+ "
\n",
+ "
1. Parent SQL Files
\n",
+ "
These SQL files are used for pre processing NWM data. In most cases, the SQL code creates a table of maximum streamflows for a given forecast. Normally these files will not need to be altered or touched, unless a new data pre-processing needs to be done.
\n",
+ "