api-nwm-gcp

REST API backed by National Water Model data, developed on Google Cloud Platform. This repository contains general architecture diagram for a general understanding, but it focuses on two GCP products and their respective set up files and configurations: Cloud Functions and API Gateway.

Infrastructure setup

This section goes over provisioning the cloud resources for the application. These resources include a Service Account, permissions for the service account, and a repository on Artifact Registry to push built Docker images to. There are a few steps...before beginning make sure you have the terraform CLI installed and the gcloud CLI installed

1. Login with gcloud application default credentials so terraform can run authenticated commands within your cloud environment:

gcloud auth application-default login

2. Update the variables in deployment.tfvars file with the correct information

3. Initialize terraform config:

terraform init

4. [Optional] Preview the plan for deployment:

terraform plan -var-file="deployment.tfvars"

5. Deploy the infrastructure 🚀 :

terraform apply -var-file="deployment.tfvars"

6. [Optional] If you ever need to tear down the cloud infrastructure then you can remove everything and clean-up using the following command:

terraform destroy -var-file="deployment.tfvars"

Deployment

Initial Docker build + Cloud Run deploy

After the cloud infrastructure is setup then the Docker image for the application needs to be built and the service deployed to Cloud Run. The Docker image build and deploy process are grouped into one Cloud Build process so it is streamlined instead of having to build and deploy in two seperate commands.

To deploy to Cloud Run from Cloud Build the Cloud Run Admin and Service Account User roles need to be granted to the Cloud Build service account (source). Open the Cloud Build settings page and set the status of the Cloud Run Admin role to ENABLED.

Next run the following command to submit a Cloud Build job to build the Docker image and deploy to Cloud Run:

# cd src/ # must be in the src/ subdirectory with the main app data
gcloud builds submit --config cloudbuild.yaml

There are Cloud Run configuration and resource parameters defined in the cloud build process. These values provided are generally "good enough" for what the NWM API is doing but if they need to be updated then do so in the cloudbuild.yaml file and run the build command again.

Deploy API Gateway

Create the API:

gcloud api-gateway apis create API_ID --project=PROJECT_ID

Create the API config:

gcloud api-gateway api-configs create CONFIG_ID \
  --api=API_ID --openapi-spec=API_DEFINITION \
  --project=PROJECT_ID --backend-auth-service-account=SERVICE_ACCOUNT_EMAIL

Deploy an API config to a gateway:

gcloud api-gateway gateways create GATEWAY_ID \
  --api=API_ID --api-config=CONFIG_ID \
  --location=GCP_REGION --project=PROJECT_ID

Continuous Deployment

TBD

Example use

export API_KEY=<YOUR-API-KEY>
export NWM_API=<GATEWAY-URL>

Geometry

curl -H "x-api-key: ${API_KEY}" "${NWM_API}/geometry?lon=-121.76&lat=37.70"

Analysis Assimilation data

curl -H "x-api-key: ${API_KEY}" \
  "${NWM_API}/analysis-assim?start_time=2018-09-17&end_time=2023-05-01&comids=15059811&output_format=csv"

Forecast data

curl -H "x-api-key: ${API_KEY}" \
  "${NWM_API}/forecast?forecast_type=long_range&reference_time=2023-05-01&ensemble=0&comids=15059811&output_format=csv"