Updates to the RnR Test API

.gitignore

@@ -79,3 +79,19 @@ bower_components/
 .grunt/
 src/vendor/
 dist/
+
+# .vscode #
+###########
+.vscode
+
+# ruff/pycache/pytest #
+#######################
+.*_cache
+
+# .venv #
+#########
+.venv
+
+# Build libs #
+##############
+build/

build.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+if ! docker login --username ${GH_USERNAME} --password ${GH_TOKEN} ghcr.io; then
+    echo "Error: Failed to login to ghcr.io. Please check your GH_USERNAME and GH_TOKEN env vars"
+    exit 1
+fi
+
+if ! docker build -t ghcr.io/NOAA-OWP/t-route/t-route-api:${TAG} -f Dockerfile.troute_api .; then
+    echo "Error: Failed to build Docker image. Please check your Dockerfile and build context."
+    exit 1
+fi
+
+if ! docker push ghcr.io/NOAA-OWP/t-route/t-route-api:${TAG}; then
+    echo "Error: Failed to push Docker image. Please check your TAG env var"
+    exit 1
+fi
+
+echo "Successfully built and pushed Docker image with tag: ${TAG}"

doc/api/api_docs.md

@@ -0,0 +1,102 @@
+# T-Route FastAPI
+
+The following doc is meant to explain the T-Route FastAPI implementation using docker compose and shared volumes. 
+
+## Why an API?
+
+T-Route is used in many contexts for hydrological river routing:
+- NGEN 
+- Scientific Python Projects
+- Replace and Route (RnR)
+
+In the latest PR for RnR (https://github.com/NOAA-OWP/hydrovis/pull/865), there is an requirement to run T-Route as a service. This service requires an easy way to dynamically create config files, restart flow from Initial Conditions, and run T-Route. To satisfy this requirement, a FastAPI endpoint was created in `/src/app` along with code to dynamically create t-route endpoints. 
+
+## Why use shared volumes?
+
+Since T-Route is running in a docker container, there has to be a connection between the directories on your machine and the directories within the container. We're sharing the following folders by default:
+- `data/rfc_channel_forcings`
+  - For storing RnR RFC channel domain forcing files (T-Route inputs)
+- `data/rfc_geopackage_data`
+  - For storing HYFeatures gpkg files 
+  - Indexed by the NHD COMID, also called hf_id. Ex: 2930769 is the hf_id for the CAGM7 RFC forecast point. 
+- `data/troute_restart`
+  - For storing TRoute Restart files
+- `data/troute_output`
+  - For outputting results from the T-Route container
+
+## Quickstart
+1. From the Root directory, run:
+```shell
+docker compose --env-file docker/test_troute_api.env -f docker/compose.yaml up --build
+```
+
+This will start the T-Route container and run the API on localhost:8004. To view the API spec and swagger docs, visit localhost:8004/docs
+
+2. Submit a request
+```shell
+curl -X 'GET' \
+  'http://localhost:8004/api/v1/flow_routing/v4/?lid=CAGM7&feature_id=2930769&hy_id=1074884&initial_start=0&start_time=2024-08-24T00%3A00%3A00&num_forecast_days=5' \
+  -H 'accept: application/json'
+```
+
+This curl command is pinging the flow_routing v4 endpoint `api/v1/flow_routing/v4/` with the following metadata:
+```
+lid=CAGM7
+feature_id=2930769
+hy_id=1074884
+initial_start=0
+start_time=2024-08-24T00:00:00
+num_forecast_days=5
+```
+
+which informs T-Route which location folder to look at, what feature ID to read a gpkg from, the HY feature_id where flow is starting, the initial start flow for flow restart, start-time of the routing run, and the number of days to forecast. 
+
+You can also run the following args from the swagger endpoint:
+
+![alt text](swagger_endpoints.png)
+
+The result for a successful routing is a status 200:
+```json
+{
+  "message": "T-Route run successfully",
+  "lid": "CAGM7",
+  "feature_id": "2930769"
+}
+```
+
+and an internal 500 error if there is something wrong. 
+
+## Building and pushing to a container registry
+
+To ensure Replace and Route is using the correct version of T-Route, it is recommended a docker container be built, and then pushed to a registry (Dockerhub, GitHub Container Registry, etc). To do this manually for the GitHub container registry, the following commands should be used within a terminal.
+
+```shell
+docker login --username ${GH_USERNAME} --password ${GH_TOKEN} ghcr.io
+```
+- This command will log the user into the GitHub container registry using their credentials
+
+```shell
+docker build -t ghcr.io/NOAA-OWP/t-route/t-route-api:${TAG} -f docker/Dockerfile.troute_api
+```
+- This command builds the T-Route API container using a defined version `${TAG}`
+
+```shell
+docker push ghcr.io/NOAA-OWP/t-route/t-route-api:${TAG}
+```
+- This commands pushes the built T-Route API container to the NOAA-OWP/t-route container registry
+
+
+The following env variables are used:
+- `${GH_USERNAME}`
+  - your github username
+- `${GH_TOKEN}`
+  - your github access token
+- `${TAG} `
+  - the version tag
+  - ex: 0.0.2
+
+If you want to build this off a forked version, change the container registry (`/NOAA-OWP/t-route/`) to your user accounts container registry.
+
+## Testing:
+
+See `test/api/README.md` for testing information

doc/docker/dockerfile_notebook.md

@@ -0,0 +1,38 @@
+# Dockerfile.Notebook
+
+This document describes the Docker setup for running JupyterLab with mounted volumes for development and analysis.
+
+## Container Overview
+
+The container provides a JupyterLab environment with:
+- Python environment for data analysis
+- Web interface accessible via port 8000
+
+This container is a great way to run examples and integrated tests
+
+## Docker Configuration
+
+### Dockerfile
+The Dockerfile sets up:
+- Base Python environment
+- JupyterLab installation
+- Volume mount points for data and code
+- Port 8000 exposed for web interface
+- Working directory configuration
+
+### Getting Started
+
+Build:
+```bash
+docker build -t troute-notebook -f docker/Dockerfile.notebook .
+```
+
+Run:
+```bash
+docker run -p 8000:8000 troute-notebook
+```
+
+Then, take the URL from the output and put that into your browser. An example one is below:
+```
+http://127.0.0.1:8000/lab?token=<token>
+```

doc/test/integration_tests.md

@@ -0,0 +1,50 @@
+# Integration Tests
+
+This directory details the integration tests within troute
+
+## Quick Start
+
+```bash
+# Run all integration tests
+pytest test/
+
+# Run tests from a specific category
+pytest test/troute-nwm
+pytest test/troute-network/
+```
+
+## Test Categories
+
+### Config Tests (`test/troute-config/`)
+
+**Usage:**
+- Validating existing Troute example configs
+- Testing the Pydantic Config Validations
+
+### Network Tests (`test/troute-network/`)
+
+**Usage:**
+- Using YAML files from `NHD` and `HYFeatures` in order to test network creation
+
+### NWM Tests (`test/troute-nwm/`)
+
+**Usage:**
+- end to end tests of individual functions within the following functions:
+  - `main_v03()`
+  - `main_v04()`
+
+### Routing Tests (`test/troute-routing/`)
+
+**Usage:**
+- Integration tests for NHD and HYFeature util functions
+
+### End to End Examples:
+##### `test/LowerColorado_TX/`
+##### `test/LowerColorado_TX_v4/`
+##### `test/LowerColorado_TX_HYFeatures_v22/`
+
+## Adding New Tests
+
+1. Create new test files in the appropriate category directory
+2. Use the naming convention `test_*.py` for test files
+3. Document any new fixtures in the relevant conftest.py

doc/test/unit_tests.md

@@ -0,0 +1,41 @@
+# Unit Tests
+
+This directory details the unit tests within troute
+
+## Quick Start
+
+```bash
+# Run tests from a specific category
+pytest src/troute-config/
+pytest src/troute-routing/
+pytest test/troute-network/
+```
+
+## Test Categories
+
+### Config Tests (`test/troute-config/`)
+
+**Usage:**
+- Validating existing Troute example configs
+- Testing the Pydantic Config Validations
+
+### Network Tests (`test/troute-network/`)
+
+**Usage:**
+- Using YAML files from `NHD` and `HYFeatures` in order to test network creation
+
+### Routing Tests (`test/troute-routing/`)
+
+**Usage:**
+- Integration tests for NHD and HYFeature util functions
+
+### End to End Examples:
+##### `test/LowerColorado_TX/`
+##### `test/LowerColorado_TX_v4/`
+##### `test/LowerColorado_TX_HYFeatures_v22/`
+
+## Adding New Tests
+
+1. Create new test files in the appropriate category directory
+2. Use the naming convention `test_*.py` for test files
+3. Document any new fixtures in the relevant conftest.py

docker/Dockerfile.compose

@@ -0,0 +1,24 @@
+FROM rockylinux:9.2 as rocky-base
+RUN yum install -y epel-release 
+RUN yum install -y netcdf netcdf-fortran netcdf-fortran-devel netcdf-openmpi
+
+RUN yum install -y git cmake python python-devel pip 
+
+WORKDIR /t-route/
+COPY . /t-route/
+
+RUN ln -s /usr/lib64/gfortran/modules/netcdf.mod /usr/include/openmpi-x86_64/netcdf.mod
+
+ENV PYTHONPATH=/t-route:$PYTHONPATH
+RUN pip install uv==0.2.5
+RUN uv venv
+
+ENV VIRTUAL_ENV=/t-route/.venv
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+RUN uv pip install --no-cache-dir -r /t-route/requirements.txt
+
+RUN ./compiler.sh no-e
+
+WORKDIR "/t-route/src/"
+RUN mkdir -p /t-route/data/troute_restart/

docker/Dockerfile.notebook

@@ -0,0 +1,29 @@
+FROM rockylinux:9.2 as rocky-base
+RUN yum install -y epel-release 
+RUN yum install -y netcdf netcdf-fortran netcdf-fortran-devel netcdf-openmpi
+
+RUN yum install -y git cmake python python-devel pip 
+
+WORKDIR "/t-route/"
+
+COPY . /t-route/
+
+RUN ln -s /usr/lib64/gfortran/modules/netcdf.mod /usr/include/openmpi-x86_64/netcdf.mod
+
+ENV VIRTUAL_ENV=/opt/venv
+RUN python3 -m venv $VIRTUAL_ENV
+
+# Equivalent to source /opt/venv/bin/activate
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+RUN python -m pip install .
+RUN python -m pip install .[jupyter]
+RUN python -m pip install .[test]
+
+RUN ./compiler.sh no-e
+
+EXPOSE 8000
+
+# increase max open files soft limit
+RUN ulimit -n 10000
+CMD ["jupyter", "lab", "--ip=0.0.0.0", "--port=8000", "--no-browser", "--allow-root"]

docker/Dockerfile.troute_api

@@ -0,0 +1,37 @@
+FROM rockylinux:9.2 as rocky-base
+
+RUN yum install -y epel-release 
+RUN yum install -y netcdf netcdf-fortran netcdf-fortran-devel netcdf-openmpi
+
+RUN yum install -y git cmake python python-devel pip 
+
+WORKDIR "/t-route/"
+
+# Copy the contents of the parent directory (repository root) into the container
+COPY . /t-route/
+
+RUN ln -s /usr/lib64/gfortran/modules/netcdf.mod /usr/include/openmpi-x86_64/netcdf.mod
+
+ENV PYTHONPATH=/t-route:$PYTHONPATH
+RUN pip install uv==0.2.5
+RUN uv venv
+
+ENV VIRTUAL_ENV=/t-route/.venv
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+RUN uv pip install --no-cache-dir -r /t-route/requirements.txt
+
+RUN ./compiler.sh no-e
+
+WORKDIR "/t-route/src/"
+RUN mkdir -p /t-route/data/troute_restart/
+
+# Create volume mount points
+RUN mkdir -p ${OUTPUT_VOLUME_TARGET} ${DATA_VOLUME_TARGET} ${CORE_VOLUME_TARGET} /t-route/test
+
+# Set the command to run the application
+CMD sh -c ". /t-route/.venv/bin/activate && uvicorn app.main:app --host 0.0.0.0 --port ${PORT}"
+
+# Add healthcheck
+HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \
+  CMD curl --fail -I http://localhost:${PORT}/health || exit 1

docker/compose.yaml

@@ -0,0 +1,27 @@
+services:
+  troute:
+    build: 
+      context: ..
+      dockerfile: docker/Dockerfile.compose
+    ports:
+      - "${PORT}:${PORT}"
+    volumes:
+      - type: bind
+        source: ${OUTPUT_VOLUME_SOURCE}
+        target: ${OUTPUT_VOLUME_TARGET}
+      - type: bind
+        source: ${DATA_VOLUME_SOURCE}
+        target: ${DATA_VOLUME_TARGET}
+      - type: bind
+        source: ${CORE_VOLUME_SOURCE}
+        target: ${CORE_VOLUME_TARGET}
+      - type: bind
+        source: ${TEST_SOURCE}
+        target: ${TEST_TARGET}
+    command: sh -c ". /t-route/.venv/bin/activate && uvicorn app.main:app --host 0.0.0.0 --port ${PORT}"
+    healthcheck:
+      test: curl --fail -I http://localhost:${PORT}/health || exit 1
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 5s