added integration tests

doc/docker/dockerfile_notebook.md

@@ -0,0 +1,38 @@
+# Dockerfile.Notebook
+
+This document describes the Docker setup for running JupyterLab with mounted volumes for development and analysis.
+
+## Container Overview
+
+The container provides a JupyterLab environment with:
+- Python environment for data analysis
+- Web interface accessible via port 8000
+
+This container is a great way to run examples and integrated tests
+
+## Docker Configuration
+
+### Dockerfile
+The Dockerfile sets up:
+- Base Python environment
+- JupyterLab installation
+- Volume mount points for data and code
+- Port 8000 exposed for web interface
+- Working directory configuration
+
+### Getting Started
+
+Build:
+```bash
+docker build -t troute-notebook -f docker/Dockerfile.notebook .
+```
+
+Run:
+```bash
+docker run -p 8000:8000 troute-notebook
+```
+
+Then, take the URL from the output and put that into your browser. An example one is below:
+```
+http://127.0.0.1:8000/lab?token=<token>
+```

docker/Dockerfile.notebook

@@ -0,0 +1,29 @@
+FROM rockylinux:9.2 as rocky-base
+RUN yum install -y epel-release 
+RUN yum install -y netcdf netcdf-fortran netcdf-fortran-devel netcdf-openmpi
+
+RUN yum install -y git cmake python python-devel pip 
+
+WORKDIR "/t-route/"
+
+COPY . /t-route/
+
+RUN ln -s /usr/lib64/gfortran/modules/netcdf.mod /usr/include/openmpi-x86_64/netcdf.mod
+
+ENV VIRTUAL_ENV=/opt/venv
+RUN python3 -m venv $VIRTUAL_ENV
+
+# Equivalent to source /opt/venv/bin/activate
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+RUN python -m pip install .
+RUN python -m pip install .[jupyter]
+RUN python -m pip install .[test]
+
+RUN ./compiler.sh no-e
+
+EXPOSE 8000
+
+# increase max open files soft limit
+RUN ulimit -n 10000
+CMD ["jupyter", "lab", "--ip=0.0.0.0", "--port=8000", "--no-browser", "--allow-root"]

pyproject.toml

@@ -0,0 +1,28 @@
+[project]
+name = "troute_project"
+authors = [
+    {name = "DongHa Kim", email = "dongha.kim@noaa.gov"},
+    {name = "Sean Horvath", email = "sean.horvath@noaa.gov"},
+    {name = "Amin Torabi", email = "amin.torabi@noaa.gov"},
+    {name = "Zach Jurgen", email = "jurgen.zach@noaa.gov"},
+    {name = "Austin Raney", email = "austin.raney@noaa.gov"},
+]
+dynamic = ["version", "dependencies"]
+
+[tool.setuptools.dynamic]
+dependencies = {file = ["requirements.txt"]}
+
+[project.optional-dependencies]
+test = [
+    "pytest==8.3.2",
+    "bmipy==2.0.0",
+]
+
+jupyter = [
+    "contextily==1.6.0",
+    "matplotlib>=3.7.0,<3.8.0",  # More stable version range
+    "ipykernel>=6.29.0,<7.0.0",
+    "jupyterlab>=3.6.7,<4.0.0",
+    "xarray>=2024.1.1",
+    "matplotlib-inline>=0.1.6"  # Add explicit version
+]

test/README.md

@@ -0,0 +1,70 @@
+# Test Directory Structure
+
+This directory is meant for the following purposes:
+- Integration testing of individual functions/class objects within T-Route
+- Hosting example data to run full-length test runs
+
+## Quick Start
+
+```bash
+# Run all tests
+pytest test/
+
+# Run tests from a specific category
+pytest test/troute-config/
+pytest test/troute-network/
+pytest test/troute-nwm/
+pytest test/troute-bmi/
+```
+
+## Test Categories
+
+### Config Tests (`test/troute-config/`)
+
+**Usage:**
+- Validating existing Troute example configs
+- Testing the Pydantic Config Validations
+
+### Network Tests (`test/troute-network/`)
+
+**Usage:**
+- Using YAML files from `NHD` and `HYFeatures` in order to test network creation
+
+### NWM Tests (`test/troute-nwm/`)
+
+**Usage:**
+- end to end tests of individual functions within the following functions:
+  - `main_v03()`
+  - `main_v04()`
+
+### BMI Tests (`test/troute-bmi/`)
+
+**Usage:**
+- Integration tests for the BMI NWM implementation of T-Route
+
+### End to End Examples:
+##### `test/LowerColorado_TX/`
+##### `test/LowerColorado_TX_v4/`
+##### `test/unit_test_hyfeature/`
+
+## Best Practices
+
+1. Always use the appropriate fixture for your test case to minimize setup code
+2. Use `temporarily_change_dir` context manager when working with paths to ensure you're able to run integration tests successfully
+3. Check fixture contents before using them in tests
+4. Use fixture combinations when testing complex interactions
+
+## Adding New Fixtures
+
+When adding new fixtures:
+1. Follow the existing naming convention
+2. Document the fixture structure and purpose
+3. Include example usage
+4. Add any necessary dependencies
+5. Update this README with new end to end test docs or examples
+
+## Adding New Tests
+
+1. Create new test files in the appropriate category directory
+2. Use the naming convention `test_*.py` for test files
+3. Document any new fixtures in the relevant conftest.py

test/__init__.py

@@ -0,0 +1,41 @@
+import os
+from contextlib import contextmanager
+from pathlib import Path
+
+
+@contextmanager
+def temporarily_change_dir(path: Path):
+    """Temporarily changes the current working directory
+
+    This context manager changes the current working directory to the specified path,
+    yields control back to the caller, and then changes back to the original directory
+    when exiting the context
+
+    Parameters
+    ----------
+    path : Path
+        The path to temporarily change the current working directory to
+
+    Yields
+    ------
+    None
+    """
+    original_cwd = Path.cwd()
+    if original_cwd != path:
+        os.chdir(path)
+    try:
+        yield
+    finally:
+        if original_cwd != path:
+            os.chdir(original_cwd)
+
+
+def find_cwd(path=None) -> Path:
+    if path is not None:
+        cwd = path
+    else:
+        cwd = Path.cwd()
+    if "test" in str(cwd):
+        while "test" in str(cwd):
+            cwd = cwd.parent
+    return cwd

test/api/README.md

@@ -0,0 +1,54 @@
+## Testing:
+
+### Set up Using a Dockerfile:
+
+To build the T-Route api from a dockerfile, you can use the following commands from the t-route root repo:
+
+```shell
+docker build -t troute_api -f docker/Dockerfile.troute_api .
+```
+
+```shell
+docker run -p 8000:8000   \
+--env-file docker/test_troute_api.env  \
+-v ${OUTPUT_VOLUME_SOURCE}:${OUTPUT_VOLUME_TARGET}  \
+-v ${DATA_VOLUME_SOURCE}:${DATA_VOLUME_TARGET}  \
+-v ${CORE_VOLUME_SOURCE}:${CORE_VOLUME_TARGET}  \
+-v ${TEST_SOURCE}:${TEST_TARGET}   troute_api
+```
+
+### Set up Docker Compose:
+
+Docker Compose uses a YAML file to configure docker containers and execution. To install compose, you can follow the examples on docker's docs: https://docs.docker.com/compose/install/linux/
+
+To run compose, you can use the following command from the root directory:
+
+```shell
+docker compose --env-file docker/test_troute_api.env -f docker/compose.yaml up --build
+```
+
+if you want to build a compose container to mimic what is used in RnR, you can run the following steps
+```shell
+docker compose --env-file docker/rnr_compose.env -f docker/compose.yaml up --build
+```
+
+#### Testing the RnR endpoint in the API: 
+The following folder contains data files that are to be used to test the T-Route FastAPI code within src/app
+
+1. Follow the steps above to build/run either the docker container, or docker compose
+2. visit `localhost:8000/docs` in your browser
+3. Enter the following parameters into the `/api/v1/flow_routing/v4` endpoint
+- lid=CAGM7
+- feature_id=2930769
+- hy_id=1074884
+- initial_start=0
+- start_time=2024-08-24T00:00:00
+- num_forecast_days=5
+4. Click execute
+5. A Status 201 code means the run ran, and test/api/data/troute_output will be populated in the `{lid}/` folder
+
+#### Testing the LowerColorado test cases through docker compose:
+1. Follow the steps above to build/run either the docker container, or docker compose
+2. visit `localhost:8000/docs` in your browser
+3. Execute the `/api/v1/flow_routing/v4/tests/LowerColorado` endpoint using the default parameter file path for LowerColorado_TX_v4 
+4. A Status 201 code means the run ran, and the defined yaml output will be populated

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411261800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411261800
+1068225,1919.8821989376002

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411270000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411270000
+1068225,1919.8821989376002

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411270600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411270600
+1068225,1885.9019830272

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411271200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411271200
+1068225,1885.9019830272

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411271800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411271800
+1068225,1851.9217671168003

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411280000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411280000
+1068225,1817.9415512064002

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411280600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411280600
+1068225,1783.961335296

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411281200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411281200
+1068225,1752.8128040448

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411281800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411281800
+1068225,1752.8128040448

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411290000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411290000
+1068225,1721.6642727936

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411290600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411290600
+1068225,1721.6642727936

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411291200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411291200
+1068225,1721.6642727936

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411291800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411291800
+1068225,1690.5157415424

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411300000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411300000
+1068225,1690.5157415424

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411300600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411300600
+1068225,1690.5157415424

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411301200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411301200
+1068225,1659.3672102912

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202411301800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202411301800
+1068225,1659.3672102912

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412010000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412010000
+1068225,1659.3672102912

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412010600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412010600
+1068225,1659.3672102912

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412011200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412011200
+1068225,1659.3672102912

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412011800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412011800
+1068225,1659.3672102912

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412020000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412020000
+1068225,1628.21867904

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412020600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412020600
+1068225,1628.21867904

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412021200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412021200
+1068225,1628.21867904

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412021800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412021800
+1068225,1628.21867904

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412030000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412030000
+1068225,1628.21867904

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412030600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412030600
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412031200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412031200
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412031800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412031800
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412040000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412040000
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412040600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412040600
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412041200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412041200
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412041800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412041800
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412050000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412050000
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412050600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412050600
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412051200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412051200
+1068225,1591.4067784704

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412051800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412051800
+1068225,1551.7631932416

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412060000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412060000
+1068225,1551.7631932416

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412060600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412060600
+1068225,1551.7631932416

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412061200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412061200
+1068225,1551.7631932416

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412061800.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412061800
+1068225,1551.7631932416

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412070000.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412070000
+1068225,1514.9512926720001

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412070600.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412070600
+1068225,1514.9512926720001

test/api/data/rfc_channel_forcings/v2.2/CAGM7/202412071200.CHRTOUT_DOMAIN1.csv

@@ -0,0 +1,2 @@
+feature_id,202412071200
+1068225,1514.9512926720001