Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated README to include AWI-006 and modified steps #161

Closed
wants to merge 2 commits into from
Closed

Updated README to include AWI-006 and modified steps #161

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
004d2c6
Updated README.md to include AWI-006 and make further changes.
arpita0911patel May 15, 2024
fc38cce
Updated README.md
arpita0911patel May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 113 additions & 122 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,159 +1,150 @@
# NextGen In A Box (NGIAB)
**NextGen In A Box (NGIAB)**

"NextGen In A Box" (NGIAB) is a containerized version of the NextGen National Water Resources Modeling Framework.
**Run the NextGen National Water Resources Modeling Framework locally with ease.**

NGIAB provides a containerized and user-friendly solution for running the NextGen framework, allowing you to control inputs, configurations, and execution on your local machine.

| | |
| --- | --- |
| ![alt text](https://ciroh.ua.edu/wp-content/uploads/2022/08/CIROHLogo_200x200.png) | Funding for this project was provided by the National Oceanic & Atmospheric Administration (NOAA), awarded to the Cooperative Institute for Research to Operations in Hydrology (CIROH) through the NOAA Cooperative Agreement with The University of Alabama (NA22NWS4320003). |

[![ARM Build and push final image](https://github.com/CIROH-UA/NGIAB-CloudInfra/actions/workflows/docker_image_main_branch.yml/badge.svg)](https://github.com/CIROH-UA/NGIAB-CloudInfra/actions/workflows/docker_image_main_branch.yml)
[![X86 Build and push final image](https://github.com/CIROH-UA/NGIAB-CloudInfra/actions/workflows/docker_image_main_x86.yml/badge.svg)](https://github.com/CIROH-UA/NGIAB-CloudInfra/actions/workflows/docker_image_main_x86.yml)

**Why NextGen In A Box?**

<p align="center">
<img src="https://github.com/CIROH-UA/NGIAB-CloudInfra/assets/54657/1a647024-67f8-489a-9f5e-86437449b6ff" width="300">
</p>
The NextGen Water Resources Modeling Framework (NextGen) is a data-centric framework developed by the NOAA OWP team to enhance the forecasting of flooding and drought, improve water resource management, and protect lives, property, and the environment.

The Cooperative Institute for Research to Operations in Hydrology (CIROH) along with Lynker has developed “NextGen In A Box” - ready-to-run, containerized and cloud-friendly version of NextGen framework, packaged with scripts to help prepare data and get you modeling more quickly. Leveraging open-source technical tools like Git, GitHub, CI/CD, Docker, NextGen In A Box fosters open research practices, enabling transparent and reproducible research outcomes within the NextGen framework.

We are doing a case study : NWM run for Sipsey Fork, Black Warrior River
- We don’t want to run all of CONUS
- We want to run NextGen locally
- We want to have control over inputs / config.
- How can we do it? Answer: NextGen In A Box (NGIAB)

This repository contains :
- **Dockerfile** for running NextGen Framework (docker/Dockerfile*)
- Documentation of how to run the model. (README.md)

## Table of Contents
* [Prerequisites:](#prerequisites-)
+ [Install docker](#install-docker-)
+ [Install WSL on Windows](#Install-WSL-on-Windows-)
+ [Download the input data in "ngen-data" folder from S3 bucket ](#download-the-input-data-in--ngen-data--folder-from-s3-bucket--)
- [Linux & Mac](#linux---mac)
* [Run NextGen-In-A-Box](#run-nextgen-in-a-box)
+ [Clone CloudInfra repo](#clone-cloudinfra-repo)
+ [How to run the model script?](#how-to-run-the-model-script-)
+ [Output of the model script](#output-of-the-model-script)


## Prerequisites:

### Install docker and validate docker is up:
- On *Windows*:
- [Install Docker Desktop on Windows](https://docs.docker.com/desktop/install/windows-install/#install-docker-desktop-on-windows)
- Once docker is installed, start Docker Destop.
- Open Powershell -> right click and `Run as an Administrator`
- Type `docker ps -a` to make sure docker is working.

- On *Mac*:
- [Install docker on Mac](https://docs.docker.com/desktop/install/mac-install/)
- Once docker is installed, start Docker Desktop.
- Open terminal app
- Type `docker ps -a` to make sure docker is working.

- On *Linux*:
- [Install docker on Linux](https://docs.docker.com/desktop/install/linux-install/)
- Follow similar steps as *Mac* for starting Docker and verifying the installation

### Install WSL on Windows:

1. Follow Microsofts latest [instructions](https://learn.microsoft.com/en-us/windows/wsl/install) to install wsl
2. Once this is complete, follow the instructions for linux inside your wsl terminal.


### Download the sample input data in "ngen-data" folder from S3 bucket :

#### Linux ,Mac, WSL(Windows)

```bash
mkdir -p NextGen/ngen-data
cd NextGen/ngen-data
wget --no-parent https://ciroh-ua-ngen-data.s3.us-east-2.amazonaws.com/AWI-004/AWI_09_004.tar.gz
tar -xf AWI_09_004.tar.gz
# to rename your folder
mv AWI_09_004 my_data
```
- **Run NextGen Locally:** Experiment with the framework and customize configurations on your local machine.
- **Control Over Inputs:** Choose specific regions or basins for analysis and modify input data as needed.
- **Simplified Setup:** Utilize Docker containers for effortless deployment, avoiding complex software installations.
- **Open Research Practices:** Promote transparency and reproducibility through open-source tools like Git and GitHub.

### How to Generate Your Own Input Data?
**Case Study:** This repository demonstrates running NWM for Provo River Basin, UT

Follow steps in our [ngen-datastream Repo](https://github.com/CIROH-UA/ngen-datastream/tree/main)
**Repository Contents:**

### Case Study Map for the Sipsey Fork, Black Warrior River, AL
- Dockerfile for running the NextGen Framework
- Guide script (guide.sh) for easy execution
- README with instructions and documentation

![AGU_113060_03W_002](https://github.com/shahab122/NGIAB-CloudInfra/assets/28275758/cc7978da-081c-44ba-8877-0e235b5cca43)
**Getting Started**

## Run NextGen In A Box
**Prerequisites**

### Clone NGIAB-CloudInfra repository
**Windows:**
1. **Install WSL:** Head over to Microsoft's official documentation and follow their comprehensive guide on installing WSL: https://learn.microsoft.com/en-us/windows/wsl/install
2. **Install Docker Desktop:** Begin by downloading and installing Docker Desktop from the official website: https://docs.docker.com/desktop/install/windows-install/#install-docker-desktop-on-windows
3. **Start Docker Desktop:** After installation, launch the Docker Desktop application.
4. **Open WSL as Admin:** Right-click on the WSL icon and select "Run as Administrator".
5. **Verify Installation:** In the WSL window, type the command docker ps -a to check if Docker is running correctly. This command should display a list of Docker containers.

Navigate to NextGen directory and clone the repository using below commands:
**Mac:**
1. **Install Docker Desktop:** Download and install Docker Desktop for Mac from: https://docs.docker.com/desktop/install/mac-install/
2. **Start Docker Desktop:** Launch the Docker Desktop application once the installation is complete.
3. **Open Terminal:** Open the Terminal application on your Mac.
4. **Verify Installation:** Similar to Windows, use the command docker ps -a in the Terminal to verify Docker is functioning as expected.

```bash
cd ../..
git clone https://github.com/CIROH-UA/NGIAB-CloudInfra.git
git checkout main
cd NGIAB-CloudInfra
```
Once you are in *NGIAB-CloudInfra* directory, you should see `guide.sh` in it. Now, we are ready to run the model using that script.
**Linux:**
1. **Install Docker:** The installation process for Linux varies depending on your distribution. Refer to the official documentation for detailed instructions: https://docs.docker.com/desktop/install/linux-install/
2. **Start Docker and Verify:** Follow the same steps as described for Mac to start Docker and verify its installation using the docker ps -a command in the terminal.

- **Input Data:**
- **Download Sample Data:** Use the provided commands to download sample data for the Sipsey Fork case study.
- **To generate your own data:** Refer to the [NGIAB-datapreprocessor](https://github.com/AlabamaWaterInstitute/NGIAB_data_preprocess) for instructions on generating custom input data.
- **To generate your own data and run using NGIAB:** Refer to the [ngen-datastream repository](https://github.com/CIROH-UA/ngen-datastream/tree/main) for instructions on generating custom input data.

### How to run the model script?
This section guides you through downloading and preparing the sample input data for the NextGen In A Box project.

#### WSL, Linux and Mac Steps:
Follow below steps to run `guide.sh` script
**Step 1: Create Project Directory**

- **Linux/Mac:** Open your terminal and go to your desired folder where you want to checkout repo and ngen-data folder and run the following commands:
```bash
./guide.sh
mkdir -p NextGen/ngen-data
```
- The script prompts the user to enter the file path for the input data directory where the forcing and config files are stored.

Run the following command and copy the path value:
```bash
# navigate to the data folder you created earlier
cd NextGen/ngen-data/my_data
pwd
# and copy the path

cd NextGen/ngen-data
```
where <path> is the location of the folder with your data in it.

- The script sets the entered directory as the `HOST_DATA_PATH` variable and uses it to find all the catchment, nexus, and realization files using the `find` command.
- Next, the user is asked whether to run NextGen or exit. If `run_NextGen` is selected, the script pulls the related image from the awiciroh DockerHub, based on the local machine's architecture:
- **WSL (Right click and run as Admin):** Open WSL with administrator privileges and execute:
```bash
cd /mnt/c/Users/<Folder>
```
For Mac with apple silicon (arm architecture), it pulls awiciroh/ciroh-ngen-image:latest.
For x86 machines, it pulls awiciroh/ciroh-ngen-image:latest-x86.

```bash
mkdir -p NextGen/ngen-data
```

- The user is then prompted to select whether they want to run the model in parallel or serial mode.
- If the user selects parallel mode, the script uses the `mpirun` command to run the model and generates a partition file for the NGEN model.
- If the user selects the catchment, nexus, and realization files they want to use.
```bash
cd NextGen/ngen-data
```
**Step 2: Download Sample Data**

Example NGEN run command for parallel mode:
- **Linux/Mac/Windows WSL:** Use wget to download the compressed data file:
```bash
/dmod/bin/partitionGenerator "/ngen/ngen/data/config/catchments.geojson" "/ngen/ngen/data/config/nexus.geojson" "partitions_2.json" "2" '' ''
mpirun -n 2 /dmod/bin/ngen-parallel \
/ngen/ngen/data/config/catchments.geojson "" \
/ngen/ngen/data/config/nexus.geojson "" \
/ngen/ngen/data/config/awi_simplified_realization.json \
/ngen/partitions_2.json
wget --no-parent https://ciroh-ua-ngen-data.s3.us-east-2.amazonaws.com/AWI-006/AWI_16_2853886_006.tar.gz
```
- If the user selects serial mode, the script runs the model directly.

Example NGEN run command for serial mode:
**Step 3: Extract and Rename**

- **All Platforms:** Extract the downloaded file and optionally rename the folder:
```bash
tar -xf AWI_16_2853886_006.tar.gz
```
### Below is Optional: Rename the folder
```bash
/dmod/bin/ngen-serial \
/ngen/ngen/data/config/catchments.geojson "" \
/ngen/ngen/data/config/nexus.geojson "" \
/ngen/ngen/data/config/awi_simplified_realization.json
mv AWI_16_2853886_006 my_data
```
- After the model has finished running, the script prompts the user whether they want to continue.
- If the user selects 1, the script opens an interactive shell.
- If the user selects 2, then the script exits.
Now you have successfully downloaded and prepared the sample input data in the NextGen/ngen-data directory. Remember to replace "my_data" with your preferred folder name if you choose to rename it.

### Output of the model guide script
### Case Study Map for the Provo River Basin, UT

The output files are copied to the `outputs` folder in the '/NextGen/ngen-data/my_data/' directory you created in the first step
![AWI_16_2853886_006](

Using the *flowveldepth.csv files from the 'outputs' folder, the streamflow at the Clear Creek gauge (USGS site ID 02450825) is displayed here. Below is the 'Modelled' vs 'Observed' plot generated in MS Excel after *flowveldepth.csv files are post-processed using Python.
**Running NGIAB**

![image](https://github.com/shahab122/NGIAB-CloudInfra/assets/28275758/58aaf351-8bb5-4b61-9f84-d9dd520053e5)
1. **Clone the Repository:**
Go to the folder created earlier during step #1 above

```bash
cd NextGen
git clone https://github.com/CIROH-UA/NGIAB-CloudInfra.git
```
```bash
cd NGIAB-CloudInfra
```

2. **Run the Guide Script:**
```bash
./guide.sh
```

3. **Follow the prompts:**
- **Input Data Path:** Enter the path to your downloaded or generated input data directory. (e.g NextGen/ngen-data/my_data)
- **Run Mode:** Choose between parallel or serial execution based on your preferences.
The script pulls the related image from the awiciroh DockerHub, based on the local machine's architecture:
```
For Mac with apple silicon (arm architecture), it pulls awiciroh/ciroh-ngen-image:latest.
For x86 machines, it pulls awiciroh/ciroh-ngen-image:latest-x86.
```
Example NGEN run command for parallel mode:
```bash
mpirun -n 10 /dmod/bin/ngen-parallel ./config/wb-2853886_subset.gpkg all ./config/wb-2853886_subset.gpkg all ./config/realization.json /ngen/ngen/data/partitions_10.json
```

Example NGEN run command for serial mode:
```bash
/dmod/bin/ngen-serial ./config/wb-2853886_subset.gpkg all ./config/wb-2853886_subset.gpkg all ./config/realization.json
```
- **Select Files (automatically):** Script selects specific catchment, nexus, and realization files based on input data.
- After the model has finished running, the script prompts the user whether they want to continue.
- If the user selects 1, the script opens an interactive shell.
- If the user selects 2, then the script exits.

**Output:**
- Model outputs will be saved in the outputs folder within your input data directory. (e.g '.../NextGen/ngen-data/my_data/')
- Refer to the provided example for post-processing and visualizing streamflow data.

**Additional Resources:**

- [Next Generation Water Modeling Engine and Framework Prototype](https://github.com/NOAA-OWP/ngen)
- [NGIAB_data_preprocessor](https://github.com/AlabamaWaterInstitute/NGIAB_data_preprocess)
- [ngen-datastream Repository](https://github.com/CIROH-UA/ngen-datastream/tree/main)
Loading