lance-modis-nrt-global-flood-mcdwd-f3.qmd

---
title: "MODIS NRT Global Flood Product"
author: 
  - "Juan F. Martinez"
  - "Iris Cano"
date: "February 2, 2024"
bibliography: nrt-flood-references.bib
---

# LANCE MODIS Near Real Time (NRT) Global Flood Product

![MODIS NRT FLOOD](https://www.earthdata.nasa.gov/sites/default/files/imported/Flood_mekong.png)[^1]

[^1]: Photo Credit, NASA OESDIS.

## Overview

In this lesson, you will use R to take a closer look at the data from the LANCE MODIS Near Real Time (NRT) Global Flood Product, including learning about what are LANCE and MODIS, and the NRT Flood products available. You will then learn to select, download, and visualize one of the NRT Flood layers...

::: column-margin
::: {.callout-tip style="color: #5a7a2b;"}
## Coding Review

This lesson uses the [R](https://www.r-project.org/about.html) language and environment. R is a popular language used for statistical computing and graphics.
:::
:::

## Learning Objectives

After completing this lesson, you should be able to:

-   Determine what NRT raster data is available by navigating the LANCE website.

-   Read a tile map and select a raster tile to download based on a point of interest.

-   Download near-real-time raster data using the application programming interfaces (APIs) with the GET HTTP request method [@wilsjame].

-   View the downloaded raster data to quickly preview.

-   Classify and place on a map the NRT flood raster data to determine areas with unusual flooding.

## Introduction

Atmospheric circulation, water evaporation, and their interactions with land surfaces can impact a region's rainfall variability. For example, California's winter is correlated to ocean evaporation near the west coast and eastern North Pacific, and ocean evaporation is a strong factor in increased flooding in the region [@wei2016]. Additionally, drought in the region is associated to high pressure systems off the U.S. west coast, with studies finding that the high pressure system is linked to the Pacific sea surface temperature anomalies, and exacerbated by high evaporation over land due to high temperatures.

It is critical to understand the water cycle and how flooding events develop, particularly as climate change intensifies extreme weather events, because the impacts of flooding can be a risk to human life, and can disrupt infrastructure, agriculture, and natural habitats.

![](https://raw.githubusercontent.com/ciesin-geospatial/TOPSTSCHOOL-module-1-water/main/images/FL_Flood_Aerials-8641.jpg)[^2]

[^2]: Photo Credit, California Department of Water Resources.

The **MODIS/Aqua+Terra Global Flood Product L3 Near Real Time (NRT) 250m Global Flood Product (MCDWD_L3_NRT) (beta)** provides daily maps of flooding globally. The product is provided over 3 compositing periods (1-day, 2-day, and 3-day) to minimize the impact of clouds and more rigorously identify flood water (the best composite will depend on the cloudiness for a particular event) [@lin2019]

## What are MODIS and LANCE?

### MODIS

The Moderate Resolution Imaging Spectroradiometer (MODIS) is a NASA Earth Observing System (EOS) satellite-based sensor system that creates data products including land surface temperatures, land surface reflectance, radiances, clouds, aerosols, water vapor, active fire, snow cover, sea ice measurements, and other factor information. The MODIS Near Real-Time (NRT) data includes the Flood product which is a daily \~250 m resolution product showing flood and surface water detected from the twice-daily overpass of the MODIS optical sensors.

The satellite data is readily available shortly after it is acquired by the MODIS instrument on board the Terra and Aqua satellites. This space-based instrument distinguishes 36 spectral bands and groups of wavelengths which helps map the extent of snow and ice caused by winter storms and frigid temperatures. Initially, the water-detecting algorithm is applied to both MODIS observations (Terra and Aqua). Due to cloud and terrain shadows create false positives.

<img src="https://modis.gsfc.nasa.gov/about/images/modisComponents.jpg" alt="MODIS Components"/>[^3]

[^3]: Photo Credit, NASA GSFC.

To minimize errors, the product is generated with three different ***compositing periods (1-day, 2-day, and 3-day)*** to compare results and decide which product has better coverage for the event. Further, they have to differentiate floods from expected surface water through the use of MODIS Land Water Mask (MOD44W), which uses a decision tree classifier trained with MODIS data to produce a global water mask [@carroll2016].

MODIS adoption aimed to surpass barriers related to satellite data, including cost, delivery timelines, limited formats, and the need for technical expertise. The transition to GFIMS establishes an operational system at FAO, ensuring continuity in meeting NASA data-user needs [@lin2019].

### LANCE

The **Land, Atmosphere Near Real-time Capability for EOS (LANCE)** is a NASA initiative that provides near real-time access to satellite data, including MODIS. It allows users to access the latest data within a few hours of satellite overpass, enabling rapid responses to environmental events such as floods. LANCE is particularly valuable for emergency response teams and researchers who require up-to-date information for monitoring and assessing natural disasters [@LANCE2024].

LANCE reduces processing time, allowing for timely computation. Users access the data through platforms like Web Map Service (WMS) and Web Coverage Service (WCS), enabling visualization and analysis for informed decision-making. This NRT approach enhances the speed and accessibility of critical information on vegetation conditions [@zhang2022].

### MODIS NRT Flood MCDWD Data Products

::: column-margin
::: {.callout-tip style="color: #5a7a2b;"}
## Data Information

The main landing pages for the MODIS NRT Global Flood Product:

[NASA EARTHDATA](https://www.earthdata.nasa.gov/learn/find-data/near-real-time/modis-nrt-global-flood-product)

[NASA CRM SEARCH](https://cmr.earthdata.nasa.gov/search/concepts/C2018599131-LANCEMODIS.html)

[MODIS NRT Global Flood Product User Guide](https://www.earthdata.nasa.gov/s3fs-public/2023-01/MCDWD_UserGuide_RevC.pdf)
:::
:::

The ***MODIS/Terra+Aqua Combined MODIS Water Detection (MCDWD)*** algorithm is tailor-made for detecting water bodies using MODIS data obtained from both the Terra and Aqua satellites. This algorithm employs various bands and spectral information to effectively identify and categorize water bodies. This enhances the accuracy and reliability of the flood product generated [@slayback2023].

The MODIS Near Real-Time (NRT) Flood dataset offers multiple products, each accompanied by corresponding layers. The specific layers depend on the temporal aggregation:

***MCDWD_F1_L3_NRT (1-Day product)*** This product type is the most basic level and provides binary information about water occurrence. Pixels are classified as either containing water or not, offering a simple way to identify flooded areas.

***MCDWD_F1CS_L3_NRT (1-Day CS)***: F1CS has a cloud shadow mask applied on the version of the MCDWD_F1_L3_NRT product

***MCDWD_F2_L3_NRT (2-Day)***: F2 provides additional information by categorizing water occurrences into three categories: no water, low-confidence water, and high-confidence water. This classification allows for a more nuanced understanding of the extent of the flood and its associated confidence levels.

***MCDWD_F3_L3_NRT (3-Day)***: The F3 product, based on the MCDWD (MODIS/Terra+Aqua Combined MODIS Water Detection) is an algorithm that further refines flood mapping by adding additional spectral information. These results create a more accurate representation of water bodies and flooded areas [@slayback2023].

## Exploring the Data

For this exercise, we will be using the MCDWD L3 F3 product: [LANCE NRT Flood](https://nrt3.modaps.eosdis.nasa.gov/archive/allData/61/MCDWD_L3_F3_NRT/)

First, install and load the R packages required for this exercise:

```{r eval=FALSE}
packages_to_check <- c("stars", "httr", "jsonlite", "tmap", "basemaps", "sp", "sf")

# Check and install packages
for (package_name in packages_to_check) {
  if (!package_name %in% rownames(installed.packages())) {
    install.packages(package_name)
    cat(paste("Package", package_name, "installed.\n"))
  } else {
    cat(paste("Package", package_name, "is already installed.\n"))
  }
  library(package_name, character.only = TRUE)
}

#in case tmap does not install
#remotes::install_github('r-tmap/tmap')
```

::: column-margin
::: {.callout-tip style="color: #5a7a2b;"}
## Coding Review

This lesson uses the [stars](https://r-spatial.github.io/stars/), [httr](https://httr.r-lib.org/), [jsonlite](https://cran.r-project.org/package=jsonlite), [tmap](https://cran.r-project.org/package=tmap), and [basemaps](https://cran.r-project.org/package=basemaps) packages. If you'd like to learn more about the functions used in this lesson you can use the help guides on their package websites.
:::
:::

#### Check what days are available for the MCDWD L3 F3 product by going to this link: [LANCE NRT Primary Server](https://nrt3.modaps.eosdis.nasa.gov/archive/allData/61/MCDWD_L3_F3_NRT) or [Secondary](https://nrt4.modaps.eosdis.nasa.gov/archive/allData/61/MCDWD_L3_F3_NRT).

Based on availability, edit the year_day variable YYYY-DD. Example: '2022-01'

```{r}

#add the year and date you want to search for (YYYY-DD, 2022-01)
year_day <- '2024-099'

```

#### Determine tiles of interest:

[MODIS NRT Tile Map](https://www.earthdata.nasa.gov/s3fs-public/2023-01/MCDWD_GlobalTileMapHoriz_Website_865x2250.jpg?VersionId=lOQ_j47U8T.j7UUqibD7SM63FVHjM_V5)

<img src="https://www.earthdata.nasa.gov/s3fs-public/2023-01/MCDWD_GlobalTileMapHoriz_Website_865x2250.jpg?VersionId=lOQ_j47U8T.j7UUqibD7SM63FVHjM_V5" alt="A global map with labeled tiles" width="100%"/>

Based on availability, edit the tile_code variable:

```{r}
#add tile code from the map above (written as h00v00)
tile_code <- 'h05v05'
```

This is the NRT Flood F3 (MCDWD_L3_F3) API URL:

```{r}
# Primary Server
API_URL <- paste0('https://nrt3.modaps.eosdis.nasa.gov/api/v2/content/details?products=MCDWD_L3_F3_NRT&archiveSets=61&temporalRanges=')

# if the primary server is down, the secondary server may be available:
#API_URL <- paste0('https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/details?products=MCDWD_L3_F3_NRT&archiveSets=61&temporalRanges=')
```

We can combine the API URL above with the year_day provided and print the available datasets:

```{r}
#pasting together URL and year_day
url <- paste0(API_URL, year_day)
print(url)
```

## Loading the Data

NASA earthdata access requires a nasa username and token. [How to generate User Token](https://wiki.earthdata.nasa.gov/display/EL/How+to+Generate+a+User+Token).

Access the NASA Earthdata with the GET function. Replace `USER_TOKEN` with the token you generated:

```{r, eval=FALSE}

api_key <- readLines("lance-modis-nrt-key.txt")
# Convert the API key to a string
api_key <- toString(api_key)
#"Bearer USER_TOKEN"
api_key <- paste("Bearer ", api_key)

# Define headers (include your token)
headers <- c(
  Authorization = api_key
)

# Make GET request
response <- httr::GET(url, httr::add_headers(.headers=headers))
```

Check the response status from the GET function:

```{r}
response

```

Out of the response from the server, we'll check if the response was a success with `if (http_status(response)$category == "Success")`. If this statement is true, then the content will be assigned to the variable `data` in JSON format, which is then parsed to a data frame using `data_parsed <- jsonlite::fromJSON(data)`. The data frame contains `data_parsed$content`, a column with content. We filter the content by tile code using the command `content_items <- data_parsed$content[grepl(tile_code, data_parsed$content$name, ignore.case = TRUE), ]` and add the results to a data frame.

```{r}
if (httr::http_status(response)$category == "Success") {
  # Read the JSON content into a data frame
  df <- httr::content(response, as = "parsed", simplifyVector = TRUE)
  df <- df$content
} else {
  print("Request failed with status code", httr::http_status(response)$status_code)
}
names(df)
```

Search the Data Frame and subset the rows that contain `tile_code` in the `downloadsLink` column.

```{r}
# Subset rows where the downloadsLink column contains the tile code
subset_df <- df[grepl(tile_code, df$downloadsLink), ]

subset_df
```

If there is only 1 row, select the string 'downloadsLink' column:

```{r}

download_link <- subset_df$downloadsLink
print(download_link)
```

## Read the Data

Use the "read_stars()" function from the "stars" R Library to read the geoTiff raster.<br> The raster is assigned to the "raster_df" variable:

```{r}
raster_df <- stars::read_stars(download_link)
```

Set the Coordinate Reference System (CRS) to a new raster "my_raster" to plot it in a map. For example, Web Mercator EPSG:3857. Use the st_transform() function:

```{r}
raster_df <- sf::st_transform(raster_df, 3857)
```

Now the raster data should be stored as a variable "my_raster" with the CRS set to Web Mercator EPSG:3857

## Visualizing NRT Flood Data

Plot the raster to quickly view it:

```{r}
plot(raster_df, axes = TRUE)
```

### Create NRT Flood Plot with Classification

Refer to the [MODIS NRT Global Flood Product User Guide](https://www.earthdata.nasa.gov/s3fs-public/2023-01/MCDWD_UserGuide_RevC.pdf) for more information.

NRT Flood data has 5 classifications:

| Code | Definition          |
|:----:|:--------------------|
|  0   | No Water            |
|  1   | Surface Water       |
|  2   | Recurring flood[^4] |
|  3   | Flood (unusual)     |
| 255  | Insufficient data   |

[^4]: Value 2 (Recurring flood) is not populated in the beta release.

To view the data in this classification, we'll need to create a classified legend; however, the NRT Flood data is stored in decimal numbers (aka floating-point). Create class breaks dividing the data by the following breaks, and corresponding colors and labels:

```{r}
class_breaks <- c(-Inf, 0.1, 1.1, 2.1, 3.1)
colors <- c( "gray", "blue", "yellow", "red")

labels = c("0: No Water", "1: Surface Water", "2: Recurring flood", "3: Flood (unusual)")

```

Add a title for the plot that includes the year, day of year, and tile code:

```{r}
title = paste("Near Real-Time Flood F3", year_day, tile_code)
```

### Generate a basemap from Esri Imagery

To generate a basemap that shows the location of our raster, we must first create a bounding box to match `raster_df`.

```{r}
bbox <-  sf::st_bbox(raster_df)
```

The `basemap_stars()` function from the `stars` library allows us to access Esri imagery layers. We choose "world_imagery" as our background and assign it to the object `bm_m`.

```{r}
bm_m <- basemaps::basemap_stars(bbox, map_service = "esri", map_type = "world_imagery")
```

The `st_rgb` function lets us turn the RGB stars item into a single image

```{r}
bm_m <- stars::st_rgb(bm_m)
```

### Plot basemap and NRT Flood data

Generate a plot from the tmap library using the `tm_shape()` function. We will plot the basemap and the raster_df items.

```{r}
## tmap mode set to "plot"
tmap::tmap_mode("plot")

## tmap mode can also be set to "view"
#tmap_mode("view")

#create an object the plots the basemap and the NRT flood raster
#with the tmap library, call the tm_shape() function for the basemap
tm_plot <- tmap::tm_shape(bm_m)+
  tmap::tm_raster()+
  #create a new tmap shape for the NRT flood raster with style as "cat," meaning categorical.
  tmap::tm_shape(raster_df, style="cat")+
  #add the classification styling to the raster
  tmap::tm_raster( palette = c(colors),
  title = title, 
  breaks = class_breaks,
  labels = labels )+
  #style the plot
  tmap::tm_layout(legend.outside = TRUE) +
  tmap::tm_graticules(lines=FALSE)
 
```

View the plot:

```{r}
tm_plot
```

To observe a location closer, we can create a new bounding box using latitude and longitude values, clip the data and replot on a map.

Select from the map four corners in degrees including North/South, East/West information. We use these coordinates to create a matrix of points that represent four corners:

```{r}
# Define the NWES coordinates
north <- 40 #negative values are used for South
south <- 36 #negative values are used for south
west <- -124 #negative numbers are used for West
east <- -120 #negative numbers are used for West

# Create a matrix with the corner coordinates
corners <- matrix(c(west, south, east, south, east, north, west, north), ncol = 2, byrow = TRUE)
```

We can create a bounding box from these corners. The corners are first placed in a Geographic Coordinate System (GCS) WGS84:

```{r}
# Create a SpatialPolygons object representing a bounding box
bbox_subset <- sp::SpatialPolygons(list(sp::Polygons(list(sp::Polygon(corners)), ID = "bbox")), proj4string = sp::CRS("+proj=longlat +datum=WGS84"))
```

Finally, these points are projected to a Projected Coordinate System (PCS) 3857 to match our raster:

```{r}

# Transform to projected coordinate system (e.g., EPSG:3857)
bbox_subset <- sp::spTransform(bbox_subset, sp::CRS("+init=epsg:3857"))
bbox_subset <- sf::st_bbox(bbox_subset)

```

We redo the same process as above to plot the data with a basemap but with a different bounding box:

```{r}

#generate a new basemap
bm_m <- basemaps::basemap_stars(bbox_subset, map_service = "esri", map_type = "world_imagery")
#combine RGB bands of the bsemap
bm_m <- stars::st_rgb(bm_m)
```

```{r}
## tmap mode set to "plot"
tmap::tmap_mode("plot")

## tmap mode can also be set to "view"
#tmap_mode("view")

#create an object the plots the basemap and the NRT flood raster
#with the tmap library, call the tm_shape() function for the basemap
tm_plot <- tmap::tm_shape(bm_m)+
  tmap::tm_raster()+
  #create a new tmap shape for the NRT flood raster with style as "cat," meaning categorical.
  tmap::tm_shape(raster_df, style="cat")+
  #add the classification styling to the raster
  tmap::tm_raster( palette = c(colors),
  title = title, 
  breaks = class_breaks,
  labels = labels )+
  #style the plot
  tmap::tm_layout(legend.outside = TRUE) +
  tmap::tm_graticules(lines=FALSE)
 
#View the plot:
tm_plot
```
The NRT flood data can be compared to the Cropland Data Layer (CDL), which provides agricultural categories based on the Farm Service Agency (FSA) Common Land Unit (CLU) Program and is produced by the U.S. Department of Agriculture National Agricultural Statistics Service [@nass2023], to quantify how many acres of crop lands may be affected by the flooding identified in the NRT Flood data. We can access the dataset by using the [Cropland Collaborative Research Outcomes System (CroplandCROS)](https://croplandcros.scinet.usda.gov/) ArcGIS REST Services. 

For this part of the excercise, we will use the package *terra*, discussed in the previous lesson, and the function `rast()` to import the CDL service.

```{r}
#Read CDL GeoTIFF
cdl<- stars::read_stars("2022_30m_cdls.tif")

```

```{r}
Create boundaing box 
bbox_5070 <- sf::st_as_sfc(bbox_subset)
bbox_5070 <- sf::st_transform(bbox_5070, 5070)
cdl <- cdl[bbox_5070]
plot(cdl, downsample = 50)
```

```{r}

flood_areas <- raster_df[raster_df == 3]
flood_areas <- sf::st_transform(flood_areas, 5070)
cdl_flooded <- cdl[flood_areas == 3]

```

```{r}
flood_areas
```
```{r}
## tmap mode set to "plot"
tmap::tmap_mode("plot")

## tmap mode can also be set to "view"
#tmap_mode("view")

#create an object the plots the basemap and the NRT flood raster
#with the tmap library, call the tm_shape() function for the basemap
tm_plot <- tmap::tm_shape(bm_m, raster.downsample = 10)+
  tmap::tm_raster( )+
  tmap::tm_shape(cdl_flooded, raster.downsample = 10)+
  #add the classification styling to the raster
  tmap::tm_raster( title = title)+
  #style the plot
  tmap::tm_layout(legend.outside = TRUE) +
  tmap::tm_graticules(lines=FALSE)
```

```{r}
tm_plot
```

```{r}
# Aggregate cropland classes within flood areas
cropland_counts <- aggregate(cdl, by = flood_areas, FUN = sum)


# Print cropland counts
print(cropland_counts)
```

Congratulations! Now you should be able to:

-   Navigate the LANCE data website and determine what data is available.

-   Select a tile and date to download NRT data.

-   Create a GET HTTP request to download near-real-time data.

-   Plot on a map and classify raster data to determine areas with unusual flooding.

## Lesson 4

In this lesson we explored the LANCE MODIS Near-Rear-Time (NRT) Flood dataset. In our next lesson we will think of water at the local level and focuse on New Nork State School Drinking Water datasets .

[Lesson 4: New York Sate School Water Quality: Exposure to Lead](https://ciesin-geospatial.github.io/TOPSTSCHOOL-module-1-water/exposure_to_lead_in_schools_nys.html){.btn .btn-primary .btn role="button"}