Skip to content

Commit

Permalink
Merge pull request blaylockbk#308 from blaylockbk/reorganize-docs
Browse files Browse the repository at this point in the history
Reorganize docs: spring cleaning 🌷🧹
  • Loading branch information
blaylockbk authored Apr 20, 2024
2 parents f31d06c + cccdaf9 commit 5968774
Show file tree
Hide file tree
Showing 61 changed files with 165 additions and 151 deletions.
12 changes: 7 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,9 @@

</div>

**Herbie** is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. **Its most popular capability is to download HRRR model data.** NWP data in GRIB2 format can be read with xarray+cfgrib. Much of this data is made available through the [NOAA Open Data Dissemination](https://www.noaa.gov/information-technology/open-data-dissemination) (NODD) Program (formerly the Big Data Program) which has made weather data more accessible than ever before.
**Herbie** is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. NWP data is distributed in GRIB2 format and can be read with xarray+cfgrib.

# 📓 [Herbie Documentation](https://herbie.readthedocs.io/)

Herbie helps you discover, download, and read data from:

Expand All @@ -48,7 +50,8 @@ Herbie helps you discover, download, and read data from:
- [Hurricane Analysis And Forecast System (HAFS)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/hafs.html)
- [High Resolution Deterministic Prediction System (HRDPS)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/hrdps.html)

# 📓 [Herbie Documentation](https://herbie.readthedocs.io/)
Much of this data is made available through the [NOAA Open Data Dissemination](https://www.noaa.gov/information-technology/open-data-dissemination) (NODD) program (formerly the Big Data Program) which has made weather data more accessible than ever before.


## Installation

Expand All @@ -75,9 +78,8 @@ conda activate herbie

Alternatively, Herbie is published on PyPI and you can install it with pip, _but_ it requires some dependencies that you will have to install yourself:

- Python 3.8 to 3.11
- Python 3.9+
- cURL
- [Cartopy](https://scitools.org.uk/cartopy/docs/latest/installing.html), which requires GEOS and Proj.
- [cfgrib](https://github.com/ecmwf/cfgrib), which requires eccodes.
- _Optional:_ wgrib2
- _Optional:_ [Carpenter Workshop](https://github.com/blaylockbk/Carpenter_Workshop)
Expand Down Expand Up @@ -191,7 +193,7 @@ If Herbie played an important role in your work, please [tell me about it](https

| | |
| :-: | --------------------------------------------------------------------------------------------------- |
| 👨🏻‍💻 | [Contributing Guide/Disclaimer](https://herbie.readthedocs.io/en/stable/user_guide/disclaimer.html) |
| 👨🏻‍💻 | [Contributing Guide/Disclaimer](https://herbie.readthedocs.io/en/stable/user_guide/housekeeping/disclaimer.html) |
| 💬 | [GitHub Discussions](https://github.com/blaylockbk/Herbie/discussions) |
| 🚑 | [GitHub Issues](https://github.com/blaylockbk/Herbie/issues) |
| 🌐 | [Personal Webpage](http://home.chpc.utah.edu/~u0553130/Brian_Blaylock/home.html) |
Expand Down
5 changes: 4 additions & 1 deletion docs/404.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
:orphan:

404: Page Not Found
^^^^^^^^^^^^^^^^^^^

[Back to Herbie docs](https://herbie.readthedocs.io/)
Back to Herbie docs: https://herbie.readthedocs.io/

.. image:: _static/empty.png
Binary file added docs/_static/empty.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion docs/api_reference/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ Docstring reference for the ``herbie`` module
:maxdepth: 2

core
fast
accessors
fast
latest
wgrib2
misc
14 changes: 14 additions & 0 deletions docs/api_reference/latest.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
=============
herbie.latest
=============

.. toctree::
:maxdepth: 2

.. currentmodule:: herbie.latest

.. autosummary::
:toctree: _autosummary

HerbieLatest
HerbieWait
11 changes: 5 additions & 6 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,18 @@
Herbie: Retrieve NWP Model Data
===============================

**Herbie** is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. **Its most popular capability is to download HRRR model data.** NWP data in the GRIB2 format can be read with xarray+cfgrib.
**Herbie** is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. NWP data is distributed in GRIB2 format and can be read with xarray+cfgrib.

Some models Herbie can retrieve data from include:

- High-Resolution Rapid Refresh (HRRR)
- Rapid Refresh (RAP)
- Global Forecast System (GFS)
- ECMWF open data forecast products (IFS and AIFS)
- National Blend of Models (NBM)
- Rapid Refresh Forecast System - Prototype (RRFS)
- ECMWF open data forecast products (ECMWF)
- Real-Time/Un-Restricted Mesoscale Analysis (RTMA/URMA)
- and others.
- and many others (see :ref:`🗺 Model Tutorials`)

.. toctree::
:maxdepth: 1
Expand Down Expand Up @@ -72,8 +72,7 @@ Specifically, Herbie can
- Download data to your computer.
- Download *subsets* of the data.
- Read the data with xarray.
- Accessors to help with plotting.
- Accessors for picking data.
- Useful xarray accessors (see :ref:`🗃️ Xarray Accessors`)

Using Herbie looks something like this...

Expand Down Expand Up @@ -102,5 +101,5 @@ Using Herbie looks something like this...
H.xarray("TMP:2 m")
More details at :ref:`👨🏻‍💻 Tutorials` in the User Guide.
More details in the :ref:`User Guide`.

24 changes: 0 additions & 24 deletions docs/user_guide/_tutorial_notebooks/index.rst

This file was deleted.

File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Herbie supports **subsetting GRIB2 files by GRIB message**, provided that an _in

Why would you want to subset GRIB2 files? Well, GRIB files provided by operational forecast centers are usually very large because they can contain hundreds of model output variables, and each variable is its own GRIB message. For example, native grid HRRR files can be ~700 MB each! That adds up quick if you need a lot of days and forecasts and all you are interested in surface temperature. Often, you only need some of the data in the file. The size of a single HRRR message is about 1 MB. If you subset the data as you download it, you will save a lot of disk space and improve your data acquisition time by reducing the download time.

![](../_static/diagrams/GRIB2_file_cURL.png)
![](../../_static/diagrams/GRIB2_file_cURL.png)

GRIB2 files are made up of messages or _fields_ that can be extracted instead of downloading the full file.

Expand Down Expand Up @@ -106,7 +106,7 @@ Each GRIB2 file has a unique index file and looks something like this example fr

Each part has meaning as described in the figure

![](../_static/diagrams/index_file_description.png)
![](../../_static/diagrams/index_file_description.png)

ECMWF products use a different pattern for their index files (I believe created with the ecCodes/grib_ls tool). The important thing here is that these also tell us what variable is in each GRIB message and the start and end byte. Here is an example of one of the ECMWF index files:

Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# 📜 History
# 📜 Herbie's History

👨🏻‍🎓 During my PhD at the University of Utah, I created, at the time, the only publicly accessible archive of HRRR data. Over 1,000 research scientists and professionals used that archive. In the latter half of 2020, the HRRR data was made available through the [NOAA Open Data Dissemination Program](https://www.noaa.gov/information-technology/open-data-dissemination) (formerly the NOAA Big Data Program). The Herbie package organizes and expands my original download scripts into a more coherent package with the extended ability to download more than just the HRRR and RAP model data and from different data sources. It will continue to evolve at my own leisure.

> I wrote a paper about archiving HRRR data in the cloud
>
> > Blaylock B., J. Horel and S. Liston, 2017: Cloud Archiving and Data Mining of High-Resolution Rapid Refresh Model Output. Computers and Geosciences. 109, 43-50. <https://doi.org/10.1016/j.cageo.2017.08.005>.
![alt text](../_static/timeline.png)
![alt text](../../_static/timeline.png)

**🌹 What's in a name?** I originally released this package under the name "HRRR-B" because it only dealt with the HRRR dataset, and the "B" was for Brian. I added the ability to access data from many more models. Thus, it was re-branded with the name "Herbie," named after a favorite childhood movie.

![](../_static/Herbie3.png)
![](../../_static/Herbie3.png)
Original file line number Diff line number Diff line change
@@ -1,61 +1,21 @@
# ℹ Model Info

References to information about each model.

> This is a work in progress. Please help with a PR 😄. Please provide links to the model homepages and GitHub links, contact for model developers,
## HRRR: High Resolution Rapid Refresh

- [HRRR Homepage (ESRL)](https://rapidrefresh.noaa.gov/hrrr/ "Details about the HRRR model and graphics for the most recent runs.")
- [GSL-Regional-Model-Forum (GitHub)](https://github.com/NOAA-GSL/GSL-Regional-Model-Forum/discussions/ "Ask the HRRR model developers and the community about the HRRR model.")

> Dowell, D. C., and Coauthors, 2022: The High-Resolution Rapid Refresh (HRRR): An Hourly Updating Convection-Allowing Forecast Model. Part I: Motivation and System Description. Wea. Forecasting, **37**, 1371–1395, <https://doi.org/10.1175/WAF-D-21-0151.1>.
> James, E. P., and Coauthors, 2022: The High-Resolution Rapid Refresh (HRRR): An Hourly Updating Convection-Allowing Forecast Model. Part II: Forecast Performance. Wea. Forecasting, **37**, 1397–1417, <https://doi.org/10.1175/WAF-D-21-0130.1>.
> James, E. P., and S. G. Benjamin, 2017: Observation system experiments with the hourly updating Rapid Refresh model using GSI hybrid ensemble-variational data assimilation. Mon. Wea. Rev., **145**, 2897–2918, <https://doi.org/10.1175/MWR-D-16-0398.1>.

```{mermaid}
gantt
title HRRR Version History
axisFormat %Y
HRRRv1 : 2014-09-30, 2016-08-23
HRRRv2 : after v1, 2018-07-12
HRRRv3 : after v2, 2020-12-02
HRRRv4 :active, after v3 2020-12-02, 2024-11-01
RRFS : after v4, 2026
```


## RAP: Rapid Refresh

ESRL: https://rapidrefresh.noaa.gov/

NCEI: https://www.ncei.noaa.gov/products/weather-climate-models/rapid-refresh-update

AWS: https://registry.opendata.aws/noaa-rap/

> Benjamin, S. G., and Coauthors, 2004: An Hourly Assimilation–Forecast Cycle: The RUC. Mon. Wea. Rev., 132, 495–518, [https://doi.org/10.1175/1520-0493(2004)132<0495:AHACTR>2.0.CO;2](https://doi.org/10.1175/1520-0493(2004)132<0495:AHACTR>2.0.CO;2).
> Benjamin, S. G., and Coauthors, 2016: A North American Hourly Assimilation and Model Forecast Cycle: The Rapid Refresh. Mon. Wea. Rev., **144**, 1669–1694, <https://doi.org/10.1175/MWR-D-15-0242.1>.
```{mermaid}
gantt
title RAP Model Sources
axisFormat %Y
section NCEI
RUC : 2005-01-01, 2012-05-01
RAP : 2012-05-01, 2022-05-01
section Big Data Program
RAP : 2021-02-22, 2022-02-22
```




## GFS:

## GEFS:

## ECMWF:
# HRRR: High Resolution Rapid Refresh

- [HRRR Homepage (ESRL)](https://rapidrefresh.noaa.gov/hrrr/ "Details about the HRRR model and graphics for the most recent runs.")
- [GSL-Regional-Model-Forum (GitHub)](https://github.com/NOAA-GSL/GSL-Regional-Model-Forum/discussions/ "Ask the HRRR model developers and the community about the HRRR model.")

> Dowell, D. C., and Coauthors, 2022: The High-Resolution Rapid Refresh (HRRR): An Hourly Updating Convection-Allowing Forecast Model. Part I: Motivation and System Description. Wea. Forecasting, **37**, 1371–1395, <https://doi.org/10.1175/WAF-D-21-0151.1>.
> James, E. P., and Coauthors, 2022: The High-Resolution Rapid Refresh (HRRR): An Hourly Updating Convection-Allowing Forecast Model. Part II: Forecast Performance. Wea. Forecasting, **37**, 1397–1417, <https://doi.org/10.1175/WAF-D-21-0130.1>.
> James, E. P., and S. G. Benjamin, 2017: Observation system experiments with the hourly updating Rapid Refresh model using GSI hybrid ensemble-variational data assimilation. Mon. Wea. Rev., **145**, 2897–2918, <https://doi.org/10.1175/MWR-D-16-0398.1>.
```{mermaid}
gantt
title HRRR Version History
axisFormat %Y
HRRRv1 : 2014-09-30, 2016-08-23
HRRRv2 : after v1, 2018-07-12
HRRRv3 : after v2, 2020-12-02
HRRRv4 :active, after v3 2020-12-02, 2024-11-01
RRFS : after v4, 2026
```
14 changes: 14 additions & 0 deletions docs/user_guide/background/model-info/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
==============
ℹ Model Info
==============

References to information about each model (links to the model homepages, and GitHub links, contact for model developers, etc.)

**This is a work in progress. Please help with a PR 😄**


.. toctree::
:glob:
:maxdepth: 1

./*
22 changes: 22 additions & 0 deletions docs/user_guide/background/model-info/rap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# RAP: Rapid Refresh

ESRL: https://rapidrefresh.noaa.gov/

NCEI: https://www.ncei.noaa.gov/products/weather-climate-models/rapid-refresh-update

AWS: https://registry.opendata.aws/noaa-rap/

> Benjamin, S. G., and Coauthors, 2004: An Hourly Assimilation–Forecast Cycle: The RUC. Mon. Wea. Rev., 132, 495–518, [https://doi.org/10.1175/1520-0493(2004)132<0495:AHACTR>2.0.CO;2](https://doi.org/10.1175/1520-0493(2004)132<0495:AHACTR>2.0.CO;2).
> Benjamin, S. G., and Coauthors, 2016: A North American Hourly Assimilation and Model Forecast Cycle: The Rapid Refresh. Mon. Wea. Rev., **144**, 1669–1694, <https://doi.org/10.1175/MWR-D-15-0242.1>.
```{mermaid}
gantt
title RAP Model Sources
axisFormat %Y
section NCEI
RUC : 2005-01-01, 2012-05-01
RAP : 2012-05-01, 2022-05-01
section Big Data Program
RAP : 2021-02-22, 2022-02-22
```
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ If Herbie played an important role in your work, please [tell me about it](https

***Suggested Citation***

> Blaylock, B. K. (2022). Herbie: Retrieve Numerical Weather Prediction Model Data (Version 2022.9.0) [Computer software]. <https://doi.org/10.5281/zenodo.4567540>
> Blaylock, B. K. (2024). Herbie: Retrieve Numerical Weather Prediction Model Data (Version 202x.x.x) [Computer software]. <https://doi.org/10.5281/zenodo.4567540>
***Suggested Acknowledgment***

Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# 👨🏻‍🏭 Disclaimer
# 👨🏻‍🏭 Disclaimer & Contributing Guide

Herbie continues to evolve and is distributed _as is_. I do not make any guarantee it will work for you out of the box or is tailored to your needs. Honestly, this package is my playground for developing my python skills and testing new ideas in my free time.

With that said, I am happy to share this project with you. If this is helpful to you in any way, I'm glad. You are welcome to open issues when you see a problem, ask questions in the discussion, submit pull requests if you want to add a new feature or fix a bug.
With that said, I am happy to share this project with you. If this is helpful to you in any way, I'm glad. You are welcome to open issues when you see a problem, ask questions in the discussion, submit pull requests if you want to add a new feature or fix a bug, and improve the documentation. I really appreciate the communities help and have learned a lot from you.

🙋🏻‍♂️ Questions: [GitHub Discussions](https://github.com/blaylockbk/Herbie/discussions)
🚑 Support: [GitHub Issues](https://github.com/blaylockbk/Herbie/issues)

**Thanks for using Herbie, and happy racing 🏁**
\- Brian
*Thanks for using Herbie, and happy racing*
*🏁 Brian*

![](https://raw.githubusercontent.com/blaylockbk/Herbie/master/images/herbie.jpg)

File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,37 +1,45 @@
# 🎨 Style Guide

Herbie is a race car themed Python package.

## Font

Franklin Gothic Heavy

## Colors

### Red : #88211b

<div style="height: 30px; width: 30px; border: 2px solid black; background-color: #88211b;"></div>

- Left racing stripe, half the width of blue racing stripe
- Left racing stripe, half the width of blue racing stripe

### White : #ffffff

<div style="height: 30px; width: 30px; border: 2px solid black; background-color: #ffffff;"></div>

- Font color if background color is Tan.
- (optional) center racing stripe, half the width of blue racing stripe. If omitted, background color should be Tan or White.

### Blue: #0c3576

<div style="height: 30px; width: 30px; border: 2px solid black; background-color: #0c3576;"></div>

- Right racign stripe, twice as wide as red racing strip

### Tan: #d8c89d

<div style="height: 30px; width: 30px; border: 2px solid black; background-color: #d8c89d;"></div>

- Font color if background color is white.

### Black: #000000

<div style="height: 30px; width: 30px; border: 2px solid black; background-color: #000000;"></div>

- Text outline

## Samples

<img src="../_static/logo_new/Herbie-logo.png" style='background-color:transparent;'>
<img src="../_static/logo_new/Herbie-logo-white.png" style='background-color:transparent;'>
<img src="../../_static/logo_new/Herbie-logo.png" style='background-color:transparent;'>
<img src="../../_static/logo_new/Herbie-logo-white.png" style='background-color:transparent;'>
Loading

0 comments on commit 5968774

Please sign in to comment.