From d99cb91f80704d297643e8568c0f3c11a56f27b7 Mon Sep 17 00:00:00 2001 From: Andrew Bennett Date: Mon, 26 Jul 2021 17:13:16 -0700 Subject: [PATCH 1/3] Update environment.yml --- binder/environment.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/binder/environment.yml b/binder/environment.yml index fd21786..8b9be2b 100644 --- a/binder/environment.yml +++ b/binder/environment.yml @@ -3,6 +3,7 @@ channels: - conda-forge dependencies: - python>=3.5 + - bmorph>= 0.0.3 - xarray - pandas - bottleneck From 26bcda8220e856b34f9dd9b05f5cefd387ee7076 Mon Sep 17 00:00:00 2001 From: steinadio Date: Thu, 19 Aug 2021 10:08:00 -0700 Subject: [PATCH 2/3] Modified the spelling and sentence structure of some of the commentary to clarify meaning --- tutorial/bmorph_tutorial.ipynb | 284 +++++++++++++++++---------------- 1 file changed, 143 insertions(+), 141 deletions(-) diff --git a/tutorial/bmorph_tutorial.ipynb b/tutorial/bmorph_tutorial.ipynb index b3a851c..dfe3518 100644 --- a/tutorial/bmorph_tutorial.ipynb +++ b/tutorial/bmorph_tutorial.ipynb @@ -2,7 +2,6 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, "source": [ "# bmorph Tutorial: Getting your first bias corrections\n", "This notebook demonstrates how to setup data for and bias correct it through bmorph, it contains the same information as the [tutorial](bmorph_tutorial.rst) page.\n", @@ -10,7 +9,7 @@ "\n", "* Independent Bias Correction: Univariate (IBC_U) : IBC_U is the traditional bias correction method. This method can only be performed at sites with reference data.\n", "* Independent Bias Correction: Conditioned (IBC_C) : IBC_C allows for correcting for specific biases that are process-dependent. This method can only be performed at sites with reference data.\n", - "* Spatially Consistent Bias Correction: Univariate (SCBC_U): SCBC_U corrects local flows at each river reach in the network, and then reroutes them to aggregate, producing bias corrected flows everywhere.\n", + "* Spatially Consistent Bias Correction: Univariate (SCBC_U): SCBC_U corrects local flows at each river reach in the network, and then reroutes them to aggregate, producing bias corrected flows at locations of interest where reference data does and does not exist.\n", "* Spatially Consistent Bias Correction: Conditioned (SCBC_C): SCBC_C also corrects the local flows like SCBC_U, but allows for conditioning on dependent processes.\n", "\n", "# Expectations using this Tutorial\n", @@ -40,7 +39,7 @@ "\n", "By the end of this tutorial you will learn\n", "\n", - "- The data requirements to perform bias corrections with bmorph\n", + "- The data requirements to perform streamflow bias corrections with bmorph\n", "- Input and output formats that bmorph expects\n", "- The meaning of parameters for algorithmic control of bias corrections\n", "- How to perform independent bias correction at locations with reference flows\n", @@ -52,14 +51,13 @@ "# Import Packages and Load Data\n", "\n", "We start by importing the necessary packages for the notebook.\n", - "This tutorial mainly shows how to use ``bmorph.core.workflows`` and ``bmorph.core.mizuroute_utils`` to bias correct streamflow data in the Yakima river basin." - ] + "This tutorial mainly shows how to use ``bmorph.core.workflows`` and ``bmorph.core.mizuroute_utils`` to bias correct streamflow data in the Yakima River Basin." + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "%pylab inline\n", "%load_ext autoreload\n", @@ -95,35 +93,36 @@ "# Import pyproj and set the data directory, this is a workaround for portability\n", "import pyproj\n", "pyproj.datadir.set_data_dir(f'{envdir}/../share/proj')" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "\n", "## Getting the sample data\n", "\n", - "The following code cell has two commands preceded by `!`, which indicates that they are shell command. They will download the sample data and unpackage it. The sample data can be viewed as a HydroShare resource [here](https://www.hydroshare.org/resource/fd2a347d34f145b4bfa8b6bff39c782b/). This cell may take a few moments." - ] + "The following code cell has two commands preceded by `!`, which indicates that they are shell commands. They will download the sample data and unpackage it. The sample data can be viewed as a HydroShare resource [here](https://www.hydroshare.org/resource/fd2a347d34f145b4bfa8b6bff39c782b/). This cell may take a few moments." + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "! wget https://www.hydroshare.org/resource/fd2a347d34f145b4bfa8b6bff39c782b/data/contents/bmorph_testdata.tar.gz\n", "! tar xvf bmorph_testdata.tar.gz" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "# Test dataset: The Yakima River Basin\n", "\n", - "Before getting into how to run bmorph, let's look at what is in the sample data. You will note that we now have a `yakima_workflow` directory. This contains all of the data that you need to run the tutorial. There are a few subdirectories:\n", + "Before getting into how to run bmorph, let's look at what is in the sample data. You will note that we now have a `yakima_workflow` directory downloaded in the same directory as this notebook. This contains all of the data that you need to run the tutorial. There are a few subdirectories:\n", "\n", " * `gis_data`: contains shapefiles, this is mainly used for plotting, not for analysis\n", " * `input`: this is the input meteorologic data, simulated streamflow to be corrected, and the reference flow dataset\n", @@ -132,7 +131,7 @@ " * `topologies`: this contains the stream network topologies that will be used for routing flows via mizuroute\n", "\n", "\n", - "The Yakima river basin is a tributary of the Columbia river basin in the Pacific northwestern United States. It's western half is situated in the Cascade mountains and receives seasonal snowpack. The eastern half is lower elevation and is semi-arid. Let's load up the shapefiles for the sub-basins and stream network and plot it. In this discretization we have 285 sub-basins (HRU) and 143 stream segments.\n", + "The Yakima River Basin is a tributary of the Columbia river basin in the Pacific northwestern United States. It's western half is situated in the Cascade mountains and receives seasonal snowpack. The eastern half is lower elevation and is semi-arid. Let's load up the shapefiles for the sub-basins and stream network and plot it. In this discretization we have 285 sub-basins (HRU) and 143 stream segments.\n", "\n", "# Setting up some metadata\n", "\n", @@ -142,13 +141,12 @@ "`site_to_seg = { site_0_name : site_0_seg, ..., site_n_name, site_n_seg}`\n", "Since it is convenient to be able to access this data in different orders we also set up \n", "some other useful forms of these gauge site mappings for later use. We will show you on the map where each of these sites are on the stream network in the next section." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "site_to_seg = {'KEE' : 4175, 'KAC' : 4171, 'EASW': 4170, \n", " 'CLE' : 4164, 'YUMW': 4162, 'BUM' : 5231,\n", @@ -159,22 +157,22 @@ "seg_to_site = {seg: site for site, seg in site_to_seg.items()}\n", "ref_sites = list(site_to_seg.keys())\n", "ref_segs = list(site_to_seg.values()) " - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ - "# Mapping the Yakima river basin\n", + "# Mapping the Yakima River Basin\n", "\n", - "With our necessary metadata defined let's make a couple of quick plots orienting you to the Yakima river basin. To do so we will read in a network topology file, and shapefiles for the region. We will make one plot which has the Yakima river basin, along with stream network, subbasins, and gauged sites labeled. We will also plot a network diagram which displays in an abstract sense how each stream segment is connected. For the latter we use the [SimpleRiverNetwork](https://bmorph.readthedocs.io/en/latest/srn.html) that we've implemented in bmorph. To set up the `SimpleRiverNetwork` we the topology of the watershed (`yakima_topo`). The river network and shapefiles that we use to draw the map, and perform simulations on is the [Geospatial Fabric]( https://doi.org/10.5066/P971JAGF). The locations of the gauged sites are shown in red, while all of the un-gauged stream segments are shown in darker grey. The sub-basins for each stream segment are shown in lighter grey." - ] + "With our necessary metadata defined let's make a couple of quick plots orienting you to the Yakima River Basin. To do so we will read in a network topology file, and shapefiles for the region. We will make one plot which has the Yakima River Basin, along with stream network, subbasins, and gauged sites labeled. We will also plot a network diagram which displays in an abstract sense how each stream segment is connected. For the latter we use the [SimpleRiverNetwork](https://bmorph.readthedocs.io/en/latest/srn.html) that we've implemented in bmorph. To set up the `SimpleRiverNetwork` we need the topology of the watershed (`yakima_topo`). The river network and shapefiles that we use to draw the map, and perform simulations on is the [Geospatial Fabric]( https://doi.org/10.5066/P971JAGF). In the Geospatial Fabric, rivers and streams are broken into segments, each with a unique identifier, as illustrated above in the `site_to_seg` definition. The locations of the gauged sites are shown in red, while all of the un-gauged stream segments are shown in darker grey. The sub-basins for each stream segment are shown in lighter grey." + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "yakima_topo = xr.open_dataset('yakima_workflow/topologies/yakima_huc12_topology.nc').load()\n", "yakima_hru = gpd.read_file('./yakima_workflow/gis_data/yakima_hru.shp').to_crs(\"EPSG:4326\")\n", @@ -209,16 +207,17 @@ "yakima_srn.draw_network(color_measure=yakima_srn.generate_node_highlight_map(ref_segs), \n", " cmap=mpl.cm.get_cmap('Set1_r'), ax=axes[1], node_size=60)\n", "plt.tight_layout(pad=0)" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "# Loading in the streamflow data and associated meteorological data\n", "\n", "Now we load in the meteorological data that will be used for conditional bias correction: daily minimum temperature (`tmin`), seasonal precipitation (`prec`),\n", - "and daily maximum temperature (`tmax`). In principle, any type of data can be used for conditioning. This data is initially arranged on the sub-basins, rather than stream segments. We will remap these onto the stream segments in a moment, so that they can be used in the bias correction process.\n", + "and daily maximum temperature (`tmax`). In principle, any type of data can be used for conditioning, (i.e. Snow-Water Equivalent (SWE), groundwater storage, atmospheric carbon-dioxide conecentrations, etc.). This data is initially arranged on the sub-basins, rather than stream segments. We will remap these onto the stream segments in a moment, so that they can be used in the bias correction process.\n", "\n", "Finally, we load the simulated flows and reference flows. \n", "bmorph is designed to bias correct streamflow simulated with [mizuroute](https://mizuroute.readthedocs.io/en/latest/). \n", @@ -226,13 +225,12 @@ "In our case the reference flows are estimated no-reservoir-no-irrigation (NRNI) flows taken from the [River Management Joint Operating Committee (RMJOC)](https://www.bpa.gov/p/Generation/Hydro/Documents/RMJOC-II_Part_II.PDF).\n", "\n", "All of the datasets discussed are in the `xarray` [Dataset format](http://xarray.pydata.org/en/stable/user-guide/data-structures.html#dataset), which contains the metadata associated with the original NetCDF files. You can inspect the data simply by printing it out. For instance, here you can see that both the reference flows and raw flows (named `IRFroutedRunoff`, for \"Impulse Response Function routed runoff\" from mizuRoute) are in cubic meters per second." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "# Meteorologic data\n", "yakima_met = xr.open_dataset('yakima_workflow/input/yakima_met.nc').load()\n", @@ -251,11 +249,12 @@ "\n", "print('Reference flow units: ', yakima_ref['reference_flow'].units)\n", "print('Raw flow units: ', yakima_raw['IRFroutedRunoff'].units)" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "## Convert from ``mizuroute`` output to ``bmorph`` format\n", "\n", @@ -263,20 +262,20 @@ "We will use the `mizutil.to_bmorph` function to merge together all of the data we previously loaded, and calculate some extra pieces of information to perform spatially consistent bias corrections (SCBC).\n", "For more information about how we perform SCBC see [the SCBC page in the documentation](https://bmorph.readthedocs.io/en/develop/bias_correction.html#spatial-consistency-reference-site-selection-cdf-blend-factor).\n", "Now we pass our data in to ``to_bmorph``, the primary utility function for automating ``bmorph`` pre-processing." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "yakima_met_seg = mizutil.to_bmorph(yakima_topo, yakima_raw, yakima_ref, yakima_met, fill_method='r2')" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "## Setting up ``bmorph`` configuration and parameters\n", "\n", @@ -294,22 +293,22 @@ "Lastly the ``reference_window`` is when the reference flows should be used to \n", "smooth the Cumulative Distribution function (CDF) of the bias corrected flows. \n", "This is recommended to be set as equivalent to the ``train_window``." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "train_window = pd.date_range('1981-01-01', '1990-12-30')[[0, -1]]\n", "reference_window = train_window\n", "apply_window= pd.date_range('1991-01-01', '2005-12-30')[[0, -1]]" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "``interval`` is the length of ``bmorph``'s application intervals, \n", "typically a factor of years to preserve hydrologic \n", @@ -323,25 +322,24 @@ "Typical values are between 60 and 120 days.\n", "\n", "The two \"smoothing\" parameters are used to smooth the timeseries before the CDFs are computed\n", - "and have two different uses. THe `n_smooth_short` is used in the actual calculation of the CDFs which are used to perform the quantile mapping. Smoothing is used to ensure that the CDFs are smooth. Setting a very low value here may cause noisier bias corrected timeseries. Setting a very high value may cause the bias corrections to not match extreme flows. Typical values are from 7-48 days.\n", + "and have two different uses. The `n_smooth_short` is used in the actual calculation of the CDFs which are used to perform the quantile mapping. Smoothing is used to ensure that the CDFs are smooth. Setting a very low value here may cause noisier bias corrected timeseries. Setting a very high value may cause the bias corrections to not match extreme flows. Typical values are from 7-48 days.\n", "\n", "`n_smooth_long` on the other hand is used to preserve long-term trends in mean flows from the raw flows. Typical values are 270 to 720 days. Using very low values may cause bias corrections to be degraded. This feature can be turned off by setting `n_smooth_long` to `None`.\n", "\n", "``condition_var`` names the variable to use in conditioning, such as maximum\n", - "temperature (tmax), 90 day rolling total precipitation (seasonal_precip), or daily\n", - "minimum temperature (tmin). At this time, only one conditioning\n", + "temperature (`tmax`), 90 day rolling total precipitation (`seasonal_precip`), or daily\n", + "minimum temperature (`tmin`). At this time, only one conditioning\n", "meteorological variable can be used per ``bmorph`` execution. In this example,\n", "``tmax`` and ``seasonal_precip`` have been commented out to select ``tmin`` as\n", - "the conditioning variable. We have precomputed the ``seasonal_precip`` to be If you wish to change this, be sure to either change which variables are commented out or change the value of ``condition_var`` itself. For now we will just use `tmin`, which is the daily minimum temperature. Our hypothesis on choosing `tmin` is that it will be a good indicator for errors in snow processes, which should provide a good demonstration for how conditional bias correction can modify flow timing in desirable ways.\n", + "the conditioning variable. If you wish to change this, be sure to either change which variables are commented out or change the value of ``condition_var`` itself. For now we will just use `tmin`, which is the daily minimum temperature. Our hypothesis on choosing `tmin` is that it will be a good indicator for errors in snow processes, which should provide a good demonstration for how conditional bias correction can modify flow timing in desirable ways.\n", "\n", "Further algorithmic controls can be used to tune the conditional bias correction as well. Here we use the histogram method for estimating the joint PDF, which is provided as `hist` as the `method`. We also have implemented a kernel density estimator which will be used if you set the `method` to `kde`. While `kde` tends to make smoother PDFs it comes with a larger computational cost. For both methods we specify the number of `xbins` and `ybins` which control how fine grained the joint PDFs should be calculated as. Setting a very high number here can potentially cause jumpy artifacts in the bias corrected timeseries." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "# bmorph parameter values\n", "interval = pd.DateOffset(years=5)\n", @@ -353,11 +351,12 @@ "#condition_var = 'tmax'\n", "#condition_var = 'seasonal_precip'\n", "condition_var = 'tmin'" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "Here we name some configuration parameters for ``bmorph``'s conditional and univariate\n", "bias correction methods, respectively. \n", @@ -366,13 +365,12 @@ "basin's name, make certain to update this with the actual name of\n", "the basin you are analyzing so you can track where different files\n", "are written." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "conditional_config = {\n", " 'data_path': './yakima_workflow',\n", @@ -401,13 +399,14 @@ " 'n_smooth_long': n_smooth_long,\n", " 'n_smooth_short': n_smooth_short,\n", "}" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ - "You made it! Now we can actually bias correction with ``bmorph``! \n", + "You made it! Now we can actually bias correct with ``bmorph``! \n", "\n", "First off we run the Independent Bias Corrections, which are completely contained\n", "in the cell below. \n", @@ -416,20 +415,19 @@ "individually. Since independent bias correction can only be performed\n", "at locations with reference data, corrections are only performed at\n", "the gauge sites here. " - ] + ], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "## Independent bias correction" - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "ibc_u_flows = {}\n", "ibc_u_mults = {}\n", @@ -455,69 +453,69 @@ " ## IBC_C (Independent Bias Correction: Conditioned)\n", " ibc_c_flows[site], ibc_c_mults[site] = bmorph.workflows.apply_bmorph(\n", " raw_ts, train_ts, obs_ts, condition_ts=cond_var, **conditional_config)" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "# Spatially consistent bias correction" - ] + ], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "Here we specify where the ``mizuroute`` executable is installed on your system." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "mizuroute_exe = f'{envdir}/route_runoff.exe'" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "Now we use ``run_parallel_scbc`` to do the rest. \n", "In the first cell we will run the spatially-consistent bias correction without any conditioning. \n", "The second cell will run the spatially-consistent bias correction with conditioning.\n", - "This produced bias corrected flows at all 143 stream segments in the Yakima river basin.\n", + "This produced bias corrected flows at all 143 stream segments in the Yakima River Basin.\n", "Finally, we select out the corrected streamflows for both cases (with and without conditioning) to only contain the gauged sites.\n", "Selecting out only the gauged locations allows us to compare the spatially-consistent methods with the independent bias corrections.\n", "Finally we combine all the data into a single xarray `Dataset` to make analysis easier." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "# SCBC without conditioning\n", "unconditioned_seg_totals = bmorph.workflows.apply_scbc(yakima_met_seg, mizuroute_exe, univariate_config)" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "# SCBC with conditioning\n", "conditioned_seg_totals = bmorph.workflows.apply_scbc(yakima_met_seg, mizuroute_exe, conditional_config)" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "# Here we select out our rerouted gauge site modeled flows.\n", "unconditioned_site_totals = {}\n", @@ -525,20 +523,20 @@ "for site, seg in tqdm(site_to_seg.items()):\n", " unconditioned_site_totals[site] = unconditioned_seg_totals['IRFroutedRunoff'].sel(seg=seg).to_series()\n", " conditioned_site_totals[site] = conditioned_seg_totals['IRFroutedRunoff'].sel(seg=seg).to_series()" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "# Merging together the results" - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "# Merge everything together\n", "yakima_analysis = xr.Dataset(coords={'site': list(site_to_seg.keys()), 'time': unconditioned_seg_totals['time']})\n", @@ -549,13 +547,13 @@ "yakima_analysis['raw'] = bmorph.workflows.bmorph_to_dataarray(raw_flows, 'raw')\n", "yakima_analysis['ref'] = bmorph.workflows.bmorph_to_dataarray(ref_flows, 'ref')\n", "yakima_analysis.to_netcdf(f'./yakima_workflow/output/{univariate_config[\"output_prefix\"]}_data_processed.nc')" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "# And also output it as some CSV files\n", "yakima_analysis['scbc_c'].to_pandas().to_csv(f'./yakima_workflow/output/{univariate_config[\"output_prefix\"]}_data_processed_scbc_c.csv')\n", @@ -564,67 +562,67 @@ "yakima_analysis['ibc_c'].to_pandas().to_csv(f'./yakima_workflow/output/{univariate_config[\"output_prefix\"]}_data_processed_ibc_u.csv')\n", "yakima_analysis['raw'].to_pandas().to_csv(f'./yakima_workflow/output/{univariate_config[\"output_prefix\"]}_data_processed_raw.csv')\n", "yakima_analysis['ref'].to_pandas().to_csv(f'./yakima_workflow/output/{univariate_config[\"output_prefix\"]}_data_processed_ref.csv')" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "## Now let's take a look at our results\n", "\n", "If you look closely, the following plots are the same ones included in [Plotting](evaluation.rst/Plotting)! Because the plotting functions expect the variable `seg`, we will need to rename `site` to `seg` for them to properly run." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "yakima_ds = xr.open_dataset(f'yakima_workflow/output/{univariate_config[\"output_prefix\"]}_data_processed.nc')\n", "yakima_ds = yakima_ds.rename({'site':'seg'})" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "Let's pick a few sites and colors to plot for consistency. To simplify our plots, we will only focus on `scbc_c` in the dataset we just created. The methods do allow for multiple methods to be compared at once however, so we will still need to store the singular `scbc_c` in a list." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "select_sites = ['KIOW','YUMW','BUM']\n", "select_sites_2 = ['KIOW','CLFW','BUM','UMTW']\n", "bcs = ['scbc_c', 'scbc_u', 'ibc_c', 'ibc_u']\n", "colors = ['grey', 'black', 'red', 'orange', 'purple', 'blue']" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "### Time Series" - ] + ], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "Here we plot the mean weekly flows for some of the sites in Yakima River Basin. You can change or add sites above, but we will start with a small number of sites to make the plots more tractable. In the following function call you \n", "\n", "As mentioned, these averages are computed on weekly intervals to simplify the figure, but can also be plotted on daily or monthly intervals for more or less granularity. You can also change the `reduce_func` to calculate any other statistic over the dataset (you might try `np.median` or `np.var` for instance). Don't forget to change the `statistic_label` for other measures!" - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "bplot.plot_reduced_flows(\n", " flow_dataset=yakima_ds, \n", @@ -637,11 +635,12 @@ " bc_vars=bcs, bc_names=[bc.upper() for bc in bcs],\n", " plot_colors=colors\n", ")" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "From the plot above we can see that the conditional corrections (`x_C` methods) have more accurate flow timings, particularly during the falling limb of the hydrograph. This hints that our hypothesis on correcting on daily minimum temperature would provide a good proxy for correcting snowmelt biases. We will explore this a little bit more later.\n", "\n", @@ -650,13 +649,12 @@ "### Scatter\n", "\n", "This compares how absolute error changes through each bias correction with Q being stream discharge. 1 to 1 and -1 to 1 lines are plotted for reference, as points plotted vertically between the lines demonstrates a reduction in absolute error while points plotted horizontally between the lines demonstrates an increase in absolute error for each flow time." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "bplot.compare_correction_scatter(\n", " flow_dataset= yakima_ds, \n", @@ -673,27 +671,27 @@ " fontsize_legend = 120,\n", " alpha = 0.3\n", ")" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "### Probabilitiy Distribtutions" - ] + ], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "Since probability distributions are used to predict extreme flow events and are what ``bmorph`` directly corrects, looking at them will give us greater insight to the changes we made." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, - "outputs": [], "source": [ "bplot.compare_mean_grouped_CPD(\n", " flow_dataset= yakima_ds, \n", @@ -709,36 +707,34 @@ " fontsize_legend = 90,\n", " legend_bbox_to_anchor = (1.9,1.0)\n", ");" - ] + ], + "outputs": [], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "This function is also capable of subsetting data by month should you want to compare only January flows for example. Because ``bmorph`` makes changes based on flow distributions, this plot is the closest to directly analyzing how the different methods correct flows." - ] + ], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "### Simple River Network" - ] + ], + "metadata": {} }, { "cell_type": "markdown", - "metadata": {}, "source": [ "Finally, we can plot information of the SCBC across the simple river network. Let's look at the difference in the average percent difference for both `SCBC_U` and `SCBC_C`. From the timeseries plots created earlier you might have noticed that the conditional bias corrections produced lower flows in the spring months. We will start by looking only at those months. You might try changing the season if you're interested. " - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": { - "tags": [] - }, - "outputs": [], "source": [ "season = 'MAM' # Choose from DJF, MAM, JJA, SON\n", "scbc_c = conditioned_seg_totals['IRFroutedRunoff']\n", @@ -758,11 +754,14 @@ "yakima_srn.draw_network(color_measure=scbc_c_percent_diff, cmap=mpl.cm.get_cmap('coolwarm_r'), node_size=40,\n", " with_cbar=True, cbar_labelsize=20, ax=axes[1], cbar_title='Mean percent difference from raw flows(%)')\n", "axes[1].set_title('SCBC_C')" - ] + ], + "outputs": [], + "metadata": { + "tags": [] + } }, { "cell_type": "markdown", - "metadata": {}, "source": [ "From the plot above we can see that the main differences between the two methods was in modifying the headwater flows, which are at higher elevations and receive more precipitation. This aligns with our hypothesis that the daily minimum temperature would provide a good proxy for erros in snow processes.\n", "\n", @@ -772,21 +771,21 @@ "- What happens if you conditionally bias correct on a different variable? Try `seasonal_precip`, or even implement a bias correction conditional on the month if you're feeling adventurous!\n", "- How do the smoothing parameters affect the bias corrected flows? Try a wide range of `n_smooth_short`, or try setting `n_smooth_long` to `None` to turn off the correction of the mean trend.\n", "- Try removing half of the gauged sites to see how it affects the spatially-consistent bias correction. You can do this by commenting out (or deleting) some of the entries in `site_to_seg` up at the top." - ] + ], + "metadata": {} }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "source": [], "outputs": [], - "source": [] + "metadata": {} } ], "metadata": { "kernelspec": { - "display_name": "bmorph", - "language": "python", - "name": "bmorph" + "name": "python3", + "display_name": "Python 3.9.2 64-bit ('bmorph': conda)" }, "language_info": { "codemirror_mode": { @@ -806,8 +805,11 @@ "version_major": 2, "version_minor": 0 } + }, + "interpreter": { + "hash": "7f3870f67d182f2e264f74948103e35c18a48c989bcf8fae698e6abb46fb3d67" } }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file From 8e96e67db315054f5923ab01abf7de29d228a1ee Mon Sep 17 00:00:00 2001 From: steinadio Date: Thu, 19 Aug 2021 11:27:34 -0700 Subject: [PATCH 3/3] Fixed link to Binder and synced rst with notebook fro bmorph_tutorial --- docs/bmorph_tutorial.rst | 78 +++++++++++++++++++--------------- docs/index.rst | 2 +- tutorial/bmorph_tutorial.ipynb | 4 +- 3 files changed, 46 insertions(+), 38 deletions(-) diff --git a/docs/bmorph_tutorial.rst b/docs/bmorph_tutorial.rst index 6a09b01..e211659 100644 --- a/docs/bmorph_tutorial.rst +++ b/docs/bmorph_tutorial.rst @@ -18,9 +18,9 @@ page. The rest of the documentation for bmorph can be found correcting for specific biases that are process-dependent. This method can only be performed at sites with reference data. - Spatially Consistent Bias Correction: Univariate (SCBC_U): SCBC_U - corrects local flows at each river reach in the network, and then - reroutes them to aggregate, producing bias corrected flows - everywhere. + corrects local flows at each river reach in the network, and then + reroutes them to aggregate, producing bias corrected flows at + locations of interest where reference data does and does not exist. - Spatially Consistent Bias Correction: Conditioned (SCBC_C): SCBC_C also corrects the local flows like SCBC_U, but allows for conditioning on dependent processes. @@ -57,7 +57,7 @@ Learning objectives & goals By the end of this tutorial you will learn -- The data requirements to perform bias corrections with bmorph +- The data requirements to perform streamflow bias corrections with bmorph - Input and output formats that bmorph expects - The meaning of parameters for algorithmic control of bias corrections - How to perform independent bias correction at locations with @@ -76,7 +76,7 @@ Import Packages and Load Data We start by importing the necessary packages for the notebook. This tutorial mainly shows how to use ``bmorph.core.workflows`` and ``bmorph.core.mizuroute_utils`` to bias correct streamflow data in the -Yakima river basin. +Yakima River Basin. .. code:: ipython3 @@ -125,7 +125,7 @@ Getting the sample data ----------------------- The following code cell has two commands preceded by ``!``, which -indicates that they are shell command. They will download the sample +indicates that they are shell commands. They will download the sample data and unpackage it. The sample data can be viewed as a HydroShare resource `here `__. @@ -186,10 +186,13 @@ This cell may take a few moments. Test dataset: The Yakima River Basin ==================================== -Before getting into how to run bmorph, let’s look at what is in the -sample data. You will note that we now have a ``yakima_workflow`` -directory. This contains all of the data that you need to run the -tutorial. There are a few subdirectories: +Before getting into how to run bmorph, let's look at what is in the +sample data. You will note that we now have a ``yakima_workflow`` +directory downloaded in the same directory as this notebook (found by +clicking on the ``1tutorial/`` tab left by Binder or navigating to +this directory in the file explorer of your choice). This contains +all of the data that you need to run the tutorial. There are a few +subdirectories: - ``gis_data``: contains shapefiles, this is mainly used for plotting, not for analysis @@ -203,7 +206,7 @@ tutorial. There are a few subdirectories: - ``topologies``: this contains the stream network topologies that will be used for routing flows via mizuroute -The Yakima river basin is a tributary of the Columbia river basin in the +The Yakima River Basin is a tributary of the Columbia river basin in the Pacific northwestern United States. It’s western half is situated in the Cascade mountains and receives seasonal snowpack. The eastern half is lower elevation and is semi-arid. Let’s load up the shapefiles for the @@ -235,24 +238,27 @@ sites are on the stream network in the next section. ref_sites = list(site_to_seg.keys()) ref_segs = list(site_to_seg.values()) -Mapping the Yakima river basin +Mapping the Yakima River Basin ============================== With our necessary metadata defined let’s make a couple of quick plots -orienting you to the Yakima river basin. To do so we will read in a +orienting you to the Yakima River Basin. To do so we will read in a network topology file, and shapefiles for the region. We will make one -plot which has the Yakima river basin, along with stream network, +plot which has the Yakima River Basin, along with stream network, subbasins, and gauged sites labeled. We will also plot a network diagram which displays in an abstract sense how each stream segment is connected. For the latter we use the `SimpleRiverNetwork `__ that we’ve implemented in bmorph. To set up the ``SimpleRiverNetwork`` -we the topology of the watershed (``yakima_topo``). The river network -and shapefiles that we use to draw the map, and perform simulations on -is the `Geospatial Fabric `__. The -locations of the gauged sites are shown in red, while all of the -un-gauged stream segments are shown in darker grey. The sub-basins for -each stream segment are shown in lighter grey. +we need the topology of the watershed (``yakima_topo``). The river +network and shapefiles that we use to draw the map, and perform +simulations on is the `Geospatial Fabric `__. +In the Geospatial Fabric, rivers and streams are broken into +segments, each with a unique identifier, as illustrated above in the +``site_to_seg`` definition. The locations of the gauged sites are +shown in red, while all of the un-gauged stream segments are shown in +darker grey. The sub-basins for each stream segment are shown in +lighter grey. .. code:: ipython3 @@ -300,11 +306,13 @@ Loading in the streamflow data and associated meteorological data Now we load in the meteorological data that will be used for conditional bias correction: daily minimum temperature (``tmin``), seasonal -precipitation (``prec``), and daily maximum temperature (``tmax``). In -principle, any type of data can be used for conditioning. This data is -initially arranged on the sub-basins, rather than stream segments. We -will remap these onto the stream segments in a moment, so that they can -be used in the bias correction process. +precipitation (``prec``), and daily maximum temperature (``tmax``). +In principle, any type of data can be used for conditioning, (i.e. +Snow-Water Equivalent (SWE), groundwater storage, landscape slope +angle, etc.). This data is initially arranged on the sub-basins, +rather than stream segments. We will remap these onto the stream +segments in a moment, so that they can be used in the bias correction +process. Finally, we load the simulated flows and reference flows. bmorph is designed to bias correct streamflow simulated with @@ -425,14 +433,14 @@ maximum temperature (tmax), 90 day rolling total precipitation (seasonal_precip), or daily minimum temperature (tmin). At this time, only one conditioning meteorological variable can be used per ``bmorph`` execution. In this example, ``tmax`` and ``seasonal_precip`` have been -commented out to select ``tmin`` as the conditioning variable. We have -precomputed the ``seasonal_precip`` to be If you wish to change this, be -sure to either change which variables are commented out or change the -value of ``condition_var`` itself. For now we will just use ``tmin``, -which is the daily minimum temperature. Our hypothesis on choosing -``tmin`` is that it will be a good indicator for errors in snow -processes, which should provide a good demonstration for how conditional -bias correction can modify flow timing in desirable ways. +commented out to select ``tmin`` as the conditioning variable. If you +wish to change this, be sure to either change which variables are +commented out or change the value of ``condition_var`` itself. For +now we will just use ``tmin``, which is the daily minimum +temperature. Our hypothesis on choosing ``tmin`` is that it will be a +good indicator for errors in snow processes, which should provide a +good demonstration for how conditional bias correction can modify +flow timing in desirable ways. Further algorithmic controls can be used to tune the conditional bias correction as well. Here we use the histogram method for estimating the @@ -495,7 +503,7 @@ written. 'n_smooth_short': n_smooth_short, } -You made it! Now we can actually bias correction with ``bmorph``! +You made it! Now we can actually bias correct with ``bmorph``! First off we run the Independent Bias Corrections, which are completely contained in the cell below. @@ -556,7 +564,7 @@ Now we use ``run_parallel_scbc`` to do the rest. In the first cell we will run the spatially-consistent bias correction without any conditioning. The second cell will run the spatially-consistent bias correction with conditioning. This produced bias corrected flows at all -143 stream segments in the Yakima river basin. Finally, we select out +143 stream segments in the Yakima River Basin. Finally, we select out the corrected streamflows for both cases (with and without conditioning) to only contain the gauged sites. Selecting out only the gauged locations allows us to compare the spatially-consistent methods with the diff --git a/docs/index.rst b/docs/index.rst index a7a24dd..dac1a5f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -33,7 +33,7 @@ Then, to install bmorph run, Getting started =============== -You can run through our interactive tutorial `here `_. +You can run through our interactive tutorial `here `_. A static version of the tutorial can be found `here `_. bmorph Overview diff --git a/tutorial/bmorph_tutorial.ipynb b/tutorial/bmorph_tutorial.ipynb index dfe3518..9752796 100644 --- a/tutorial/bmorph_tutorial.ipynb +++ b/tutorial/bmorph_tutorial.ipynb @@ -122,7 +122,7 @@ "source": [ "# Test dataset: The Yakima River Basin\n", "\n", - "Before getting into how to run bmorph, let's look at what is in the sample data. You will note that we now have a `yakima_workflow` directory downloaded in the same directory as this notebook. This contains all of the data that you need to run the tutorial. There are a few subdirectories:\n", + "Before getting into how to run bmorph, let's look at what is in the sample data. You will note that we now have a `yakima_workflow` directory downloaded in the same directory as this notebook (found by clicking on the `tutorial/` tab left by Binder or navigating to this directory in the file explorer of your choice). This contains all of the data that you need to run the tutorial. There are a few subdirectories:\n", "\n", " * `gis_data`: contains shapefiles, this is mainly used for plotting, not for analysis\n", " * `input`: this is the input meteorologic data, simulated streamflow to be corrected, and the reference flow dataset\n", @@ -217,7 +217,7 @@ "# Loading in the streamflow data and associated meteorological data\n", "\n", "Now we load in the meteorological data that will be used for conditional bias correction: daily minimum temperature (`tmin`), seasonal precipitation (`prec`),\n", - "and daily maximum temperature (`tmax`). In principle, any type of data can be used for conditioning, (i.e. Snow-Water Equivalent (SWE), groundwater storage, atmospheric carbon-dioxide conecentrations, etc.). This data is initially arranged on the sub-basins, rather than stream segments. We will remap these onto the stream segments in a moment, so that they can be used in the bias correction process.\n", + "and daily maximum temperature (`tmax`). In principle, any type of data can be used for conditioning, (i.e. Snow-Water Equivalent (SWE), groundwater storage, landscape slope angle, etc.). This data is initially arranged on the sub-basins, rather than stream segments. We will remap these onto the stream segments in a moment, so that they can be used in the bias correction process.\n", "\n", "Finally, we load the simulated flows and reference flows. \n", "bmorph is designed to bias correct streamflow simulated with [mizuroute](https://mizuroute.readthedocs.io/en/latest/). \n",