DR_method_comparison.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Technical demand response potential meta analysis - method comparison (overview)\n",
    "\n",
    "**Puporse and background**: This notebook serves for evaluating the methods of different publications for assessing technical demand response potentials for Germany. The data from the publications has been collected in order to carry out a meta analysis. The main use case of this particular notebook is to create bar charts for the methological elements evaluated. The method comparison analysis self has widely already been carried out using Excel.\n",
    "\n",
    "## Method applied\n",
    "In the following, a brief description of the method applied is given for the sake of replicability.\n",
    "\n",
    "### Selection of publications\n",
    "Criterions for the selection of publications were:\n",
    "* Regional focus: Federal republic of Germany\n",
    "* Temporal restricitions: Publications until 2005\n",
    "* Contents: technical demand response potentials\n",
    "* Sectoral coverage: At least one of households, trade, commerce and services or industry assessed; no publications focusing on single appliances\n",
    "* Own analysis carried out (no other meta analysis included)\n",
    "* Inclusion of at least one of the parameters of interest (usually at least some information on potential in MW)\n",
    "\n",
    "### Methological elements evaluated\n",
    "The following method elements have been evaluated and data has been collected and processed within an Excel workbook:\n",
    "* overall methological approach applied and data sources used\n",
    "* main assumptions made for potential determination\n",
    "* data basis in detail\n",
    "* method for further processing the potentials derived\n",
    "* processes and appliances evaluated (per sector)\n",
    "* parameters given to describe the technical demand response potential. These include:\n",
    "    * capacity-related parameters: capacity available for load shifting or load shedding as well as minimum and maximum load level\n",
    "    * time-reletad parameters: duration for activation, interference time, shifting time, regenaration time, time-dependent availability of potentials\n",
    "    * cost-related parameters: specific investments, variable costs (for activation), fixed costs\n",
    "    * These parameters have been separately collected in another Excel workbook (Potenziale_Lastmanagement.xlsx) and processed using another jupyter notebook (DR_potential_evaluation_VXX.ipynb).\n",
    "* kind of potential evaluated in the study (besides the technical potential, such as theoretical, economic etc.)\n",
    "* temporal horizon covered\n",
    "* base year\n",
    "\n",
    "### Filtering and data collection\n",
    "* A total of 83 publications has been selected in the first place in a literature research based on the titles, keywords and abstracts using terms like \"Demand Response\", \"Demand Side Management\", \"Demand Side Integration\", \"load management\", \"load shifting\", \"load shedding\", \"potential\", \"Germany\" as well as their German equivalents to formulate search strings. \n",
    "* Of these publications, 37 have met the above stated criteria and were further evaluated. \n",
    "* Because sometimes, several publications belonged to the same research project, these were grouped together resulting in 24 publications resp. publication groups for which potential information has been evaluated.\n",
    "* The data for the method comparison has been collected in an Excel Workbook (Methodenvergleich_Potenzialstudien.xlsx). The latest version of this Excel Workbook is evaluated with this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Package imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parameter settings\n",
    "Set some parameters here in order to ensure code adaptability to future versions of potential collection (Excel workbook)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_folder_in = \"./inputs/\"\n",
    "path_folder_out = \"./out/\"\n",
    "path_folder_plots = \"plots/\"\n",
    "path_folder_sources = \"sources/\"\n",
    "filename = \"Methodenvergleich_Potenzialstudien.xlsx\"\n",
    "filename_plot = \"method_comparison\"\n",
    "\n",
    "# boolean parameters for controlling function behaviour\n",
    "savefig = True\n",
    "IEEE = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read in data\n",
    "Read in the relevant sheets of the Excel Workbook containing all data to be processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in data using pandas and parse it to individual DataFrames for each publication\n",
    "xls = pd.ExcelFile(f\"{path_folder_in}{filename}\")\n",
    "\n",
    "method = xls.parse(\"01_Methode_Daten_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "assumptions = xls.parse(\"01_Annahmen_Daten_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "data = xls.parse(\"01_Datenbasis_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "analysis = xls.parse(\"01_Folgeanalyse_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "citation = xls.parse(\"01_Zitationsanalyse_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "ind_processes = xls.parse(\"02_Ind_Prozesse_kodiert\", header=[0], index_col=0)\n",
    "ind_crosscutting = xls.parse(\"02_Ind_QST_kodiert\", header=[0], index_col=0)\n",
    "tcs_crosscutting = xls.parse(\"02_GHD_QST-Branchen_kodiert\", header=[0], index_col=0)\n",
    "hoho_appliances = xls.parse(\"02_HaHa_kodiert\", header=[0], index_col=0)\n",
    "flex_parameters = xls.parse(\"03_Flexparameter_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "availability = xls.parse(\"03_Zeitverfuegbarkeit_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "kind_potential = xls.parse(\"04_Potenzialbegriff_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "time_horizon = xls.parse(\"05_Betrachtungshorizont_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "base_year = xls.parse(\"05_Basisjahr_kodiert\", skiprows=0, header=[1], index_col=0)\n",
    "\n",
    "# read in overview containing all sources information\n",
    "if IEEE:\n",
    "    overview = xls.parse(\"Gesamtueberblick\", skiprows=0, skipfooter=1, header=[1], usecols=[el for el in range(1,81)])\n",
    "else:\n",
    "    overview = xls.parse(\"Gesamtueberblick\", skiprows=0, skipfooter=1, header=[1], usecols=[el for el in range(1,81)], index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare sources information\n",
    "* Transform the sources information into IEEE style.\n",
    "* Concat the strings.\n",
    "* Write the results to Excel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols_to_use = [el for el in overview.columns if \"Quellen\" in el]\n",
    "if IEEE:\n",
    "    for el in cols_to_use:\n",
    "        overview[el] = \"[\" + (overview.index + 1).astype(str) + \"], S. \" + overview[el].astype(str) + \"; \"\n",
    "else:\n",
    "    for el in cols_to_use:\n",
    "        overview[el] = overview.index + \", S. \" + overview[el].astype(str) + \"; \"\n",
    "overview.loc[\"sum\"] = overview[cols_to_use].sum()\n",
    "overview.loc[\"sum\", cols_to_use].to_excel(f\"{path_folder_out}{path_folder_sources}sources_method_comparison.xlsx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare data for further analysis\n",
    "* Define a dict for iterating over all DataFrames (worksheets) which has a string as key and a list as value consisting of\n",
    "    * the DataFrame itself,\n",
    "    * the columns to be removed for creating bar plots as well as\n",
    "    * the xaxis title for the individual subplot.\n",
    "* Drop column Häufigkeit and calculate it using pd builtins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "method_dict = {\n",
    "    \"method\": [method, ['Quellen Methodik', 'Bezugsjahr(e) der Datenbasis', 'Quellen Daten'], \"Methode und Datenquellen\"],\n",
    "    \"assumptions\": [assumptions, ['Quellen zentrale Annahmen'], \"Zentrale Annahmen\"],\n",
    "    \"data\": [data, ['Quellen Daten'], \"Datenquellen\"],\n",
    "    \"analysis\": [analysis, [], \"Folgeverwertung der technischen Potenzialschätzung\"],\n",
    "    \"citation\": [citation, ['Fundstellen'], \"Zitationshäufigkeiten\"],\n",
    "    \"ind_processes\": [ind_processes, [], \"Industrielle Prozesse\"],\n",
    "    \"ind_crosscutting\": [ind_crosscutting, [], \"Industrielle Querschnittstechnologien\"],\n",
    "    \"tcs_crosscutting\": [tcs_crosscutting, [], \"Anwendungen und Branchen im GHD-Sektor\"],\n",
    "    \"hoho_appliances\": [hoho_appliances, [], \"Anwendungen im Haushaltssektor\"],\n",
    "    \"flex_parameters\": [flex_parameters, ['Quellen Flexibilitätsparameter'], \"ausgewertete Flexibilitätsparameter\"],\n",
    "    \"availability\": [availability, ['Quellen Zeitverfügbarkeit'], \"Angaben zur Zeitverfügbarkeit\"],\n",
    "    \"kind_potential\": [kind_potential, [], \"Verwendeter Potenzialbegriff\"],\n",
    "    \"time_horizon\": [time_horizon, [], \"abgedeckter Zeithorizont\"],\n",
    "    \"base_year\": [base_year, [], \"Basisjahr\"]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop \"Häufigkeit\" columns and calculate sum as well as frequency using built-in functions\n",
    "# Row Scholz et al. 2014 is dropped since it has been decided to group this publication together with the publications from Gils, esp. Gils 2015\n",
    "for k in method_dict.keys():\n",
    "    method_dict[k][0] = method_dict[k][0].drop([\"[27]\", \"[32]\", \"Häufigkeit\"]).fillna(0).rename_axis(None)\n",
    "    method_dict[k][0].loc[\"Summe\"] = method_dict[k][0].sum()\n",
    "    method_dict[k][0].loc[\"Häufigkeit \\n(inkl. teilweise Erfüllung)\"] = method_dict[k][0].iloc[:-1].astype(bool).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualize the data\n",
    "* Plot the data using a bar chart plot.\n",
    "* Save the plot(s) output to a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SMALL_SIZE = 13\n",
    "MEDIUM_SIZE = 15\n",
    "BIGGER_SIZE = 16\n",
    "\n",
    "plt.rc('font', size=SMALL_SIZE)          # controls default text sizes\n",
    "plt.rc('axes', titlesize=BIGGER_SIZE)    # fontsize of the axes title\n",
    "plt.rc('axes', labelsize=MEDIUM_SIZE)    # fontsize of the x and y labels\n",
    "plt.rc('xtick', labelsize=SMALL_SIZE)    # fontsize of the tick labels\n",
    "plt.rc('ytick', labelsize=SMALL_SIZE)    # fontsize of the tick labels\n",
    "plt.rc('legend', fontsize=SMALL_SIZE)    # legend fontsize\n",
    "plt.rc('figure', titlesize=BIGGER_SIZE)  # fontsize of the figure title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, (k, v) in enumerate(method_dict.items()):\n",
    "    \n",
    "    max_label_length = max([\n",
    "        len(str(label)) \n",
    "        for label in v[0].drop(method_dict[k][1], axis=1).columns\n",
    "    ])\n",
    "    \n",
    "    fig, ax = plt.subplots(figsize=(15, 5 + 0.05 * max_label_length))\n",
    "    \n",
    "    to_plot = v[0].drop(method_dict[k][1], axis=1).loc[[\"Summe\", \"Häufigkeit \\n(inkl. teilweise Erfüllung)\"]].T\n",
    "    _ = ax.set_ylim([0, 34])\n",
    "    _ = ax.set_yticks(range(0, 34, 2))\n",
    "    _ = ax.set_title(method_dict[k][2], pad=10)\n",
    "    _ = ax.set_ylabel('Absolute Häufigkeit', labelpad=10)\n",
    "    _ = to_plot.plot(kind=\"bar\", ax=ax, color=[\"lightgray\", \"black\"], edgecolor=\"black\", hatch=\"/\", zorder=10)\n",
    "    _ = plt.xticks(rotation=60, ha=\"right\")\n",
    "    _ = ax.grid(True, linestyle='--', linewidth=0.5, zorder=-1)\n",
    "\n",
    "    plt.tight_layout()    \n",
    "    plt.savefig(f\"{path_folder_out}{path_folder_plots}{filename_plot}_{k}.png\", dpi=300, bbox_inches=\"tight\")\n",
    "#plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create some separate plots (for the presentation only)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(2, 1, figsize=(15,13))\n",
    "    \n",
    "to_plot_1 = method_dict[\"ind_processes\"][0].drop(method_dict[\"ind_processes\"][1], axis=1).loc[[\"Summe\", \"Häufigkeit \\n(inkl. teilweise Erfüllung)\"]].T\n",
    "_ = to_plot_1.plot(kind=\"bar\", ax=axs[0], color=[\"lightgray\", \"black\"], edgecolor=\"black\", hatch=\"/\", zorder=10)\n",
    "_ = axs[0].set_xticklabels(axs[0].get_xticklabels(), rotation=60, ha=\"right\")\n",
    "_ = axs[0].set_ylim([0, 34])\n",
    "_ = axs[0].set_yticks(range(0, 34, 2))\n",
    "_ = axs[0].set_ylabel('absolute Häufigkeit', labelpad=10)\n",
    "_ = axs[0].grid(True, linestyle='--', linewidth=0.5, zorder=-1)\n",
    "\n",
    "to_plot_2 = method_dict[\"ind_crosscutting\"][0].drop(method_dict[\"ind_crosscutting\"][1], axis=1).loc[[\"Summe\", \"Häufigkeit \\n(inkl. teilweise Erfüllung)\"]].T\n",
    "_ = to_plot_2.plot(kind=\"bar\", ax=axs[1], color=[\"lightgray\", \"black\"], edgecolor=\"black\", hatch=\"/\", zorder=10)\n",
    "_ = axs[1].set_xticklabels(axs[1].get_xticklabels(), rotation=60, ha=\"right\")\n",
    "_ = axs[1].set_ylim([0, 34])\n",
    "_ = axs[1].set_yticks(range(0, 34, 2))\n",
    "_ = axs[1].set_ylabel('absolute Häufigkeit', labelpad=10)\n",
    "_ = axs[1].grid(True, linestyle='--', linewidth=0.5, zorder=-1)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(f\"{path_folder_out}{path_folder_plots}{filename_plot}_industry.png\", dpi=300, bbox_inches=\"tight\")\n",
    "#plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print some information (for the presentation only)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in method_dict.keys():\n",
    "    print(\"------------------------------------------------------------------\")\n",
    "    print(k)\n",
    "    print(\"------------------------------------------------------------------\")\n",
    "    display(method_dict[k][0].loc[[\"Summe\", \"Häufigkeit \\n(inkl. teilweise Erfüllung)\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "toc-autonumbering": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}