ncoda_07fv1_HUA_PREC_NSI.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "nDetVnGnoHR-"
   },
   "source": [
    "# Housing Unit Allocation with Person Record File Full Workflow\n",
    "\n",
    "## Overview\n",
    "This code works with the National Structures Inventory to run the housing unit allocation (HUA) and the person record file (PREC) workflow.\n",
    "The HUA process is generalizable to any county in the United States. The HUA process will work for any file that has locations of structures and some basic information about the buildings.\n",
    "The process is designed to work with [IN-CORE](https://incore.ncsa.illinois.edu/), a community resilience modeling environment.\n",
    "Using IN-CORE requires an account and access to the IN-CORE Dataservice.\n",
    "\n",
    "Functions are provided to obtain and clean data required for the version 2 Housing Unit Allocation. \n",
    "\n",
    "## Required Inputs\n",
    "Program requires the following inputs:\n",
    "If using the National Structures Inventory there are no required inputs.\n",
    "    \n",
    "## Output Description\n",
    "The output of this workflow is a CSV file with the housing unit inventory allocated to a building inventory using the housing unit allocation model.\n",
    "\n",
    "The output CSV is designed to be used in the Interdependent Networked Community Resilience Modeling Environment (IN-CORE).\n",
    "\n",
    "IN-CORE is an open source python package that can be used to model the resilience of a community. To download IN-CORE, see:\n",
    "\n",
    "https://incore.ncsa.illinois.edu/\n",
    "\n",
    "\n",
    "## Instructions\n",
    "Users can run the workflow by executing each block of code in the notebook."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description of Program\n",
    "- program:    ncoda_07fv1_HUA_PREC_NSI\n",
    "- task:       Start with NSI building inventory, run housing unit allocation algorithm, and then run person record file algorithm\n",
    "- See github commits for description of program updates\n",
    "- Current Version: v1 - \n",
    "- 2024-02-20 - Combine code from 07c, 07d, and 07e into one notebook\n",
    "- 2024-05-22 - removed the drop down menu, did not work consistently\n",
    "- project:    Interdependent Networked Community Resilience Modeling Environment (IN-CORE), Subtask 5.2 - Social Institutions\n",
    "- funding:\t  NIST Financial Assistance Award Numbers: 70NANB15H044 and 70NANB20H008 \n",
    "- author:     Nathanael Rosenheim\n",
    "\n",
    "## Required Citations:\n",
    "Rosenheim, Nathanael, Roberto Guidotti, Paolo Gardoni & Walter Gillis Peacock. (2021). Integration of detailed household and housing unit characteristic data with critical infrastructure for post-hazard resilience modeling. _Sustainable and Resilient Infrastructure_. 6(6), 385-401. https://doi.org/10.1080/23789689.2019.1681821\n",
    "\n",
    "Rosenheim, Nathanael (2021) “Detailed Household and Housing Unit Characteristics: Data and Replication Code.” _DesignSafe-CI_. \n",
    "https://doi.org/10.17603/ds2-jwf6-s535."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To reload submodules need to use this magic command to set autoreload on\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from pyncoda.ncoda_00g_community_options import *\n",
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to set up the Community Dictionary\n",
    "Please review the python code in the file pyncoda/ncoda_00g_community_options.py\n",
    "\n",
    "In this file you will find a collection of data dictionaries with various ways to setup the inputs for the Housing Unit Allocation process. \n",
    "\n",
    "The basic dictionary includes the name of the community, the county FIPS code, your input building inventory file, and key variables in the building inventory file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select a community from this list\n",
    "# if your community is not in this list, add it to the file ncoda_00g_community_options.py\n",
    "list_community_options(communities_dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "community_id_by_name = 'Brazos County, TX: NSI Building inventory for Brazos County, TX'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "community_id, focalplace, countyname, countyfips = get_community_id_by_name(community_id_by_name)\n",
    "communities = {community_id : communities_dictionary[community_id]}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Python Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import geopandas as gpd # For reading in shapefiles\n",
    "import numpy as np\n",
    "import sys # For displaying package versions\n",
    "import os # For managing directories and file paths if drive is mounted\n",
    "import scooby # Reports Python environment\n",
    "\n",
    "import contextily as cx # For adding basemap tiles to plot\n",
    "import matplotlib.pyplot as plt # For plotting and making graphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# open, read, and execute python program with reusable commands\n",
    "from pyncoda.ncoda_00d_cleanvarsutils import *\n",
    "from pyncoda.ncoda_04c_poptableresults import *\n",
    "from pyncoda.ncoda_07i_process_communities import process_community_workflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate report of Python environment\n",
    "base_packages = ['pandas','ipyleaflet','seaborn','contextily']\n",
    "incore_packages = ['pyincore','pyincore_viz']\n",
    "check_packages = base_packages + incore_packages\n",
    "print(scooby.Report(additional=check_packages))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check working directory - good practice for relative path access\n",
    "os.getcwd()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Housing Unit Allocation\n",
    "The following code will produce the following outputs:\n",
    "1. Housing Unit Inventory\n",
    "2. Address Point Inventory\n",
    "3. Housing Unit Allocation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow = process_community_workflow(communities)\n",
    "hua_hui_gdf = workflow.process_communities()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Person Record File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "version = '3.0.0'\n",
    "version_text = 'v3-0-0'\n",
    "\n",
    "# open, read, and execute python program with reusable commands\n",
    "from pyncoda.ncoda_07e_generate_prec import generate_prec_functions\n",
    "\n",
    "# Save Outputfolder - due to long folder name paths output saved to folder with shorter name\n",
    "# files from this program will be saved with the program name - \n",
    "# this helps to follow the overall workflow\n",
    "outputfolder = \"OutputData\"\n",
    "# Make directory to save output\n",
    "if not os.path.exists(outputfolder):\n",
    "    os.mkdir(outputfolder)\n",
    "\n",
    "# Set random seed for reproducibility\n",
    "seed = 1000\n",
    "basevintage = 2010\n",
    "\n",
    "generate_prec_df = generate_prec_functions(\n",
    "                    communities =   communities,\n",
    "                    seed =          seed,\n",
    "                    version =       version,\n",
    "                    version_text=   version_text,\n",
    "                    basevintage=    basevintage,\n",
    "                    outputfolder=   outputfolder\n",
    "                    )\n",
    "\n",
    "prec_df = generate_prec_df.generate_prec_v300()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore and Validate Housing Unit Allocation\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look at population characteristics and compare to US Census"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "where = communities[community_id]['community_name']\n",
    "print(where, focalplace, countyname, countyfips)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add race ethnicity to data frame for better map legends\n",
    "hua_hui_race_gdf = PopResultsTable.add_race_ethnicity_to_pop_df(hua_hui_gdf)\n",
    "hua_hui_race_gdf = PopResultsTable.add_hhinc_df(hua_hui_gdf)\n",
    "\n",
    "# add category for missing building id\n",
    "bldg_uniqueid = communities[community_id]['building_inventory']['bldg_uniqueid']\n",
    "# add category for missing building id\n",
    "buildingdata_conditions = {'cat_var' : {'variable_label' : 'Building Data Availability',\n",
    "                         'notes' : 'Does Housing Unit have building data?'},\n",
    "              'condition_list' : {\n",
    "                1 : {'condition': f\"(df['{bldg_uniqueid}'] == 'missing building id')\", 'value_label': \"0 Missing Building Data\"},\n",
    "                2 : {'condition': f\"(df['{bldg_uniqueid}'] != 'missing building id')\", 'value_label': \"1 Building Data Available\"}}\n",
    "            }\n",
    "hua_hui_gdf = add_label_cat_conditions_df(hua_hui_gdf, conditions = buildingdata_conditions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hua_hui_gdf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set dataframe for focal place\n",
    "focalplace_hua_hui_gdf =  hua_hui_gdf.loc[hua_hui_gdf['placeNAME10'] == focalplace].copy(deep=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PopResultsTable.pop_results_table(\n",
    "                  input_df = hua_hui_gdf, \n",
    "                  who = \"Total Population by Households\", \n",
    "                  what = \"by Race, Ethnicity\",\n",
    "                  where = countyname,\n",
    "                  when = \"2010\",\n",
    "                  row_index = \"Race Ethnicity\",\n",
    "                  col_index = 'Tenure Status')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PopResultsTable.pop_results_table(\n",
    "                  input_df = hua_hui_gdf, \n",
    "                  who = \"Total Population by Households\", \n",
    "                  what = \"by Race, Ethnicity\",\n",
    "                  where = countyname,\n",
    "                  when = \"2010\",\n",
    "                  row_index = \"Race Ethnicity\",\n",
    "                  col_index = 'Hispanic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PopResultsTable.pop_results_table(\n",
    "                  input_df = prec_df, \n",
    "                  who = \"Total Population by Persons\", \n",
    "                  what = \"by Race, Ethnicity\",\n",
    "                  where = countyname,\n",
    "                  when = \"2010\",\n",
    "                  row_index = \"Race\",\n",
    "                  col_index = 'Hispanic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PopResultsTable.pop_results_table(hua_hui_gdf, \n",
    "                  who = \"Total Population by Households\", \n",
    "                  what = \"by Race, Ethnicity\",\n",
    "                  where = where,\n",
    "                  when = \"2010\",\n",
    "                  row_index = \"Race Ethnicity\",\n",
    "                  col_index = 'Building Data Availability_str',\n",
    "                  row_percent = '0 Missing Building Data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    print(\"Attempting to generate the population results table...\")\n",
    "    table1 = PopResultsTable.pop_results_table(\n",
    "        focalplace_hua_hui_gdf,\n",
    "        who=\"Total Population by Households\",\n",
    "        what=\"by Tenure\",\n",
    "        where=focalplace,\n",
    "        when=\"2010\",\n",
    "        row_index=\"Tenure Status\",\n",
    "        col_index=\"Building Data Availability_str\",\n",
    "        row_percent=\"0 Missing Building Data\"\n",
    "    )\n",
    "    print(\"Population results table generated successfully.\")\n",
    "except Exception as e:\n",
    "    table1 = \"no table generated\"\n",
    "    print(f\"No Missing Building Data: {e}\")\n",
    "\n",
    "table1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PopResultsTable.pop_results_table(hua_hui_gdf, \n",
    "                   who = \"Median Household Income\", \n",
    "                  what = \"by Race, Ethnicity\",\n",
    "                  where = where,\n",
    "                  when = \"2010\",\n",
    "                  row_index = \"Race Ethnicity\",\n",
    "                  col_index = 'Tenure Status')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PopResultsTable.pop_results_table(focalplace_hua_hui_gdf, \n",
    "                   who = \"Median Household Income\", \n",
    "                  what = \"by Race, Ethnicity\",\n",
    "                  where = focalplace,\n",
    "                  when = \"2010\",\n",
    "                  row_index = \"Race Ethnicity\",\n",
    "                  col_index = 'Tenure Status')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Validate the Housing Unit Allocation has worked\n",
    "Notice that the population count totals for the community\n",
    "should match (pretty closely) data collected for the 2010 Decennial Census.\n",
    "This can be confirmed by going to data.census.gov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Total Population by Race and Ethnicity:\")\n",
    "print(f\"https://data.census.gov/cedsci/table?g=050XX00US{countyfips}&tid=DECENNIALSF12010.P5\")\n",
    "\n",
    "print(\"Median Income by Race and Ethnicity:\")\n",
    "print(f\"All Households: https://data.census.gov/cedsci/table?g=050XX00US{countyfips}&tid=ACSDT5Y2012.B19013\")\n",
    "print(f\"Black Households: https://data.census.gov/cedsci/table?g=050XX00US{countyfips}&tid=ACSDT5Y2012.B19013B\")\n",
    "print(f\"White, not Hispanic Households: https://data.census.gov/cedsci/table?g=050XX00US{countyfips}&tid=ACSDT5Y2012.B19013H\")\n",
    "print(f\"Hispanic Households: https://data.census.gov/cedsci/table?g=050XX00US{countyfips}&tid=ACSDT5Y2012.B19013I\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Differences in the housing unit allocation and the Census count may be due to differences between political boundaries and the building inventory. See Rosenheim et al 2019 for more details.\n",
    "\n",
    "The housing unit allocation, plus the building results will become the input for the social science models such as the population dislocation model."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore Maps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#mapname = 'hhincdotmap'\n",
    "mapname = 'hhracedotmap'\n",
    "# Map column\n",
    "#map_var = 'Household Income Group'\n",
    "map_var = 'Race Ethnicity'\n",
    "place = focalplace\n",
    "\n",
    "condition1 = \"(hua_hui_race_gdf.race >= 1)\"\n",
    "condition2 = f\"(hua_hui_race_gdf.placeNAME10 == '{place}')\"\n",
    "conditions = f\"{condition1} & {condition2}\"\n",
    "\n",
    "county_hua_gdf = hua_hui_race_gdf.loc[eval(condition1)].copy(deep=True)\n",
    "county_hua_gdf = county_hua_gdf.to_crs(epsg=4326)\n",
    "focal_place_hua_gdf = hua_hui_race_gdf.loc[eval(conditions)].copy(deep=True)\n",
    "focal_place_hua_gdf = focal_place_hua_gdf.to_crs(epsg=4326)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyncoda.ncoda_04b_foliummaps import *\n",
    "# plot png file\n",
    "from IPython.display import Image\n",
    "\n",
    "bldg_inv_id = communities[community_id]['building_inventory']['id']\n",
    "outputfolder = 'OutputData'\n",
    "community = communities[community_id]['community_name']\n",
    "\n",
    "county_map = plot_dotmap_map(gdf=county_hua_gdf,\n",
    "                        mapname=mapname,\n",
    "                        map_var=map_var,\n",
    "                        bldg_inv_id=bldg_inv_id,\n",
    "                        community=community_id,\n",
    "                        place = community,\n",
    "                        outputfolder=outputfolder,\n",
    "                        condition_id = \"1\",\n",
    "                        basemap_source = cx.providers.CartoDB.Positron)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get xlim and ylim for focal place\n",
    "xlim = [focal_place_hua_gdf.total_bounds[0], focal_place_hua_gdf.total_bounds[2]]\n",
    "ylim = [focal_place_hua_gdf.total_bounds[1], focal_place_hua_gdf.total_bounds[3]]\n",
    "print(xlim, ylim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "focal_place_map = plot_dotmap_map(gdf=county_hua_gdf,\n",
    "                        mapname=mapname,\n",
    "                        map_var=map_var,\n",
    "                        bldg_inv_id=bldg_inv_id,\n",
    "                        community=community_id,\n",
    "                        place = focalplace,\n",
    "                        outputfolder=outputfolder,\n",
    "                        condition_id = \"2\",\n",
    "                        basemap_source = cx.providers.CartoDB.Positron,\n",
    "                        xlim = xlim,\n",
    "                        ylim = ylim,\n",
    "                        focal_gdf = focal_place_hua_gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(focal_place_map+'.png', width= 800, height=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore by Income"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapname = 'hhincdotmap'\n",
    "\n",
    "# Map column\n",
    "map_var = 'Household Income Group'\n",
    "\n",
    "# Ensure 'hhinc' is numeric\n",
    "hua_hui_race_gdf['hhinc'] = pd.to_numeric(hua_hui_race_gdf['hhinc'], errors='coerce')\n",
    "\n",
    "condition1 = \"(hua_hui_race_gdf.hhinc >= 1)\"\n",
    "\n",
    "county_hua_gdf = hua_hui_race_gdf.loc[eval(condition1)].copy(deep=True)\n",
    "county_hua_gdf = county_hua_gdf.to_crs(epsg=4326)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get xlim and ylim for focal place\n",
    "xlim = [focal_place_hua_gdf.total_bounds[0], focal_place_hua_gdf.total_bounds[2]]\n",
    "ylim = [focal_place_hua_gdf.total_bounds[1], focal_place_hua_gdf.total_bounds[3]]\n",
    "\n",
    "focal_place_map = plot_dotmap_map(gdf=county_hua_gdf,\n",
    "                        mapname=mapname,\n",
    "                        map_var=map_var,\n",
    "                        bldg_inv_id=bldg_inv_id,\n",
    "                        community=community_id,\n",
    "                        place = focalplace,\n",
    "                        outputfolder=outputfolder,\n",
    "                        condition_id = \"2\",\n",
    "                        basemap_source = cx.providers.CartoDB.Positron,\n",
    "                        xlim = xlim,\n",
    "                        ylim = ylim,\n",
    "                        focal_gdf = focal_place_hua_gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(focal_place_map+'.png', width= 800, height=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View Codebook\n",
    "The Housing Unit Allocation methodology generates a codebook for the housing unit inventory.\n",
    "\n",
    "Look in the OutputData folder to find the codebook."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "IN-CORE_1dv1_Lumberton_CleanLODESdata_2021-05-06.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "pyincoreEnv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "71c2a55d887d60c7c9ec28a057f4ab524298d1d6ada186bb9b26cd7328acad59"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}