Chapter 14 - Linear Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14. Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Regression** is a method for studying the relationship between a **response variable** $Y$ and a **covariates** $X$.  The covariate is also called a **predictor variable** or **feature**.  Later we will generalize and allow for more than one covariate.  The data are of the form\n",
    "\n",
    "$$ (Y_1, X_1), \\dots, (Y_n, X_n) $$\n",
    "\n",
    "One way to summarize the relationship between $X$ and $Y$ is through the **regression function**\n",
    "\n",
    "$$ r(x) = \\mathbb{E}(Y | X = x) = \\int y f(y | x) dy $$\n",
    "\n",
    "Most of this chapter is concerned with estimating the regression function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.1 Simple Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest version of regression is when $X_i$ is simple (a scalar, not a vector) and $r(x)$ is assumed to be linear:\n",
    "\n",
    "$$r(x) = \\beta_0 + \\beta_1 x$$\n",
    "\n",
    "This model is called the **simple linear regression model**.  Let $\\epsilon_i = Y_i - (\\beta_0 + \\beta_1 X_i)$.  Then:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{E}(\\epsilon_i | Y_i) &= \\mathbb{E}(Y_i - (\\beta_0 + \\beta_1 X_i) | X_i)\\\\\n",
    "&= \\mathbb{E}(Y_i | X_i) - (\\beta_0 + \\beta_1 X_i)\\\\\n",
    "&= r(X_i) - (\\beta_0 + \\beta_1 X_i)\\\\\n",
    "&= 0\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Let $\\sigma^2(x) = \\mathbb{V}(\\epsilon_i | X_i = x)$.  We will make the further simplifying assumption that $\\sigma^2(x) = \\sigma^2$ does not depend on $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The Linear Regression Model**\n",
    "\n",
    "$$Y_i = \\beta_0 + \\beta_1 X_i + \\epsilon_i$$\n",
    "\n",
    "where $\\mathbb{E}(\\epsilon_i | X_i) = 0$ and $\\mathbb{V}(\\epsilon_i | X_i) = \\sigma^2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unknown models in the parameter are the intercept $\\beta_0$, the slope $\\beta_1$ and the variance $\\sigma^2$.  Let $\\hat{\\beta_0}$ and $\\hat{\\beta_1}$ denote the estimates of $\\beta_0$ and $\\beta_1$.  The **fitted line** is defined to be\n",
    "\n",
    "$$\\hat{r}(x) = \\hat{\\beta}_0 + \\hat{\\beta}_1 x$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **predicted values** or **fitted values** are $\\hat{Y}_i = \\hat{r}(X_i)$ and the **residuals** are defined to be\n",
    "\n",
    "$$\\hat{\\epsilon}_i = Y_i - \\hat{Y}_i = Y_i - (\\hat{\\beta}_0 + \\hat{\\beta}_1 X_i)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **residual sum of squares** or RSS is defined by\n",
    "\n",
    "$$ \\text{RSS} = \\sum_{i=1}^n \\hat{\\epsilon}_i^2$$\n",
    "\n",
    "The quantity RSS measures how well the fitted line fits the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **least squares estimates** are the values $\\hat{\\beta}_0$ and $\\hat{\\beta}_1$ that minimize $\\text{RSS} = \\sum_{i=1}^n \\hat{\\epsilon}_i^2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 14.4**.  The least square estimates are given by\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\hat{\\beta}_1 &= \\frac{\\sum_{i=1}^n (X_i - \\overline{X}_n) (Y_i - \\overline{Y}_n)}{\\sum_{i=1}^n (X_i - \\overline{X}_n)^2}\\\\\n",
    "\\hat{\\beta}_0 &= \\overline{Y}_n - \\hat{\\beta}_1 \\overline{X}_n\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "An unbiased estimate of $\\sigma^2$ is\n",
    "\n",
    "$$\n",
    "\\hat{\\sigma}^2 = \\left( \\frac{1}{n - 2} \\right) \\sum_{i=1}^n \\hat{\\epsilon}_i^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.2 Least Squares and Maximum Likelihood"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we add the assumption that $\\epsilon_i | X_i \\sim N(0, \\sigma^2)$, that is,\n",
    "\n",
    "$Y_i | X_i \\sim N(\\mu_i, \\sigma_i^2)$\n",
    "\n",
    "where $\\mu_i = \\beta_0 + \\beta_i X_i$.  The likelihood function is\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\prod_{i=1}n f(X_i, Y_i) &= \\prod_{i=1}^n f_X(X_i) f_{Y|X}(Y_i | X_i)\\\\\n",
    "&= \\prod_{i=1}^n f_X(X_i) \\times \\prod_{i=1}^n f_{Y|X}(Y_i | X_i) \\\\\n",
    "&= \\mathcal{L}_1 \\times \\mathcal{L}_2\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "where $\\mathcal{L}_1 = \\prod_{i=1}^n f_X(X_i)$ and $\\mathcal{L}_2 = \\prod_{i=1}^n f_{Y|X}(Y_i | X_i)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The term $\\mathcal{L}_1$ does not involve the parameters $\\beta_0$ and $\\beta_1$.  We shall focus on the second term $\\mathcal{L}_2$ which is called the **conditional likelihood**, given by\n",
    "\n",
    "$$\\mathcal{L}_2 \\equiv \\mathcal{L}(\\beta_0, \\beta_1, \\sigma)\n",
    "= \\prod_{i=1}^n f_{Y|X}(Y_i | X_i)\n",
    "\\propto \\sigma^{-n} \\exp \\left\\{ - \\frac{1}{2 \\sigma^2} \\sum_i (Y_i - \\mu_i)^2 \\right\\}\n",
    "$$\n",
    "\n",
    "The conditional log-likelihood is\n",
    "\n",
    "$$\\ell(\\beta_0, \\beta_1, \\sigma) = -n \\log \\sigma - \\frac{1}{2 \\sigma^2} \\sum_{i=1}^n \\left(Y_i - (\\beta_0 + \\beta_1 X_i) \\right)^2$$\n",
    "\n",
    "To find the MLE of $(\\beta_0, \\beta_1)$ we maximize the conditional log likelihood. We can see from the equation above that this is the same as minimizing the RSS.  Therefore, we have shown the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 14.7**.  Under the assumption of Normality, the least squares estimator is also the maximum likelihood estimator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also maximize $\\ell(\\beta_0, \\beta_1, \\sigma)$ over $\\sigma$ yielding the MLE\n",
    "\n",
    "$$ \\hat{\\sigma}^2 = \\frac{1}{n} \\sum_{i=1}^n \\hat{\\epsilon}_i^2 $$\n",
    "\n",
    "This estimator is similar to, but not identical to, the unbiased estimator.  Common practice is to use the unbiased estimator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.3 Properties of the Least Squares Estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 14.8**.  Let $\\hat{\\beta}^T = (\\hat{\\beta}_0, \\hat{\\beta}_1)^T$ denote the least squares estimators.  Then,\n",
    "\n",
    "$$\n",
    "\\mathbb{E}(\\hat{\\beta} | X^n) = \\begin{pmatrix}\\beta_0 \\\\ \\beta_1 \\end{pmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\mathbb{V}(\\hat{\\beta} | X^n) = \\frac{\\sigma^2}{n s_X^2} \\begin{pmatrix} \n",
    "\\frac{1}{n} \\sum_{i=1}^n X_i^2 & -\\overline{X}_n \\\\\n",
    "-\\overline{X}_n & 1\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "where $s_X^2 = n^{-1} \\sum_{i=1}^n (X_i - \\overline{X}_n)^2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimated standard errors of $\\hat{\\beta}_0$ and $\\hat{\\beta}_1$ are obtained by taking the square roots of the corresponding diagonal terms of $\\mathbb{V}(\\hat{\\beta} | X^n)$ and inserting the estimate $\\hat{\\sigma}$ for $\\sigma$.  Thus,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\hat{\\text{se}}(\\hat{\\beta}_0) &= \\frac{\\hat{\\sigma}}{s_X \\sqrt{n}} \\sqrt{\\frac{\\sum_{i=1}^n X_i^2}{n}}\\\\\n",
    "\\hat{\\text{se}}(\\hat{\\beta}_1) &= \\frac{\\hat{\\sigma}}{s_X \\sqrt{n}}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "We should write $\\hat{\\text{se}}(\\hat{\\beta}_0 | X^n)$ and $\\hat{\\text{se}}(\\hat{\\beta}_1 | X^n)$ but we will use the shorter notation $\\hat{\\text{se}}(\\hat{\\beta}_0)$ and $\\hat{\\text{se}}(\\hat{\\beta}_1)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 14.9**. Under appropriate conditions we have:\n",
    "\n",
    "1. (Consistency) $\\hat{\\beta}_0 \\xrightarrow{\\text{P}} \\beta_0$ and $\\hat{\\beta}_1 \\xrightarrow{\\text{P}} \\beta_1$\n",
    "\n",
    "2. (Asymptotic Normality):\n",
    "\n",
    "$$\n",
    "\\frac{\\hat{\\beta}_0 - \\beta_0}{\\hat{se}(\\hat{\\beta}_0)} \\leadsto N(0, 1)\n",
    "\\quad \\text{and} \\quad\n",
    "\\frac{\\hat{\\beta}_1 - \\beta_1}{\\hat{se}(\\hat{\\beta}_1)} \\leadsto N(0, 1)\n",
    "$$\n",
    "\n",
    "3. Approximate $1 - \\alpha$ confidence intervals for $\\beta_0$ and $\\beta_1$ are\n",
    "\n",
    "$$\n",
    "\\hat{\\beta}_0 \\pm z_{\\alpha/2} \\hat{\\text{se}}(\\hat{\\beta}_0)\n",
    "\\quad \\text{and} \\quad\n",
    "\\hat{\\beta}_1 \\pm z_{\\alpha/2} \\hat{\\text{se}}(\\hat{\\beta}_1)\n",
    "$$\n",
    "\n",
    "The Wald statistic for testing $H_0 : \\beta_1 = 0$ versus $H_1: \\beta_1 \\neq 0$ is: reject $H_0$ if $W > z_{\\alpha / 2}$ where $W = \\hat{\\beta}_1 / \\hat{\\text{se}}(\\hat{\\beta}_1)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.4 Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we have estimated a regression model $\\hat{r}(x) = \\hat{\\beta}_0 + \\hat{\\beta}_1 x$ from data $(X_1, Y_1), \\dots, (X_n, Y_n)$.  We observe the value $X_* = x$ of the covariate for a new subject and we want to predict the outcome $Y_*$.  An estimate of $Y_*$ is \n",
    "\n",
    "$$ \\hat{Y}_* = \\hat{\\beta}_0 + \\hat{\\beta}_1 X_*$$\n",
    "\n",
    "Using the formula for the variance of the sum of two random variables,\n",
    "\n",
    "$$ \\mathbb{V}(\\hat{Y}_*) = \\mathbb{V}(\\hat{\\beta}_0 + \\hat{\\beta}_1 x_*) = \\mathbb{V}(\\hat{\\beta}_0) + x_* \\mathbb{V}(\\hat{\\beta}_1) + 2 x_* \\text{Cov}(\\hat{\\beta}_0, \\hat{\\beta}_1) $$ \n",
    "\n",
    "Theorem 14.8 gives the formulas for all terms in this equation.  The estimated standard error $\\hat{\\text{se}}(\\hat{Y}_*)$ is the square root of this variance, with $\\hat{\\sigma}^2$ in place of $\\sigma^2$.  However, **the confidence interval for $\\hat{Y}_*$ is not of the usual form** $\\hat{Y}_* \\pm z_{\\alpha} \\hat{\\text{se}}(\\hat{Y}_*)$.  The appendix explains why.  The correct form is given in the following theorem.  We can the interval a **prediction interval**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 14.11 (Prediction Interval)**.  Let\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\hat{\\xi}_n^2 &= \\hat{\\text{se}}^2(\\hat{Y}_*) + \\hat{\\sigma}^2 \\\\\n",
    "&= \\hat{\\sigma}^2 \\left(\\frac{\\sum_{i=1}^n (X_i - X_*)^2}{n \\sum_{i=1}^n (X_i - \\overline{X})^2} + 1 \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "An approximate $1 - \\alpha$ prediction interval for $Y_*$ is\n",
    "\n",
    "$$ \\hat{Y}_* \\pm z_{\\alpha/2} \\xi_n$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.5 Multiple Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now suppose we have $k$ covariates $X_1, \\dots, X_k$.  The data are of the form\n",
    "\n",
    "$$(Y_1, X_1), \\dots, (Y_n, X_n)$$\n",
    "\n",
    "where\n",
    "\n",
    "$$ X_i = (X_{i1}, \\dots, X_{ik}) $$\n",
    "\n",
    "Here, $X_i$ is the vector of $k$ covariates for the $i-$th observation.  The linear regression model is\n",
    "\n",
    "$$ Y_i = \\sum_{i=1}^k \\beta_j X_{ij} + \\epsilon_i $$\n",
    "\n",
    "for $i = 1, \\dots, n$ where $\\mathbb{E}(\\epsilon_i | X_{1i}, \\dots, X_{ik}) = 0$.  Usually we want to include an intercept in the model which we can do by setting $X_{i1} = 1$ for $i = 1, \\dots, n$.  At this point it will become more convenient to express the model in matrix notation.  The outcomes will be denoted by\n",
    "\n",
    "$$ Y = \\begin{pmatrix}\n",
    "Y_1 \\\\\n",
    "Y_2 \\\\\n",
    "\\vdots \\\\\n",
    "Y_n\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "and the covariates will be denoted by\n",
    "\n",
    "$$ X = \\begin{pmatrix}\n",
    "X_{11} & X_{12} & \\cdots & X_{1k} \\\\\n",
    "X_{21} & X_{22} & \\cdots & X_{2k} \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "X_{n1} & X_{n2} & \\cdots & X_{nk}\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "Each row is one observation; the columns represent to the $k$ covariates.  Thus, $X$ is a $n \\times k$ matrix.  Let\n",
    "\n",
    "$$\n",
    "\\beta = \\begin{pmatrix}\n",
    "\\beta_1 \\\\\n",
    "\\vdots \\\\\n",
    "\\beta_k\n",
    "\\end{pmatrix}\n",
    "\\quad \\text{and} \\quad\n",
    "\\epsilon = \\begin{pmatrix}\n",
    "\\epsilon_1 \\\\\n",
    "\\vdots \\\\\n",
    "\\epsilon_k\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "Then we can write the linear regression model as\n",
    "\n",
    "$$ Y = X \\beta + \\epsilon $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 14.13**. Assuming that the $k \\times k$ matrix $X^TX$ is invertible, the least squares estimate is\n",
    "\n",
    "$$ \\hat{\\beta} = (X^T X)^{-1} X^T Y $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimated regression function is\n",
    "\n",
    "$$ \\hat{r}(x) = \\sum_{j=1}^k \\hat{\\beta}_j x_j$$\n",
    "\n",
    "The variance-covariance matrix of $\\hat{\\beta}$ is\n",
    "\n",
    "$$ \\mathbb{V}(\\hat{\\beta} | X^n) = \\sigma^2 (X^T X)^{-1} $$\n",
    "\n",
    "An unbiased estimate of $\\sigma^2$ is\n",
    "\n",
    "$$ \\hat{\\sigma}^2 = \\left( \\frac{1}{n - k} \\right) \\sum_{i=1}^n \\hat{\\epsilon}_i^2 $$\n",
    "\n",
    "where $\\hat{\\epsilon} = X \\hat{\\beta} - Y$ is the vector of residuals.  An approximate $1 - \\alpha$ confidence interval for $\\beta_j$ is\n",
    "\n",
    "$$ \\hat{\\beta}_j \\pm z_{\\alpha/2} \\hat{\\text{se}}(\\hat{\\beta}_j) $$\n",
    "\n",
    "where $\\hat{\\text{se}}^2(\\hat{\\beta}_j)$ is the $j$-th diagonal element of the matrix $\\hat{\\sigma}^2 (X^T X)^{-1}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.6 Model Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may have data on many covariates but we may not want to include all of them in the model.  A smaller model with fewer covariates has two advantages: it might give better predictions than a big model and it is more parsimonious (simpler).  Generally, as you add more variables to a regression, the bias of the predictions decreases and the variance increases.  Too few covariates yields high bias; too many covariates yields high variance.  Good predictions result from achieving a good balance between bias and variance.\n",
    "\n",
    "In model selection there are two problems: assigning a score to each model which measures, in some sense, how good the model is, and searching through all models to find the model with the best score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $S \\subset \\{1, \\dots, k\\}$ and let $\\mathcal{X}_S = \\{ X_j : j \\in S \\}$ denote a subset of the covariates.  Let $\\beta_S$ denote the coefficients of the corresponding set of covariates and let $\\hat{\\beta}_S$ denote the least squares estimate of $\\beta_S$.  Let $X_S$ denote the $X$ matrix for this subset of covariates, and let $\\hat{r}_S(x)$ to be the estimated regression function.  The predicted values from model $S$ are denoted by $\\hat{Y}_i(S) = \\hat{r}_S(X_i)$.  \n",
    "\n",
    "The **prediction risk** is defined to be\n",
    "\n",
    "$$R(S) = \\sum_{i=1}^n \\mathbb{E} (\\hat{Y}_i(S) - Y_i^*)^2 $$\n",
    "\n",
    "where $Y_i^*$ denotes the value of the future observation of $Y_i$ at covariate value $X_i$.  Our goal is to choose $S$ to make $R(S)$ small.\n",
    "\n",
    "The **training error** is defined to be\n",
    "\n",
    "$$\\hat{R}_\\text{tr}(S) = \\sum_{i=1}^n (\\hat{Y}_i(S) - Y_i)^2 $$\n",
    "\n",
    "This estimate is very biased and under-estimates $R(S)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 14.15**.  The training error is a downward biased estimate of the prediction risk:\n",
    "\n",
    "$$ \\mathbb{E}(\\hat{R}_\\text{tr}(S)) < R(S) $$\n",
    "\n",
    "In fact,\n",
    "\n",
    "$$\\text{bias}(\\hat{R}_\\text{tr}(S)) = \\mathbb{E}(\\hat{R}_\\text{tr}(S)) - R(S) = -2 \\sum_{i=1}^n \\text{Cov}(\\hat{Y}_i, Y_i)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason for the bias is that the data is being used twice: to estimate the parameters and to estimate the risk.  When fitting a model with many variables, the covariance $\\text{Cov}(\\hat{Y_i}, Y_i)$ will be large and the bias of the training error gets worse.\n",
    "\n",
    "In summary, the training error is a poor estimate of risk.  Here are some better estimates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Mallow's $C_p$ statistic** is defined by\n",
    "\n",
    "$$\\hat{R}(S) = \\hat{R}_\\text{tr}(S) + 2 |S| \\hat{\\sigma}^2$$\n",
    "\n",
    "where $|S|$ denotes the number of terms in $S$ and $\\hat{\\sigma}^2$ is the estimate of $\\sigma^2$ obtained from the full model (with all covariates).  Think of the $C_p$ statistic as lack of fit plus complexity penalty."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A related method for estimating risk is **AIC (Akaike Information Criterion)**.  The idea is to choose $S$ to maximize\n",
    "\n",
    "$$ \\ell_S - |S|$$\n",
    "\n",
    "where $\\ell_S$ is the log-likelihood of the model evaluated at the MLE.  In linear regression with Normal errors, maximizing AIC is equivalent to minimizing Mallow's $C_p$; see exercise 8.\n",
    "\n",
    "*Some texts use a slightly different definition of AIC which involves multiplying this definition by 2 or -2.  This has no effect on which model is selected.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yet another method for estimating risk is **leave-one-out cross-validation**.  In this case, the risk estimator is\n",
    "\n",
    "$$\\hat{R}_\\text{CV}(S) = \\sum_{i=1}^n (Y_i - \\hat{Y}_{(i)})^2 $$\n",
    "\n",
    "where $\\hat{Y}_{(i)}$ is the prediction for $Y_i$ obtained by fitting the model with $Y_i$ omitted.  It can be shown that\n",
    "\n",
    "$$\\hat{R}_\\text{CV}(S) = \\sum_{i=1}^n \\left( \\frac{Y_i - \\hat{Y}_i(S)}{1 - U_{ii}(S)} \\right)^2 $$\n",
    "\n",
    "where $U_ii$ is the $i$-th diagonal element of the matrix\n",
    "\n",
    "$$U(S) = X_S (X_S^T X_S)^{-1} X_S^T$$\n",
    "\n",
    "Thus one need not actually drop each observation and re-fit the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A generalization is **k-fold cross-validation**.  Here we divide the data into $k$ groups; often people take $k = 10$.  We omit one group of data and fit the models on the remaining data.  We use the fitted model to predict the data in the group that was omitted.  We then estimate the risk by $\\sum_i (Y_i - \\hat{Y}_i)^2$ where the sum is over the data points in the omitted group.  This process is repeated for each of the $k$ groups and the resulting risk estimates are averaged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For linear regression, Mallows $C_p$ and cross-validation often yield essentially the same results so one might as well use Mallow's method.  In some of the more complex problems we will discuss later, cross-validation will be more useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another scoring method is **BIC (Bayesian Information Criterion)**.  Here we choose a model to maximize\n",
    "\n",
    "$$ \\text{BIC}(S) = \\text{RSS}(S) = 2 |S| \\hat{\\sigma}^2 $$\n",
    "\n",
    "The BIC score has a Bayesian interpretation.  Let $\\mathcal{S} = \\{ S_1, \\dots, S_m \\}$ denote a set of models.  Suppose we assign the uniform prior $\\mathbb{P}(S_j) = 1 / m$ over the models.  Also assume we put a smooth prior on the parameters within each model.  It can be shown that the posterior probability for a model is approximately\n",
    "\n",
    "$$ \\mathbb{P}(S_j | \\text{data}) \\approx \\frac{e^{\\text{BIC}(S_j)}}{\\sum_r e^{\\text{BIC}(S_r)}}$$\n",
    "\n",
    "so choosing the model with highest BIC is like choosing the model with highest posterior probability.\n",
    "\n",
    "The BIC score also has an information-theoretical interpretation in terms of something called minimum description length.\n",
    "\n",
    "The BIC score is identical to Mallows $C_p$ except that it puts a more severe penalty for complexity.  It thus leads one to choose a smaller model than the other methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there are $k$ covariates then there are $2^k$ possible models. We need to search through all of those models, assign a score to each one, and choose the model with the best score.  When $k$ is large, this is infeasible; in that case, we need to search over a subset of all the models.  Two common methods are **forward and backward stepwise regression**.\n",
    "\n",
    "In forward stepwise regression, we start with no covariates in the model, and keep adding variables one at a time that lead to the best score.  In backward stepwise regression, we start with the biggest model (all covariates) and drop one variable at a time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.7 The Lasso"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method, due to Tibshirani, is called the **Lasso**. Assume that all covariates have been rescaled to have the same variance.  Consider estimating $\\beta = (\\beta_1, \\dots, \\beta_k)$ by minimizing the loss function\n",
    "\n",
    "$$ \\sum_{i=1}^n (Y_i - \\hat{Y}_i)^2 + \\lambda \\sum_{j=1}^k | \\beta_j |$$\n",
    "\n",
    "where $\\lambda > 0$.  The idea is to minimize the sums of squares but there is a penalty that gets large if any $\\beta_j$ gets large.  It can be shown that some of the $\\beta_j$'s will be 0.  We interpret this as having the $j$-th covariate omitted from the model; thus we are doing estimation and model selection simultaneously.\n",
    "\n",
    "We need to choose a value of $\\lambda$.  We can do this by estimating the prediction risk $R(\\lambda)$ as a function of $\\lambda$ and choosing to minimize it.  For example, we can estimate the risk using leave-one-out cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.8 Technical Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The prediction interval is of a different form than other confidence intervals we have seen -- the quantity of interest  $Y_*$ is equal to a parameter $\\theta$ plus a random variable.\n",
    "\n",
    "We can fix this by defining:\n",
    "\n",
    "$$ \\xi_n^2 = \\mathbb{V}(\\hat{Y}_*) + \\sigma^2 = \\left[\\frac{\\sum_i (x_i - x_*)^2}{n \\sum_i (x_i - \\overline{x})^2} + 1\\right] \\sigma^2$$\n",
    "\n",
    "In practice, we substitute $\\hat{\\sigma}$ for $\\sigma$ and we denote the resulting quantity by $\\hat{\\xi}_n$.  Now,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{P}(\\hat{Y}_* - z_{\\alpha/2} \\hat{\\xi}_n < Y_* < \\hat{Y}_* + z_{\\alpha/2} \\hat{\\xi}_n) &=\n",
    "\\mathbb{P}\\left(-z_{\\alpha/2} < \\frac{\\hat{Y}_* - Y_*}{\\hat{\\xi}_n} < z_{\\alpha/2} \\right)\\\\\n",
    "&= \\mathbb{P}\\left(-z_{\\alpha/2} < \\frac{\\hat{\\theta} - \\theta - \\epsilon}{\\hat{\\xi}_n} < z_{\\alpha/2} \\right) \\\\\n",
    "&\\approx \\mathbb{P}\\left(-z_{\\alpha/2} < \\frac{N(0, s^2 + \\sigma^2)}{\\hat{\\xi}_n} < z_{\\alpha/2} \\right)  \\\\\n",
    "&\\approx \\mathbb{P}\\left(-z_{\\alpha/2} < \\frac{N(0, s^2 + \\sigma^2)}{\\xi_n} < z_{\\alpha/2} \\right)  \\\\\n",
    "&= \\mathbb{P}(-z_{\\alpha/2} < N(0, 1) < z_{\\alpha/2}) \\\\\n",
    "&= 1 - \\alpha\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.9 Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.1**.  Prove Theorem 14.4:\n",
    "\n",
    "The least square estimates are given by\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\hat{\\beta}_1 &= \\frac{\\sum_{i=1}^n (X_i - \\overline{X}_n) (Y_i - \\overline{Y}_n)}{\\sum_{i=1}^n (X_i - \\overline{X}_n)^2}\\\\\n",
    "\\hat{\\beta}_0 &= \\overline{Y}_n - \\hat{\\beta}_1 \\overline{X}_n\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "An unbiased estimate of $\\sigma^2$ is\n",
    "\n",
    "$$\n",
    "\\hat{\\sigma}^2 = \\left( \\frac{1}{n - 2} \\right) \\sum_{i=1}^n \\hat{\\epsilon}_i^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.  We can obtain the estimates $\\hat{\\beta}_0$ and $\\hat{\\beta}_1$ by minimizing the RSS -- by taking the partial derivatives with respect to $\\beta_0$ and $\\beta_1$:\n",
    "\n",
    "$$\\text{RSS} = \\sum_i \\hat{\\epsilon}_i^2 = \\sum_i (Y_i - (\\beta_0 + \\beta_1 X_i))^2$$\n",
    "\n",
    "Derivating RSS on $\\beta_0$:\n",
    "\n",
    "$$\\frac{d}{d \\beta_0}\\text{RSS} = \\sum_i \\frac{d}{d \\beta_0} (Y_i - (\\beta_0 + \\beta_1 X_i))^2\n",
    "= \\sum_i 2 (\\beta_0 - (Y_i - \\beta_1 X_i))$$\n",
    "\n",
    "Making this derivative equal to 0 at $\\hat{\\beta}_0$, $\\hat{\\beta}_1$ gives:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "0 &= \\sum_i 2 (\\hat{\\beta}_0 - (Y_i - \\hat{\\beta}_1 X_i))\\\\\n",
    "n \\hat{\\beta}_0 &= n \\sum_i Yi - \\hat{\\beta}_1 n \\sum_i X_i\\\\\n",
    "\\hat{\\beta}_0 &= \\overline{Y}_n - \\hat{\\beta}_1 \\overline{X}_n\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Replacing $\\overline{Y}_n - \\beta_1 \\overline{X}_n$ for $\\beta_0$ and derivating on $\\beta_1$:\n",
    "\n",
    "$$\\frac{d}{d \\beta_1}\\text{RSS} = \\sum_i \\frac{d}{d \\beta_1} (Y_i - (\\beta_0 + \\beta_1 X_i))^2\n",
    "\\sum_i \\frac{d}{d \\beta_1} (Y_i - \\overline{Y}_n - \\beta_1 (X_i - \\overline{X}_n)))^2\n",
    "= \\sum_i -2 (X_i - \\overline{X}_n) (Y_i - \\overline{Y}_n - \\beta_1 (X_i - \\overline{X}_n)))$$\n",
    "\n",
    "Making this derivative equal to 0 at $\\hat{\\beta}_1$ gives:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "0 &= \\sum_i (X_i - \\overline{X}_n) (Y_i - \\overline{Y}_n - \\beta_1 (X_i - \\overline{X}_n))) \\\\\n",
    "0 &= \\hat{\\beta}_1 \\sum_i (\\overline{X}_n - X_i)^2 + \\sum_i (\\overline{X}_n - X_i)(Y_i - \\overline{Y}_n) \\\\\n",
    "\\hat{\\beta}_1 &= \\frac{\\sum_i (X_i - \\overline{X}_n)(Y_i - \\overline{Y}_n)}{\\sum_i (X_i - \\overline{X}_n)^2}\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the unbiased estimate, let's adapt a more general proof from Greene (2003), restricted to $k = 2$ dimensions, where the first dimension is set to all ones to represent the intercept, and the second dimension represents the one-dimensional covariates $X_i$.\n",
    "\n",
    "The vector of least square residuals is\n",
    "\n",
    "$$ \\hat{\\epsilon} = \\begin{pmatrix}\n",
    "\\hat{\\epsilon}_1 \\\\\n",
    "\\hat{\\epsilon}_2 \\\\\n",
    "\\vdots \\\\\n",
    "\\hat{\\epsilon}_n\n",
    "\\end{pmatrix} = \\begin{pmatrix}\n",
    "Y_1 - (\\hat{\\beta}_0 \\cdot 1 + \\hat{\\beta}_1 X_1) \\\\\n",
    "Y_2 - (\\hat{\\beta}_0 \\cdot 1 + \\hat{\\beta}_1 X_2) \\\\\n",
    "\\vdots \\\\\n",
    "Y_n - (\\hat{\\beta}_0 \\cdot 1 + \\hat{\\beta}_1 X_n)\n",
    "\\end{pmatrix} = \n",
    "y - X \\hat{\\beta}\n",
    "$$\n",
    "\n",
    "where $$\n",
    "y = \\begin{pmatrix}\n",
    "Y_1 \\\\\n",
    "Y_2 \\\\\n",
    "\\vdots \\\\\n",
    "Y_n\n",
    "\\end{pmatrix}\n",
    ", \\quad\n",
    "X = \\begin{pmatrix}\n",
    "1 & X_1 \\\\\n",
    "1 & X_2 \\\\\n",
    "\\vdots & \\vdots \\\\\n",
    "1 & X_n\n",
    "\\end{pmatrix},\n",
    "\\quad \\text{and} \\quad\n",
    "\\hat{\\beta} = \\begin{pmatrix}\n",
    "\\hat{\\beta}_0 \\\\\n",
    "\\hat{\\beta}_1\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "The least squares solution can be written as:\n",
    "\n",
    "$$\\hat{\\beta} = (X^T X)^{-1} X^T y$$\n",
    "\n",
    "Replacing it on the definition of $\\hat{\\epsilon}$, we get\n",
    "\n",
    "$$ \\hat{\\epsilon} = y - X (X^T X)^{-1} X^T y = (I - X (X^T X)^{-1} X^T) y = M y$$\n",
    "\n",
    "where $M = I - X (X^T X)^{-1} X^T$ is known as the **residual maker** matrix.\n",
    "\n",
    "Note that $M$ is symmetric, that is, $M^T = M$:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "M^T &= (I - X (X^T X)^{-1} X^T)^T  \\\\\n",
    "&= I^T - (X (X^T X)^{-1} X^T)^T \\\\\n",
    "&= I - (X^T)^T ((X^T X)^{-1})^T X^T \\\\\n",
    "&= I - X ((X^{-1}(X^T)^{-1})^T X^T \\\\\n",
    "&= I -  X (X^T X)^{-1} X^T \\\\\n",
    "&= M\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Note also that $M$ is idempotent, that is, $M^2 = M$:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "M^2 &= (I - X (X^T X)^{-1} X^T) (I - X (X^T X)^{-1} X^T) \\\\\n",
    "&= I - X (X^T X)^{-1} X^T - X (X^T X)^{-1} X^T + X \\left( (X^T X)^{-1} X^T X \\right) (X^T X)^{-1} X^T \\\\\n",
    "&= I - X (X^T X)^{-1} X^T - X (X^T X)^{-1} X^T + X (X^T X)^{-1} X^T \\\\\n",
    "&= I - X (X^T X)^{-1} X^T \\\\\n",
    "&= M\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Now, we have that $MX = 0$, as running least squares on a regression where the covariates match the target variables should yield a model that just copies the covariate over, where all residuals are zero ($\\beta_0 = 0$ and $\\beta_1 = 1$).  So:\n",
    "\n",
    "$$\\hat{\\epsilon} = M y = M( X \\beta + \\epsilon) = M\\epsilon$$\n",
    "\n",
    "where $\\epsilon = Y - X\\beta$ are the population residuals.\n",
    "\n",
    "We can then write an estimator for $\\sigma^2$:\n",
    "\n",
    "$$ \\hat{\\epsilon}^T \\hat{\\epsilon} = \\epsilon^T M^T M \\epsilon = \\epsilon^T M^2 \\epsilon = \\epsilon^T M \\epsilon$$\n",
    "\n",
    "Taking the expectation with respect to the data $X$ on both sides,\n",
    "\n",
    "$$\\mathbb{E}(\\hat{\\epsilon}^T \\hat{\\epsilon} | X) = \\mathbb{E}(\\epsilon^T M \\epsilon | X)$$\n",
    "\n",
    "But $\\epsilon^T M \\epsilon$ is a scalar ($1 \\times 1$ matrix), so it is equal to its trace -- and we can use the cyclic permutation property of the trace:\n",
    "\n",
    "$$ \\mathbb{E}(\\epsilon^T M \\epsilon | X) = \\mathbb{E}(\\text{tr}(\\epsilon^T M \\epsilon) | X) = \\mathbb{E}(\\text{tr}(M \\epsilon \\epsilon^T) | X)$$\n",
    "\n",
    "Since $M$ is a function of $X$, we can take it out of the expectation:\n",
    "\n",
    "$$\\mathbb{E}(\\text{tr}(M \\epsilon \\epsilon^T) | X) = \\text{tr}(\\mathbb{E}(M \\epsilon \\epsilon^T | X))\n",
    "= \\text{tr}(M \\mathbb{E}(\\epsilon \\epsilon^T | X))\n",
    "= \\text{tr}(M \\sigma^2 I_1)\n",
    "= \\sigma^2 \\text{tr}(M)\n",
    "$$\n",
    "\n",
    "Finally, we can compute the trace of $M$:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{tr}(M) &= \\text{tr}(I_n - X(X^T X)^{-1}X^T)\\\\\n",
    "&= \\text{tr}(I_n) - \\text{tr}(X(X^T X)^{-1}X^T) \\\\\n",
    "&= \\text{tr}(I_n) - \\text{tr}((X^T X)^{-1}X^T X) \\\\\n",
    "&= \\text{tr}(I_n) - \\text{tr}(I_k)  \\\\\n",
    "&= n - k\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Therefore, the unbiased estimator is\n",
    "\n",
    "$$\\hat{\\sigma}^2 = \\frac{\\hat{\\epsilon}^T \\hat{\\epsilon}}{n - k}$$\n",
    "\n",
    "or, for our case where $k = 2$,\n",
    "\n",
    "$$\\hat{\\sigma}^2 = \\left( \\frac{1}{n - 2} \\right) \\sum_{i=1}^n \\hat{\\epsilon}_i^2$$\n",
    "\n",
    "Reference:  Greene, William H. Econometric analysis. Pearson Education India, 2003.  Chapter 4, pages 61-62."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.2**.  Prove the formulas for the standard errors in Theorem 14.8.  You should regard the $X_i$'s as fixed constants.\n",
    "\n",
    "$$\n",
    "\\mathbb{E}(\\hat{\\beta} | X^n) = \\begin{pmatrix}\\beta_0 \\\\ \\beta_1 \\end{pmatrix}\n",
    "\\quad \\text{and} \\quad\n",
    "\\mathbb{V}(\\hat{\\beta} | X^n) = \\frac{\\sigma^2}{n s_X^2} \\begin{pmatrix} \n",
    "\\frac{1}{n} \\sum_{i=1}^n X_i^2 & -\\overline{X}_n \\\\\n",
    "-\\overline{X}_n & 1\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "$$s_X^2 = n^{-1} \\sum_{i=1}^n (X_i - \\overline{X}_n)^2$$\n",
    "\n",
    "$$\n",
    "\\hat{\\text{se}}(\\hat{\\beta}_0) = \\frac{\\hat{\\sigma}}{s_X \\sqrt{n}} \\sqrt{\\frac{\\sum_{i=1}^n X_i^2}{n}}\n",
    "\\quad \\text{and} \\quad\n",
    "\\hat{\\text{se}}(\\hat{\\beta}_1) = \\frac{\\hat{\\sigma}}{s_X \\sqrt{n}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.  \n",
    "\n",
    "The formulas follow immediately by performing the suggested replacement on the diagonal elements of $\\mathbb{V}(\\hat{\\beta} | X^n)$ from Theorem 14.8. From the diagonals, replacing $\\hat{\\sigma}$ for $\\sigma$:\n",
    "\n",
    "$$\n",
    "\\hat{\\text{se}}(\\hat{\\beta}_0)^2 = \\frac{\\hat{\\sigma}^2}{n s_X^2} \\frac{\\sum_{i=1}^n X_i^2}{n}\n",
    "\\quad \\text{and} \\quad\n",
    "\\hat{\\text{se}}(\\hat{\\beta}_1)^2 = \\frac{\\hat{\\sigma}^2}{n s_X^2} \\cdot 1$$\n",
    "\n",
    "Results follow by taking the square root.\n",
    "\n",
    "We will also prove the variance matrix result itself, following from the notation and proof used on exercise 14.9.1 (again, adapting from Greene):\n",
    "\n",
    "$$\n",
    "\\hat{\\beta} = (X^T X)^{-1}X^T y = (X^T X)^{-1}X^T (X \\beta + \\epsilon) = \\beta + (X^T X)^{-1}X^T \\epsilon\n",
    "$$\n",
    "\n",
    "Taking the variance conditional on $X$,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{V}(\\hat{\\beta} | X) &= \\mathbb{V}(\\hat{\\beta} - \\beta | X) \\\\\n",
    "&= \\mathbb{E}((\\hat{\\beta} - \\beta)(\\hat{\\beta} - \\beta)^T | X)  \\\\\n",
    "&= \\mathbb{E}((X^T X)^{-1}X^T \\epsilon\\epsilon^T X(X^T X)^{-1} | X) \\\\\n",
    "&= (X^T X)^{-1}X^T \\mathbb{E}(\\epsilon\\epsilon^T | X) X(X^T X)^{-1} \\\\\n",
    "&= (X^T X)^{-1}X^T \\sigma^2 I X(X^T X)^{-1} \\\\\n",
    "&= \\sigma^2 (X^T X)^{-1} \n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "But we have:\n",
    "\n",
    "$$ X^T X = \\begin{pmatrix}\n",
    "1 & 1 & \\cdots & 1 \\\\\n",
    "X_1 & X_2 & \\cdots & X_n\n",
    "\\end{pmatrix} \\begin{pmatrix}\n",
    "1 & X_1 \\\\\n",
    "1 & X_2 \\\\\n",
    "\\vdots & \\vdots \\\\\n",
    "1 & X_n\n",
    "\\end{pmatrix} = \n",
    "n \\begin{pmatrix}\n",
    "1 & \\overline{X}_n \\\\\n",
    "\\overline{X}_n & \\frac{1}{n}\\sum_{i=1}^n X_i^2\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "so we can verify its inverse is\n",
    "\n",
    "$$(X^T X)^{-1} = \\frac{1}{n s_X^2} \\begin{pmatrix}\n",
    "\\frac{1}{n} \\sum_{i=1}^n X_i^2 & - \\overline{X}_n \\\\\n",
    "-\\overline{X}_n & 1\n",
    "\\end{pmatrix}$$\n",
    "\n",
    "and so the result follows.\n",
    "\n",
    "Reference:  Greene, William H. Econometric analysis. Pearson Education India, 2003.  Chapter 4, page 59."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.3**.  Consider the **regression through the origin** model:\n",
    "\n",
    "$$Y_i = \\beta X_i + \\epsilon$$\n",
    "\n",
    "Find the least squares estimate for $\\beta$.  Find the standard error of the estimate.  Find conditions that guarantee that the estimate is consistent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.  Once more adopting notation from the solution of 14.9.1,  let\n",
    "\n",
    "$$ y = \\begin{pmatrix} Y_1 \\\\ \\vdots \\\\ Y_n \\end{pmatrix},\n",
    "\\quad\n",
    "X = \\begin{pmatrix} X_1 \\\\ \\vdots \\\\ X_n \\end{pmatrix}$$\n",
    "\n",
    "and $\\beta$ is a scalar (or a $1 \\times 1$ matrix).\n",
    "\n",
    "The least squares estimate is, again,\n",
    "\n",
    "$$\\hat{\\beta} = (X^T X)^{-1} X^T y$$\n",
    "\n",
    "which simplifies in this one-dimensional case to:\n",
    "\n",
    "$$\\hat{\\beta} = \\frac{\\sum_{i=1}^n X_i Y_i}{\\sum_{i=1}^n X_i^2}$$\n",
    "\n",
    "The unbiased estimator for $\\sigma^2$ is, with $k = 1$,\n",
    "\n",
    "$$\\hat{\\sigma}^2 = \\frac{1}{n - 1} \\sum_{i=1}^n \\hat{\\epsilon}_i^2$$\n",
    "\n",
    "and the variance of $\\hat{\\beta}$ conditional of $X$ is:\n",
    "\n",
    "$$\\mathbb{V}(\\hat{\\beta} | X) = \\sigma^2 (X^T X)^{-1} = \\frac{\\sigma^2}{\\sum_{i=1}^n X_i^2}$$\n",
    "\n",
    "so the standard error of the estimate is\n",
    "\n",
    "$$\\hat{\\text{se}}(\\hat{\\beta}) = \\frac{\\hat{\\sigma}}{\\sqrt{\\sum_{i=1}^n X_i^2}}$$\n",
    "\n",
    "These, of course, make the assumption that $X^T X$ is invertible -- that is, that the sum of squares of the $X_i$ variables is greater than 0.  This is only not the case when all covariates are 0, in which case the value of our estimator $\\hat{\\beta}$ would be irrelevant to determining the prediction outcome -- the system would be undetermined.\n",
    "\n",
    "Finally, note that $\\hat{\\beta}$ is also the MLE in the parameter space for regression through the origin.  As each measurement error $\\epsilon_i = Y_i - \\beta X_i$ is drawn from a normal distribution $N(0, \\sigma^2)$, the log-likelihood for a given parameter $\\beta$ is\n",
    "\n",
    "$$\\ell_n(\\beta) = -\\frac{n}{2} \\log \\sigma^2 - \\frac{1}{2 \\sigma^2} \\sum_{i=1}^n (Y_i - \\beta X_i)^2 + C$$\n",
    "\n",
    "and so maximizing it is equivalent to minimizing $\\sum_{i=1}^n (Y_i - \\beta X_i)^2$, which is what the least squares procedure does.\n",
    "\n",
    "Since the MLE is consistent, the least squares estimate is also consistent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.4**.  Prove equation (14.24).\n",
    "\n",
    "$$ \\text{bias}(\\hat{R}_\\text{tr}(S)) = \\mathbb{E}(\\hat{R}_\\text{tr}(S)) - R(S) = -2 \\sum_i \\text{Cov}(\\hat{Y}_i, Y_i) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "The bias is:\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "\\mathbb{E}(\\hat{R}_\\text{tr}(S)) - R(S) &= \\mathbb{E}\\left[\\sum_i (\\hat{Y}_i- Y_i)^2 \\right] - \\mathbb{E} \\left[ \\sum_i (\\hat{Y}_i - Y_i^*)^2 \\right] \\\\\n",
    "&= \\sum_i \\left( \\mathbb{E} \\left[\\hat{Y}_i^2 \\right] - 2 \\mathbb{E} \\left[\\hat{Y}_i Y_i \\right] + \\mathbb{E}\\left[Y_i^2 \\right] - \\mathbb{E} \\left[\\hat{Y}_i^2 \\right] + 2 \\mathbb{E} \\left[\\hat{Y}_i Y_i^* \\right] - \\mathbb{E}\\left[{Y_i^*}^2 \\right] \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "The random variables $Y_i = \\beta X_i + \\epsilon_\\text{train}$ and $Y_i^* = \\beta X_i + \\epsilon_\\text{pred}$ are independent, as the errors during training and predition are independent, and the covariates $X_i$ and true parameter $\\beta$ are constant.  Since they are independent, $\\mathbb{E}[Y_i Y_i^*] = \\mathbb{E}[Y_i] \\mathbb{E}[Y_i^*]$.\n",
    "\n",
    "We also have $\\mathbb{E} [ {Y_i^*}^2 ] = \\mathbb{V}[Y_i^*] + \\mathbb{E}[Y_i^*]^2 = \\mathbb{V}[\\epsilon_\\text{pred}] + \\mathbb{E}[X \\beta]^2 = \\mathbb{V}[\\epsilon_\\text{train}] + \\mathbb{E}[Y_i]^2 = \\mathbb{E} [Y_i^2]$.\n",
    "\n",
    "Replacing both of those in the expression of bias above, we get the result:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{E}(\\hat{R}_\\text{tr}(S)) - R(S) &= \n",
    "-2 \\sum_i \\left( \\mathbb{E}[Y_i \\hat{Y}_i] - \\mathbb{E}[Y_i] \\mathbb{E}[\\hat{Y}_i] \\right) \\\\\n",
    "&= -2 \\sum_i \\text{Cov}(\\hat{Y}_i, Y_i)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.5**.  In the simple linear regression model, construct a Wald test for $H_0 : \\beta_1 = 17 \\beta_0$ versus $H_1 : \\beta_1 \\neq 17 \\beta_0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.  Let $\\delta = \\beta_1 - 17 \\beta_0$.  The MLE is $\\hat{\\delta} = \\hat{\\beta}_1 - 17 \\hat{\\beta}_0$, with estimated standard error $\\hat{\\text{se}}(\\hat{\\delta})$, where\n",
    "\n",
    "$$\\hat{\\text{se}}(\\hat{\\delta})^2 = \\hat{\\text{se}}(\\hat{\\beta}_1 - 17 \\hat{\\beta}_0)^2 = \\hat{\\text{se}}(\\hat{\\beta}_1)^2 + 17^2 \\hat{\\text{se}}(\\hat{\\beta}_0)^2 $$\n",
    "\n",
    "and the estimates for the parameter standard deviations are\n",
    "\n",
    "\n",
    "$$\n",
    "\\hat{\\text{se}}(\\hat{\\beta}_0) = \\frac{\\hat{\\sigma}}{s_X \\sqrt{n}} \\sqrt{\\frac{\\sum_{i=1}^n X_i^2}{n}}\n",
    "\\quad \\text{and} \\quad\n",
    "\\hat{\\text{se}}(\\hat{\\beta}_1) = \\frac{\\hat{\\sigma}}{s_X \\sqrt{n}}\n",
    "$$\n",
    "\n",
    "The Wald test then checks if $|W| < z_{\\alpha / 2}$, where \n",
    "\n",
    "$$W = \\frac{\\hat{\\delta} - 0}{\\hat{\\text{se}}(\\hat{\\delta})} \n",
    "= \\frac{\\hat{\\beta}_1 - 17 \\hat{\\beta}_0}{\\sqrt{\\hat{\\text{se}}(\\hat{\\beta}_1)^2 + 17^2 \\hat{\\text{se}}(\\hat{\\beta}_0)^2}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.6**.  Get the passenger car mileage data from \n",
    "\n",
    "http://lib.stat.cmu.edu/DASL/Datafiles/carmpgdat.html\n",
    "\n",
    "**(a)** Fit a simple linear regression model to predict MPG (miles per gallon) from HP (horsepower).  Summarize your analysis including a plot of the data with the fitted line.\n",
    "\n",
    "**(b)** Repeat the analysis but use log(MPG) as the response.  Compare the analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "We would ordinarily use a library to do linear regression, but given this chapter is specifically on linear regression formulas, let's do all of the calculations on matrix algebra \"by hand\" instead -- and compare the results with statsmodels OLS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.stats import t\n",
    "import statsmodels.api as sm\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Provided link is dead.  Data was found elsewhere online.\n",
    "data = pd.read_csv('data/carmileage.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_regression(X, Y):\n",
    "    X = X.copy()\n",
    "    \n",
    "    # Create new column with all 1s for intercept at start\n",
    "    X.insert(0, 'const', 1)\n",
    "    \n",
    "    # Least squares solution\n",
    "    beta_hat = (np.linalg.inv(X.T @ X) @ X.T @ Y).to_numpy()\n",
    "\n",
    "    # Predicted solutions\n",
    "    Y_pred = X @ beta_hat\n",
    "\n",
    "    # Prediction errors\n",
    "    epsilon_hat = Y_pred - Y\n",
    "\n",
    "    # Error on training data\n",
    "    training_error = epsilon_hat.T @ epsilon_hat\n",
    "    \n",
    "    # Estimated error variance\n",
    "    sigma2_hat = (training_error / (Y.shape[0] - X.shape[1]))\n",
    "\n",
    "    # Parameter estimated standard errors\n",
    "    se_beta_hat = np.sqrt(sigma2_hat * np.diag(np.linalg.inv(X.T @ X))).T\n",
    "\n",
    "    # t statistic for estimated parameters being non-zero\n",
    "    t_values = beta_hat.reshape(-1) / se_beta_hat\n",
    "\n",
    "    # p-values for estimated parameters being non-zero\n",
    "    p_values = 2 * (1 - t.cdf(np.abs(t_values), X.shape[0] - 1))\n",
    "    \n",
    "    return pd.DataFrame({\n",
    "        'coef': beta_hat.reshape(-1),\n",
    "        'std err': se_beta_hat.reshape(-1),\n",
    "        't': t_values.reshape(-1),\n",
    "        'P > |t|': p_values.reshape(-1) \n",
    "        }, index=X.columns)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(a)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = data['MPG']\n",
    "X = data[['HP']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P &gt; |t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>const</th>\n",
       "      <td>50.066078</td>\n",
       "      <td>1.569487</td>\n",
       "      <td>31.899650</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HP</th>\n",
       "      <td>-0.139023</td>\n",
       "      <td>0.012069</td>\n",
       "      <td>-11.519295</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            coef   std err          t  P > |t|\n",
       "const  50.066078  1.569487  31.899650      0.0\n",
       "HP     -0.139023  0.012069 -11.519295      0.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using manually coded solution\n",
    "get_regression(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            OLS Regression Results                            \n",
      "==============================================================================\n",
      "Dep. Variable:                    MPG   R-squared:                       0.624\n",
      "Model:                            OLS   Adj. R-squared:                  0.619\n",
      "Method:                 Least Squares   F-statistic:                     132.7\n",
      "Date:                Sun, 15 Mar 2020   Prob (F-statistic):           1.15e-18\n",
      "Time:                        17:06:26   Log-Likelihood:                -264.61\n",
      "No. Observations:                  82   AIC:                             533.2\n",
      "Df Residuals:                      80   BIC:                             538.0\n",
      "Df Model:                           1                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "==============================================================================\n",
      "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
      "------------------------------------------------------------------------------\n",
      "const         50.0661      1.569     31.900      0.000      46.943      53.189\n",
      "HP            -0.1390      0.012    -11.519      0.000      -0.163      -0.115\n",
      "==============================================================================\n",
      "Omnibus:                       22.759   Durbin-Watson:                   0.721\n",
      "Prob(Omnibus):                  0.000   Jarque-Bera (JB):               31.329\n",
      "Skew:                           1.246   Prob(JB):                     1.57e-07\n",
      "Kurtosis:                       4.722   Cond. No.                         299.\n",
      "==============================================================================\n",
      "\n",
      "Warnings:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n"
     ]
    }
   ],
   "source": [
    "# Using statsmodels\n",
    "results = sm.OLS(Y, sm.add_constant(X)).fit()\n",
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtYAAAHwCAYAAACCDShwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5yVdZ3//+eLAYQxS1EgEplxS0uRxJatxP1os8ha7Xyh7VOrw2RI62c20L72hTI3vrctP7t8Pv1QytaGbTKJz+5A9XUr+E5li7NjbtCnFooSNL+aDUgSEGapBxOG1/eP6zpw5sw5M+fMXOdc13XO4367ze3M9b7Oj9d4OPicN+/r9TZ3FwAAAICxGRd3AQAAAEAtIFgDAAAAESBYAwAAABEgWAMAAAARIFgDAAAAESBYAwAAABEgWAOoW2Y2y8yeN7OGCr7Gl83sHyrwvA+a2Y1RPy8AYPQI1gBqnpn1m9nRMERnv17l7vvc/WXuPhDeb0hYNTM3s9fEUzlGy8xuMLPvFxjvN7Orc+4zEP55+L2Z7TKz1upXC6BWEKwB1Iv/IwzR2a+n4y6oXlXyXwhG4Qfu/jJJZ0r6kqSvmdmUmGsCkFIEawB1y8yawxnp8Wa2RtJ/kXR3OIN5t5k9FN71p+HYteHjWsPZzWfNbLuZvT7nOS8zsx+b2XNm9lVJk4q89mnh4y/JGZsazqxPM7OzzKzHzA6b2W/D72cWea6Pm9m/FPq5wuNXmNmXzOyAmf3KzP4hG27N7DVm9j0z+52Z/SasudBr3G9mN+eN/dTM3hl+/zoz22pmz5jZY2b2Vzn3+7KZrTOzb5vZC5JazOztZvZI+N/pV2b2ofC+Q2aac//VoNjjxsrdT0i6V9JkSX8UxXMCqD8EawCQ5O6rJf2HpJvDGe2b3f3K8PSl4dhXzewNCgLY30g6W9IXJG0Jg/JESd+U9M+Spkj6fyT91yKv9wdJX5fUljP8V5K+5+6HFPz9vF5Sk6RZko5KunuUP94GScclvUbSZZL+XFJ2ycvfS/o3SWdJminpH4s8x8bcWs3s4rC2b5nZ6ZK2hveZFt6v08xm5zx+iaQ1ks6Q9H0Fs8N/4+5nSLpE0r+X+LOM9nHDCn8JuVHS85Iej+I5AdQfgjWAevHNcIb4WTP75hie579J+oK7/9DdB9x9g6Q/SHpz+DVB0mfd/Zi73yfpP4d5rkFhVUH43ChJ7n7E3f/V3TPu/pyCUHpVucWa2XRJb5P0QXd/IQztn5F0XXiXYwoC8qvc/UV3H7IuOfQNSXPNrCk8bpf09fAXhFZJ/e6+3t2Pu/uPJf2rpHflPH6zu29z9xPu/mL4uheb2cvd/bfhY0pRzuPenPOeP2tmzyr4JWXIfST9WsF78Zfu/rsSawGAQQjWAOrFO9z9zPDrHWN4niZJq/LC2nmSXhV+/crdPef+e4d5rn+XNNnM3hQG1rkKAqzMrNHMvmBme83s95IeknTmKNYnNykI+wdy6v2CgpllSbpVkkn6kZntMbP3FXqSMNx/S6cC+XWSunNe4015/03aJb0y5ymeynvK/yrp7ZL2hktRLi/x5ynncf875z0/093PlLSvyH3Ocfc3u/sDJdYBAEOMj7sAAEgQH/kuekrSGndfk3/CzK6SdK6ZWU64niXpFwVfzP2EmX1NwUzpQUk9YYCVpFWSXivpTe7+azObK+knCkJwvhckNeYc5wfaP0g6x92PF6jh1wpm4WVmfyrpATN7yN2fKPA6myR9LFx7PllSX85rfM/dFxb6ObMvlfe6/ylpsZlNkHSzpK8p+AVl0M9iZq8s8XEAEDtmrAHglIMaeuFa/tgXJb0/nGU2MzvdzP7CzM6Q9AMFa5n/TwsuiHynpDeO8JobJV2rYIZ3Y874GQrWVT9rQZeKjw3zHLskXWlBX+5XSPrb7Al3P6BgDfWdZvZyMxtnZq8OfwmQmb0756LI3yoIwANFXufbCman/7ukr4YX/ElSj6QLzex6M5sQfv2JmV1U6EnMbKKZtZvZK9z9mKTf57zmTyXNNrO5ZjZJ0sdLfBwAxI5gDQCn3CXpXRZ04fhcOPZxSRvCJQ5/5e47FMzw3q0giD4h6QZJcveXJL0zPP6tgsD89eFe0N1/qGCW9lWSvpNz6rMKZoV/I+l/S7p/mOfYKumrkn4maaeCoJvrvZImSnokrOs+STPCc38i6Ydm9rykLZJucfdfFnmd7AWXVyvnl4Bwlv3PFSwPeVrBeuVPSjptmB/9ekn94TKX90t6T/hc/5+C4P6AgosI89d8F3wcACSBDV4KCAAAAGA0mLEGAAAAIkCwBgAAACJAsAYAAAAiQLAGAAAAIkCwBgAAACJQExvEnHPOOd7c3Bx3GQAAAKhxO3fu/I27Ty10riaCdXNzs3bs2BF3GQAAAKhxZra32DmWggAAAAARIFgDAAAAESBYAwAAABGoiTXWAAAAqIxjx45p//79evHFF+MupaomTZqkmTNnasKECSU/hmANAACAovbv368zzjhDzc3NMrO4y6kKd9eRI0e0f/9+nX/++SU/jqUgAAAAKOrFF1/U2WefXTehWpLMTGeffXbZs/QEawAAAAyrnkJ11mh+ZoI1AAAAEs3MdP311588Pn78uKZOnarW1lZJ0pe//GVNnTpVc+fO1cUXX6wvfvGLJ+97//33641vfKNe97rXae7cubr22mu1b9++itRJsAYAAEBkurul5mZp3Ljgtrt77M95+umna/fu3Tp69KgkaevWrTr33HMH3efaa6/Vrl279OCDD+qjH/2oDh48qN27d+sDH/iANmzYoJ///OfatWuX2tvb1d/fP/aiCiBYAwAAIBLd3VJHh7R3r+Qe3HZ0RBOu3/a2t+lb3/qWJGnTpk1qa2sreL9p06bp1a9+tfbu3atPfvKT+uhHP6qLLrro5PlFixbpyiuvHHtBBRCsAQAAEInVq6VMZvBYJhOMj9V1112nr3zlK3rxxRf1s5/9TG9605sK3u/JJ5/Uk08+qde85jXas2eP3vCGN4z9xUtEsAYAAEAkii1djmJJ8+tf/3r19/dr06ZNevvb3z7k/Fe/+lXNnTtXbW1t+sIXvqApU6YMOn/kyBHNnTtXF154oe64446xF1QAfawBAAAQiVmzguUfhcajsGjRIn3oQx/Sgw8+qCNHjgw6d+211+ruu+8eNDZ79mz9+Mc/1qWXXqqzzz5bu3bt0h133KHnn38+moLyMGMNAACASKxZIzU2Dh5rbAzGo/C+971Pf/d3f6c5c+aUdP9bb71Va9as0aOPPnpyLJO/ViVCzFgDAAAgEu3twe3q1cHyj1mzglCdHR+rmTNn6pZbbin5/nPmzNFdd92l9773vXruued09tlna9asWbr99tujKSiPuXtFnria5s2b5zt27Kjqa3Z3V+4PDQAAQFI8+uijg7pq1JNCP7uZ7XT3eYXuz4z1KGRbyWT/JSHbSkYiXAMAANQr1liPQiVbyQAAACCdCNajUMlWMgAAAEgngvUoFGsZE1UrGQAAAKQPwXoUKt1KBgAAAOlDsB6F9napq0tqapLMgtuuLi5cBAAAqGcE61Fqb5f6+6UTJ4JbQjUAAECor09qbg5uI/C5z31OF110kc466yx94hOfkCR985vf1COPPHLyPl/+8pf19NNPl/W8/f39uuSSSyKpUSJYAwAAIEp9fVJra9CPuLU1knDd2dmpb3/72/rtb3+r2267TVI0wTpqBGsAAABEIxuqs32JM5kxh+v3v//9evLJJ7Vo0SJ95jOf0c0336zt27dry5Yt+vCHP6y5c+fqk5/8pHbs2KH29nbNnTtXR48e1c6dO3XVVVfpj//4j3XNNdfowIEDkqSdO3fq0ksv1eWXX67Pf/7zUfzUJxGsAQAAMHb5oTprjOH6n/7pn/SqV71KfX19OuussyRJ8+fP16JFi/TpT39au3bt0kc+8hHNmzdP3d3d2rVrl8aPH68PfOADuu+++7Rz5069733v0+pww5Fly5bpc5/7nH7wgx+M6ccthJ0XAQAAMHbLlg0N1VmZTHC+v78qpTz22GPavXu3Fi5cKEkaGBjQjBkz9Lvf/U7PPvusrrrqKknS9ddfr+985zuRvS7BGgAAAGO3fn3hGWsp6Eu8fn3VSnF3zZ49e8is9LPPPiszq9jrshQEAAAAY9fSIvX0FN7so6cnOB+hM844Q88991zB49e+9rU6fPjwyWB97Ngx7dmzR2eeeaZe8YpX6Pvf/74kqbu7O9KaCNYAAACIRn64rlColqTrrrtOn/70p3XZZZfpF7/4hW644Qa9//3v19y5czUwMKD77rtPH/nIR3TppZdq7ty52r59uyRp/fr1uummm3T55Zdr8uTJkdZk7h7pE8Zh3rx5vmPHjrjLAAAAqDmPPvqoLrroovIe1NcXrKlev74iobpaCv3sZrbT3ecVuj9rrAEAABCtlpaqXaiYJCwFAQAAACJAsAYAAAAiQLAGAADAsGrhmrxyjeZnJlgDAACgqEmTJunIkSN1Fa7dXUeOHNGkSZPKehwXLwIAAKComTNnav/+/Tp8+HDcpVTVpEmTNHPmzLIeQ7AGAABAURMmTND5558fdxmpwFIQAAAAIAIEawAAACACBGsAAAAgAgRrAAAAIAIEawAAACACsQZrMzvTzO4zs5+b2aNmdrmZTTGzrWb2eHh7Vpw1AgAAAKWIe8b6Lkn3u/vrJF0q6VFJt0nqdfcLJPWGxwAAAECixRaszezlkq6U9CVJcveX3P1ZSYslbQjvtkHSO+KpEAAAAChdnDPWfyTpsKT1ZvYTM7vHzE6XNN3dD0hSeDstxhoBAACAksQZrMdLeoOkde5+maQXVMayDzPrMLMdZraj3rbYBAAAQPLEGaz3S9rv7j8Mj+9TELQPmtkMSQpvDxV6sLt3ufs8d583derUqhQct+5uqblZGjcuuO3ujrsiAAAAZMUWrN3915KeMrPXhkMLJD0iaYukpeHYUkmbYygvcbq7pY4Oae9eyT247eggXAMAACSFuXt8L242V9I9kiZKelLSMgVh/2uSZknaJ+nd7v7McM8zb94837FjR4WrjVdzcxCm8zU1Sf391a4GAACgPpnZTnefV+jc+GoXk8vdd0kqVNiCateSdPv2lTcOAACA6oq7jzVKNGtWeeMAAACoLoJ1SqxZIzU2Dh5rbAzGAQAAED+CdUq0t0tdXcGaarPgtqsrGAcAAED8Yl1jjfK0txOkAQAAkooZawAAACACBGsAAAAgAgRrAAAAIAIEawAAACACBGsAAAAgAgRrAAAAIAIEawAAACACBGsAAAAgAgRrAAAAIAIE6yrq7paam6Vx44Lb7u64KwIAAEBU2NK8Srq7pWXLpGPHguO9e4NjiW3KAQAAagEz1lVyyy2nQnXWsWPBOAAAANKPYF0lR46UNw4AAIB0IVgDAAAAESBYj9HGjRtLut/ZZ5c3DgAAgHQhWI/Rpk2bSrrfXXdJEycOHps4MRgHAABA+hGsKyR/Jru9Xbr3XqmpSTILbu+9l44gAAAAtYJgXSG5M9nZkN3eLvX3SydOBLeEagAAgNpBsK6C/OUiCxcujKkSAAAAVArBOgYPPfRQ3CUAAAAgYgRrAAAAIAIEawAAACACBGsAAAAgAgRrAAAAIAIE6xTp7paamyUzV3NzcAwAAIBkGB93AShNd7fU0SFlMpJk2rs3OJbohw0AAJAEzFinxOrV2VB9SiYTjAMAACB+BOuU2LevvPH8LdUBAABQWQTrlJg1q7zx/N0eAQAAUFkE65RYs0ZqbBw81tgYjAMAACB+BOuUaG+XurqkpiZJOqGmpuB4LBcu3nTTTVGVBwAAUPcI1inS3i7190vTps1Qf//Yu4Hcd999UZQFAAAAEaxRwMKFC+MuAQAAIHUI1hjioYceOvk93UUAAABKQ7DGsOguAgAAUBqCNQAAABABgjXGhM4iAAAAAYI1xqRQZxHWZQMAgHpEsEbksuuymc0GAAD1hGCNiqFPNgAAqCcEa8SC2WwAAFBrYg3WZtZvZg+b2S4z2xGOTTGzrWb2eHh7Vpw1ojKYzQYAALUmCTPWLe4+193nhce3Sep19wsk9YbHAAAAQKIlIVjnWyxpQ/j9BknviLEWAAAAoCRxB2uX9G9mttPMOsKx6e5+QJLC22mxVQcAAACUKO5gfYW7v0HS2yTdZGZXlvpAM+swsx1mtuPw4cOVqxCJMWfOnLhLAAAAKCrWYO3uT4e3hyR9Q9IbJR00sxmSFN4eKvLYLnef5+7zpk6dWq2SEaM9e/bEXQIAAEBRsQVrMzvdzM7Ifi/pzyXtlrRF0tLwbkslbY6nwtrV3S01N0uHDh1Qc3NwnFYve9nLJLHbIwAAiF+cM9bTJX3fzH4q6UeSvuXu90v6hKSFZva4pIXhMSLS3S11dEh790rSOO3dGxynNVy/8MILkk7t9ggAABCX8XG9sLs/KenSAuNHJC2ofkX1YfVqKZMZPJbJBOPt7fHUVAnnnXeennrqqbjLAAAAdSTuixdRZfv2lTeeVvv37z/5PctEAABANRCs68ysWeWN14L8ZSIEbQAAUAkE6zqzZo3U2Dh4rLExGK8XrMcGAACVQLCuM+3tUleX1NQkSSfU1BQc19L66nLRHxsAAESBYF2H2tul/n5p2rQZ6u+v71AtFe6Pfd5558VQCQAASDOCNVBA9uJH1mMDAIBSEayBYbAeGwAAlIpgDQAAAESAYB2x7HbhPT2bU79dOAAAAEoX286LtSi7XXiws+Gp7cIvvPCquEsDAABAhTFjHaFi24Xv2rWK2WsAAIAax4x1hIpvC24nZ68BAABQm5ixjtBI24JnMsGsdtyy68APHTrATDoAAEBECNYRKrRdeL7is9rVkV0HvnevlLsOnHANAAAwNgTrCA3eLtwL3mekWe1KK7YOPAkz6fWO3R4BAEg3gnXEstuFz51755DZ68bGYFY7TsVmzOOeScep3R4BAEA6EawrZObM7+XMXp9QU5O0dGkwM/zSS0djW9tcbMY87pl0AACAtCNYV1B29rq1dbHWrJE2bIh/bXOhdeBJmEkHAABIO4J1lSRlbfPgdeDBTHpXVzCO9Ni4cWPcJQAAgDwE6ypJ0trm7Ez6tGkz1N9PqE6jTZs2xV0CAADIQ7CuEtY2AwAA1DaCdZWwthkAAKC2EayrhLXNAAAAtW183AXUk/b24Ou00yarv/8PcZcDAACACDFjDQAAAESAYA3gpIULF8ZdAgAAqUWwBnDSQw89FHcJAACkFsF6LPr6dE9vr9TXF3clidDdLTU3Sz09m9XcLB09+pdxlwQAAFA1BOvR6uuTWls1/ehRqbW17sN1d3ewRXvulu3PPbe26lu2AwAAxIVgPRphqD65R3kmMzhc1+FMdqEt26XGqm/ZDgAAEBeCdbnyQ3VWNlyvXTt4Jnvt2roI2Unash0AACAOBOtyLVtWaGo2kMlIH/rQ4JnsVasGLxdZsUIvvvSStGJF9WquArZsBwAA9Y5gXa7164fuTZ7LvfB4JiMtXCitWyeTpHXraipcF9qyXcqwZTtOmjNnTtwlAABQUQTrcrW0SD09Q1Ok2ciPHRgYfFxD4brQlu1nnLGSLdtx0p49e+IuAQCAiiJYj0Z+uG5slO64Y/iZ7GJqLFz390utrYvV3y9NnvyNSJ4328bP/biam0WnEQAAkEgE69EKw/XByZODkL1yZeGZ7FKsWxd9fTWiUBu/jg7CNQAASB6C9Vi0tOjGBQuCkB0ejypcL18efW01olAbv0xGtPGrcRs3boy7BAAAykawjlruTPadd54K2Q0Nhe+/fLnU2Vn6869dq18dOhS09asDtPGrT5s2bYq7BAAAykawroTsTHa4POTg5MnS1q1DZ6ZHEaq1apXGS9KqVXURrmnjBwAA0oJgXWm5y0U6O6Xly+XSqEP1IHUQrgu18WtsFG38AABA4hCsq62zU5MmThx7qM4qFK5HuaV6tvvGSy8dTUz3jUJt/Lq6RBs/nMR6bABAUhCs0+DWW0s/H265Pmi3xxIU676xf/9Voy47Ktk2fmbj1d8fHF99dbZ1+AmZSdu33x5vkYgN67EBAElBsE6DT32qtPNhqB60pXqJ4bpY941du1YlZvY66+qrpd7e7FGwMc8zz1ymq6+OrSQAAACCdSqsXBl0GCnkzjuD8/mhOqtYuM7rLlK8y4Ylrnf0qVCdy4qMAwAAVAfBOi0KhetsqJakZcuGhuqsTCY4n1Wgu8iUKcO/PL2jAQAAhhd7sDazBjP7iZn1hMdTzGyrmT0e3p4Vd42JEYbr49LgUC1J69cX35imsTE4LxXtLvL+o8HM9d1aoRMy3a2h26ynoXd09gJMaSBxS1gAAEBtiz1YS7pF0qM5x7dJ6nX3CyT1hsfIWrlS506bNjhUS8V3fWxsDMZbWobtLvL3mVX6D12hFVonk7RC6/QNLdYv1ay3KFhGkpTe0QsWFBp1XXxxci/ABAAAtS/WYG1mMyX9haR7coYXS9oQfr9B0juqXVdq5Yfr3FAtDdtdxCRdoe3hpYDB8WJtUbP2qketeutpfYnpHf3AA7nh2iVJU6b8RC+8UHiJ+WOPvbeq9aE20MYPAFCuuGesPyvpVkkncsamu/sBSQpvpxV6oJl1mNkOM9tx+PDhyleaEt1Pt6jtjB71a5bazuhR99Mtp04O013EpZOhOit7fLoy+n+9Ve2vKq8vdiU98IDkLknj5C7Nn/+xoktVjh49p5qloUbQxg8AUK7YgrWZtUo65O47R/N4d+9y93nuPm/q1KkRV5dO2V7UXznYovO1V1852DK4m8cw3UXyQ3W+8S8V6C6S11kka//+q9TcLB06dKCq65yLLVWZPPk31SkANe+mm26KuwQAQILFOWN9haRFZtYv6SuS/szM/kXSQTObIUnh7aH4SkyXYr2oB3XzKBSu588PF1SMILe7SIHOIpL0wOo+fXPXP+r8vX3KXedcjXBdbPvz1772f1X+xVEX7rvvvrhLAAAkWGzB2t3/1t1nunuzpOsk/bu7v0fSFklLw7stlbQ5phJTp9hSiCHj+d1Ftm3TP79s+cjhOttdZO1aed5FkL5qlbRiheb/z1Y1aZ/u1zU6rgZ9UGur1qqv2PbnM2d+r/Ivjro2Z86cuEsAACRA3GusC/mEpIVm9rikheExSlBsKUTB8bzuIje80KlOnQrXQ0J29kLIn/xEvmpVwfXYvm6dGj2YMj9Nx9SgE1qrVfqg1kbeqi+/rR6dPxCnPXv2xF0CACABEhGs3f1Bd28Nvz/i7gvc/YLw9pm460uLYkshSunmMWuWdLNOhevNWqQXVKC7yK23Fl2PXWjcJK3VKt1+Zs467CJrs0uVXUue21bv4Ydv1ooVtNsDAADxSUSwRjSKLYVobx/5sdlQfrM6NU6uv9Rmveu0Hj094azBLfs+9anS1mPnMEn/92/DddhF1maXo9Ba8oGBSerqot0eAACID8G6xrS3S/39UmvrYvX3lxaqs4/LD+Xv+VKL/uaaK06FaklauVL/cNadQ8L1SGHbJOlDHyq462O54TqYkR5qYKDwOO32AABANRCscVI2lE+cOHnYUH7gupVaqTsHrcfu1HIdswnDv4AXid+jnLkuFe32AABANRCsUbZvf1v6rIJwfVwNWqk7dbM69d5p35UmFAnXNkKn7GF2hSwH7fYAAEBcCNYoW7bDx2e1UhN0XJ9V0Fnkq4dapO9+V78fP37wA+68U7rjjuGfdJhdISVJixdrwF1avHjYu9FuD7WGrdUBID0I1ijbsG39WlrU/ta3Du6TvXLlsLs+nrxPMYsXS1u2BOu0t2zRf5wdhOtf6HydkOkXOl+Sa8GCU8tZpIay1pgDScXW6gCQHgRrlK2ktn55fbKzY0PCdYmhOtefHtmijE3W+eqXSTpf/Tqm8XpgdV/h5wAAAKgCgjXKNpa2fkN2fSwzVGdN9hdP9s02SeN1QlqwQOrrk664Qick6YorSv+hAAAAxohgjVHJLrmYNm1G+UsuCs1mF1IkVBflLv3Zn0nbtwehe/t26ZJLdE9vbxC4gRrHemwAiBfBGsm1aNHYn2PPHk0/elRqbSVco+axHhsA4kWwRnJt3hxNuJaCLRhbW6W1a5nBBgAAFUGwRrIVCteTJo3uuTIZadWqwTPYRdv4tam5Werp2azmZqm7e3QvCQAA6gfBGlW1YoU0frx06NCvNX58cDyiMFy7FITso0el5uaxFZLJSFdfPaiNXzZcByH6i+HW6eO0d6/U0UG4BgAAwyNYo2pWrJDWrZMGBiTJNDAQHJcarhvMgpAtSb/8pdTcHITt7K6OI+3umO/EicHHYbi+cNkVOqGX6We6RL9Us96iPmUy0urV5T09AACoLwRrVE1XV3njI/rlL4M/wL29Ojh5stTbK82fP8onC23ZonnHgq4il2iPmrVXPWrVW9R3csdJAACAQgjWqJpgprr08ZK1tOjGBQuklhZp2zZp/vxgJnv27KE72ZTA8m5PV0Y9atW107jgEQAAFEewRtU0NJQ3PmrbtgV/sHfvlnp6gtnsO+8cVcjOOl0Z/fNvadkHAACKI1ijajo6yhuPRHY2e+VKqafnVLgeV/4f/fEvZaRlyyIuEAAA1AqCNSK3f/9Vam6WDh06MKhVXWentHx5doba1dAQHHd2Dv982U4i7gOldxIppKXl1Az2Aw8MaePnIz2+sVFav36ULw4AAGodwRqR6u6WHn745qKt6jo7pePHpWnTXqnjx0sL1aPuJFJI7nrsvDZ+2zS/eLhubAxmvFtaRvnCAACg1hGsEanVq6WBgcEbuIylVV3knUTy5bTx+y/aNihcZ29fEKEaAACMjGCNSBVrSTfaVnUV6yRSwNlna1C43q3Z6leT2l9OqAYAACMjWCNSs2aVNz6SqnUSkXTXXdLEiUG4HifX67Vbf2SP692dVQjVV1yhE+EtAABIJ4I1IrVmjdTQ8OKgscbGYHw0qtlJpL1duvdeqalJkk6oqUm69NK71N4e/WsNcsUV0vZgUxpt3064BgAgpQjWiFR7uzRnzt2DwmlXl0YdTkfbSWS02tul/n5JalB/vzRz5vcq80JZYagehOM6/pcAAB8SSURBVHANAEAqEawRuZkzv6f+fmnatBnq7x99qM7KdhIxayipk0hqFArVWYRrAABSh2ANxKVYqB7pfF+f7untZRdIAAAShmANxGV+8b7ZHp4foq9Pam3V9KNHpVa2WAcAIEkI1kCO7m6puVmSBtTcHOwiWTHbthXclMYlbdN8adu2wSfCUK1MJjjOZAjXAAAkCMEaCHV3B91GcneNfPjhm0/uGlkJ72kauinNNs3Xe5pGCNVZhGsAABKDYA2EVq8emlsHBiaNetfIUqxZI13TeCpcb9N8XdO4bWh7wmXLhhaXlckE5wEAQKwI1kAomKkufTwK7e3S0qXSWxq2aZxO6C0N27R0aYFOKuvXBw3BC2lsDM4DAIBYEayBUDV3eczq7pY2bMhu0W4aGAiOhyw/aWmRenqGhuvGxmCcLdcBAIgdwRo15eqrJTNJOiGz4LhUQbgtfTwKhZafZDIqvPwkP1xHGKqzF22+9NJRNTcXCPYAAGBEBGvUjKuvlnp7s0cmKTjevv32kh5/9tnljUdh377yxrPh+uDkyZGG6o4O6cN7V+iEGvThvSvU0UG4BgCgXARr1IxToXqwZ565rLqFlGHWrPLGJUktLbpxwYLIln+sXi19KrNCK7ROJmmF1ulTmRUVvWgTAIBaRLAGQs88U954FNasKbxsekhXkAq6de+pUC3pZLi+de+K6hUBAEANIFgj5drU3Cz19Gwu+5HZdcXux9XcLE2ZUvh+w84ej1F7u9TVJTU1SdIJNTUFx0O6glTKihVanhOqs0zScq2TVhCuAQAoFcEaqRWsAf7iyQ1dipky5Scn758N4eecI91ww+DNYJ59Vpo4cfBjGxperPjs8bZt0v79kmTav3/ohosVtW5oqM6y8DwAACgNwRqpFawBPr3I2WAvwwULpPnzPzZkV8UjR6Tjxwc/YmBAmjBh8OzxnDl3V3T2eMWKILvmtttbV82J4uXLx3YeAACcRLBGahXrnBG02xsnd+mBB4KxQm3tCnnhBam/X5Ia1N8vzZz5vShKLaqrq7zxyHV26rEFy09uqZ7lkh5bsFzq7KxSIQAApB/BGqlVTkeNou3rYhZH7+x81zzRqU6dCtcuqVPLdc0TEYbqxYs14C4tXhzdcwIAkDAEa6RWsPb5hUFjxTpqlHoBYiV7VhcSx26P+fbtk27WqXDdqeW6WZ3R/TKyeLG0ZUuwZnvLFsI1AKBmEayRWsHa5/9WUkeNQm3t8k2YIN11V/R1Dqejo7zxSsj+0nGzOjVOrpvVOWh8TMJQPQjhOnK5F+aycyYAxIdgPUZtbW1xl1DnNqm/X2ptXaz+/uJt6gq1tVu+fPDx+vVVbHMX6uwM6ghmqF0NDcFxNZc2V6yXdqFQnUW4jkz+hbl794qdMwEgJrEFazObZGY/MrOfmtkeM7s9HJ9iZlvN7PHw9qy4aizFkiVL4i4BJWpv16AQ3tkZHJuNHzaUV1pnZ7ZDyTgdP1796wUr1ku7WKge6Xxfn+7p7ZX6+sZYQH0odGFuJiN2zgSAGMQ5Y/0HSX/m7pdKmivprWb2Zkm3Sep19wsk9YbHqcNMNtIk+0vHxImTo/slY9Gi8s/39UmtrZp+9KjU2kq4LkGxtfBJvWAXAGrZsMHazGaa2Z/mHK80s78Lv14zlhf2wPPh4YTwyyUtlrQhHN8g6R1jeZ245M5kE7JRlzZvLh6uFy0KzucKQ/XJ6ddMhnBdgnK64wAAKmukGetPSzoz5/hvFLRhcEm3j/XFzazBzHZJOiRpq7v/UNJ0dz8gSeHttCKP7TCzHWa24/Dhw2MtpaJYLjI2+VuPs3Y0RQqF61JCdRbhekQVWyMPACjbSMH6te7ek3Occfc73f3vJY15PsTdB9x9rqSZkt5oZpeU8dgud5/n7vOmTp061lKq6sorr4y7hNTgwqwaEIZrlwqHaklatqz4Dj6ZTHC+ENZjV26NPFJh48aNcZcAIMdIwXpS3vGCnO8j6/jr7s9KelDSWyUdNLMZkhTeHorqdZJi69atcZeQGlyYVSM2b1aDWeFQLQUtWYr1Q2xsDM7nYz32SfkX5hKq68emTZsGHRO0gXiNFKyfM7MLswfu/owkmdnrJD1f9FElMLOpZnZm+P1kSVdL+rmkLZKWhndbKqnI/4lRD7gwq060tEg9PYXXNPT0BOdzVXA9Nj2hkWb5QRtAdY0UrD8mqcfMlprZnPDrBgXh92NjfO0ZkvrM7GeS/lPBGuseSZ+QtNDMHpe0MDxGneLCrDqSH65LDdVZEYRrlh4BAMZi2GDt7vdLeqeCJSBfDr9aJL3T3b8zlhd295+5+2Xu/np3v8Td/3s4fsTdF7j7BeHtM2N5HaQbF2bVmTBcH5w8uXColka/HrsELD0CAIxFKX2sD0r6nKQF7v7H7r7U3XdXuC4M413velfcJVQNF2bVoZYW3bhgQeFQLY1uPXaJWHoEABiLkfpY3yhpj6R/lPRzMxthxwdUw+c///m4S6iq7IVZce+QiIQodz12GVh6hHrHxY/A2Iw0Y/1BSbPd/XJJ8yX9beVLAqIx0kVoK1ZI48dL7gMaPz44jkO2TmkgtovlsjW89NLRdFywV+p67DJllx69RX36pZr1FvWx9Ah1hYsfgbEZP8L5l9z9sCS5+5NmdloVakIE2O2xTR0d2fWypy5Cy1qxQlq3LntkGhg4ddzZWb0qsxfLFaqzWjPzSahhVLLrsf/iLzQ9glAtBT/v9Ef6NP9/tqrRM/qWtWr7B3t0dfvYnxsAUPtGmrGeaWafy34VOEZCsdvj/xj2IrSursKPKjZeKUm4WC4JNWSV3epupPXY5err09WfDUK1JDV6Rld/Nuw0snatfnXokLR2bTSvBQCoOSPNWH8473hnpQoBolV4Uey+fdKcOdLAQOFHFRuvlCRcLJeEGqQEzJwP18bvmmukY8eCvzBXrQrGV66sQlEAgDQZNli7+4ZqFYLqq+3uIvskNQ8ZzV6E1tBQOEQ3NFS0qIL1BD2Th47XUw3S8DPnVQnWw7XxO3Zs8HFuuF68WAPu0uLFxXeWBADUhZG6gmwZ7qtaRaIyaru7yEeH7X+du946V7HxSklCn+4k1CAlYOZ8uDZ+haxaJV1yibRli0yStmwJwjUAoG6NtMb6ckkzJf2HpDsk3Zn3hRp05ZVXnvw+vRdBbhq2/3Vnp7R8eXaG2tXQEBxX88JFKRl9upNQg5SAVnfF2vgNZ8+ewceEawCoayMF61dK+qikSyTdpWCL8d+4+/fc/XuVLg7x2Lp168nv03wRZLb/dWvr4oL9rzs7pePHJbMGHT9e/VCdla1TaoitT3e2hokTJ8dWQyJmzvPD9YQJ5T8H4RoA6tZIW5oPuPv97r5U0pslPSHpQTP7QFWqA1A3kjJzPmhb9e9+V7pzFP84t4WVcgBQj0bc0tzMTjOzd0r6F0k3Kdje/OuVLgzpUNsXQKLaRvpXhqrJbeO3cuXQcD179vCPX8QmtQBQj0a6eHGDpO2S3iDpdnf/E3f/e3f/VVWqQ+LV9gWQQCgM18elIGTv3l08PC9aRHcQAKhTI81YXy/pQkm3SNpuZr8Pv54zs99XvjykUXoveASGsXKlzp027VT/6s2bh4brqEN1X5/u6e0NemwDABJvpDXW49z9jPDr5TlfZ7j7y6tVJNIle8Ejy0RQ88Jw7VJFQrVaWzX96NFg4xrCNaCNGzfGXQIwrBHXWAOjxTKR6JW95Tcqb/NmNZhVJFSf3LAmkyFcA5I2bdoUdwnAsAjWiAWz2eXLbvkd7JJ4astvwnWNGW5rdcI1ACQawRqxYDa7fMNt+Z1k2Vl29+PMspdiuK3VM5ngfCGsxwaA2BGsgZSIfcvvUWCWfRSG21q9sTE4n4/12ACQCARrICVi3/J7FNI6yx6rYlurNzYG4y0tg8dZjw0AiUGwRmrMHmlTjhqXiC2/y5TGWfZEyA/XpYbqLMI1AMSCYI3UePjhh+MuIVaJ2fK7DGmcZU+M3K3VC4VqafTrsQEAFUGwRqqdfvrpcZdQVYnZ8rtEaZxlT5TcrdULGc16bABAxRCskWrPP/+8JHZ7TKo0zrKnSrnrsQEAFUWwRk3I7vaYNXPmzJgqAaqs1PXYAICKI1ijJj311FMnv2c2Oz6026uSUtZjAwAqjmCNmpc/m03Qrp5KtttLwvbus2dLZpL7gMyC49iMtB4bGEYSPk9ptnHjxrhLQEIQrFF38oM2KqdS7faqPROeDR2HDh04GTpmz5YeeSR7D5MUHNd5V0ikEP+yNHrZvxva26/jFxJIIlgDdd8fu5Iq1W6vmhvPFAsdp0L1YMXGgaRiI6fR4RcSFEKwRt0r1B+bix+jUal2e9XceKZY6ABqBRs5jQ6/kKAQgjVQQPbiR9Zjj017u3T55dkjlxQcj7XdXjU3nglmo4DaxUZOo8MvJCiEYA0Mg/XYY7NihdTbmz0K1iH39gbjY1HNjWcaGsq7/8UXR18DUEls5DQ6/EKCQgjWACqmq6u88VJVc+OZgYHi506FaD95vGdP9DUAlcRGTqPDLyQohGANoGKKhdLhwmqpqrW9exA2Co/v2SO5S2YNcidUI72q9XmqJfxCgkII1gAqptgyinKXV8SJWSkAxfALCfIRrAFUTEdHeeNJxKwUAKBU4+MuAEDt6uwMbru6pIEBV0ODqaPj1HhatLcHX9Onz1B//8G4ywEAJBQz1gAqqrNTOn48WId8/Hj6QjUApAHbqicDwRoAACDlNm3aFHcJEMEaSAx2ewQAIN0I1kBCZHd7BAAA6USwBgAAACJAsAYAAAAiQLAGUqitrS3uEgAAQJ7YgrWZnWdmfWb2qJntMbNbwvEpZrbVzB4Pb8+Kq0YgqZYsWRJ3CQAAIE+cM9bHJa1y94skvVnSTWZ2saTbJPW6+wWSesNjAAAAINFiC9bufsDdfxx+/5ykRyWdK2mxpA3h3TZIekc8FQIAAAClS8QaazNrlnSZpB9Kmu7uB6QgfEuaFl9lAAAAQGliD9Zm9jJJ/yrpg+7++zIe12FmO8xsx+HDhytXIADEoLtbam6Weno2q7k5OAYAJFuswdrMJigI1d3u/vVw+KCZzQjPz5B0qNBj3b3L3ee5+7ypU6dWp2AAqILubqmjQ9q7V5LGae/e4JhwDQDJFmdXEJP0JUmPuvvanFNbJC0Nv18qaXO1awPq1ZVXXhl3CZC0erWUyQwey2SCcQBAco2P8bWvkHS9pIfNbFc49lFJn5D0NTP7a0n7JL07pvqAurN169a4S4CkffvKGwcAJENswdrdvy/JipxeUM1aACBJZs3KLgMZOg4ASK7YL14EAAy2Zo3U2Dh4rLExGAcAJBfBGgASpr1d6uqSmpok6YSamoLj9va4KwMADIdgDQAJ1N4u9fdLra2L1d9PqAYA6VQr0nHjlMhWpHFevAgAAACUJNuKNNs1KduKVErO5AMz1gAAAEi8NLQiJVgDqIrZs2fHXQIAIMXS0IqUYA2gKh5++OG4SwBqS1+f7untlfr64q4EqIpiLUeT1IqUYA0AQNr09UmtrZp+9KjU2qoHVvep7ZV9+seen6ntlX2Ju6ALiEIaWpESrAEASJMwVJ9cbJrJ6Ir/8TatP/g2NWuf7jnYqn/569oK19lOED09mxPZCQLVkYZWpARrAInT1tYWdwlAMuWH6tBk/UGT9AdJ0unK6L4/tKpnVW0sEcl2ggh2Ix13shME4bo+Jb0VKcEaQOIsWbIk7hKAZFq2bGhbhAJOV0b3HGytifXXaegEESdm85OFYA0AQFqsXz90kWkRpysTBPGUS0MniLgwm588BGsAqcayEdSVlhapp6ekcH18YmMQxFMuDZ0g4sJsfvIQrAGkGstGUHfyw/Vpp2lg/GmD7nJ8YqPG398T3LdUCW3fl4ZOEHFhNj95CNYAAKRNGK4PTp4sfec7avi375xKn42jC9W57fuSFK7T0AkiLszmJw/BGgCANGpp0Y0LFgQBOjdo94wuVOe270tiuE5yJ4i4MJufPARrAABqQW7QLlWR9n1JDNcYitn85CFYAwBQr4Zr35epja4itY7Z/GQhWAMAUK+Ga9/XWBtdRYBqIlgDAFCvirXva2wsf602AII1ABRCf2zUjfxwTagGRo1gDQAFFOqP/a53vSuGSoAqGEtXEQAnEawBoESf//zn4y4BqJzRdBUBMAjBGgAAAIgAwRoAxmj27NlxlwAASACCNQCM0cMPPxx3CQCABCBYAwCA5Ovr0z29vewGiUQjWAMAgGQLt16ffvQoW60j0QjWAAAgucJQfXLr9UyGcI3EIlgDAJAy3d1Sc7PU07NZzc3BcU3KD9VZhGsk1Pi4CwAAAKXr7pY6OrJZc5z27g2Oa9KyZUNDdVYmE5zv769qScBwmLEGgARja3XkW7268ATu6tXx1FNR69ef2mo9X2NjcB5IEII1ACRYoa3VUd/27StvPNXCrdaHhOvGRrZeRyIRrAEASJFZs8obT738cE2oRoIRrAEASJE1awpP4K5ZE089VRGG64OTJxOqkWgEawCoEazHrg/t7VJXl9TUJEkn1NQUHLe3x11ZhbW06MYFCwjVSDSCNQDUCNZj14/29qAZRmvrYvX310GoBlKCYA0AAABEgGANAAAARIBgDQAAMFp9fbqnt5ddICGJYA0AADA64Zbr048eZYt1SCJYAwAAlC8M1Se3wcxkCNcgWAMAAJQlP1RnEa7rHsEaAACgHMuWDQ3VWZlMcB51iWANAABQjvXrh25/mdXYGJxHXYo1WJvZvWZ2yMx254xNMbOtZvZ4eHtWnDUCAAAMEm6xXnBvebZcr2txz1h/WdJb88Zuk9Tr7hdI6g2PAQAAkiM/XBOqoZiDtbs/JOmZvOHFkjaE32+Q9I6qFgUAAFCKMFwfnDyZUA1J8c9YFzLd3Q9IUng7LeZ6AAAACmtp0Y0LFhCqISmZwbokZtZhZjvMbMfhw4fjLgcAAAB1LonB+qCZzZCk8PZQoTu5e5e7z3P3eVOnTq1qgQAAAInBtuqJkcRgvUXS0vD7pZI2x1gLAABAcrGteqLE3W5vk6QfSHqtme03s7+W9AlJC83scUkLw2MAAADkYlv1xBkf54u7e1uRUwuqWggAAECajLStOl1KYpHEpSAAAAAYDtuqJxLBGgAAIG3YVj2RCNYAAABpw7bqiUSwBgAASCO2VU8cgjUAAEBasa16ohCsAQAA0oxt1RODYA0AAABEgGANAAAARIBgDQAAAESAYA0AAABEgGANAAAARIBgDQAAAESAYA0AAABEgGANAAAARIBgDQAAAESAYA0AAABEgGANAACA9Ojr0z29vVJfX9yVDEGwBgAAQDr09UmtrZp+9KjU2pq4cE2wBgAAQPKFoVqZTHCcySQuXBOsAQAAkGz5oTorYeGaYA0AAIBkW7ZsaKjOymSC8wlAsAYAAECyrV8vNTYWPtfYGJxPAII1AAAAkq2lRerpGRquGxuD8ZaWeOrKQ7AGAABA8uWH64SFaolgDQBAarW1tcVdAlBdYbg+OHly4kK1RLAGACC1lixZEncJQPW1tOjGBQsSF6olgjUAADWDGWwgXgRrAABqBDPYQLwI1gAAAEAECNYAAABABAjWAAAAQAQI1gAAQBIXPwJjRbAGAACSuPgRGCuCNQAAABABgjUAAAAQAYI1AAAAEAGCNQAAABABgjUAAAAQAYI1AAAAEAGCNQAAABABgjUAAAAQAYI1AAAAEAGCNQAAABABgjUAAAAQAYI1AAAAEIHEBmsze6uZPWZmT5jZbXHXAwAA4tXW1hZ3CcCwEhmszaxB0uclvU3SxZLazOzieKsCAABxWrJkSdwlAMNKZLCW9EZJT7j7k+7+kqSvSFocc00AAABAUUkN1udKeirneH84BgAAACRSUoO1FRjzQXcw6zCzHWa24/Dhw1UqCwAAACgsqcF6v6Tzco5nSno69w7u3uXu89x93tSpU6taHAAAAJAvqcH6PyVdYGbnm9lESddJ2hJzTQAAAEBR4+MuoBB3P25mN0v6rqQGSfe6+56YywIAAACKSmSwliR3/7akb8ddBwAAAFCKpC4FAQAAAFKFYA0AAABEgGANAAAARIBgDQAAAESAYA0AAABEgGANAAAwBm1tbXGXgIQgWAMAAIzBkiVL4i4BCUGwBgAAACJAsAYAAAAiQLAGAAAAIkCwBgAAACJAsAYAAAAiQLAGAAAAIkCwBgAAACJAsAYAAAAiQLAGAAAAIkCwBgAAACJAsAYAAEi5tra2uEuACNYAAACpt2TJkrhLgAjWAAAAQCQI1gAAAEAECNYAAABABAjWAAAAQAQI1gAAAEAECNYAAABABAjWAAAAQAQI1gAAAEAECNYAAABABAjWAAAAQAQI1gAAAEAECNYAAABABAjWAAAAQAQI1gAAAEAECNYAAABABAjWAAAAQAQI1gAAAEAECNYAAABABAjWAAAAQAQI1gAAAEiVtra2uEsoiGANAACAVFmyZEncJRREsAYAAAAiQLAGAAAAIkCwBgAAACJAsAYAAAAiQLAGAAAAIhBLsDazd5vZHjM7YWbz8s79rZk9YWaPmdk1cdQHAAAAlGt8TK+7W9I7JX0hd9DMLpZ0naTZkl4l6QEzu9DdB6pfIgAAAFC6WGas3f1Rd3+swKnFkr7i7n9w919KekLSG6tbHQAAAFC+pK2xPlfSUznH+8MxAAAAINEqthTEzB6Q9MoCp1a7++ZiDysw5kWev0NShyTNmjVrVDUCAAAAUalYsHb3q0fxsP2Szss5ninp6SLP3yWpS5LmzZtXMHwDAAAA1ZK0pSBbJF1nZqeZ2fmSLpD0o5hrAgAAAEYUV7u9vzSz/ZIul/QtM/uuJLn7Hklfk/SIpPsl3URHEAAAAKRBLO323P0bkr5R5NwaSWuqWxEAAAAwNklbCgIAAACkEsEaAAAAiADBGgAAAIgAwRoAAACIgLmnvwW0mR2WtDfmMs6R9JuYa8Do8N6lG+9fevHepRvvX3rx3o1Nk7tPLXSiJoJ1EpjZDnefF3cdKB/vXbrx/qUX71268f6lF+9d5bAUBAAAAIgAwRoAAACIAME6Ol1xF4BR471LN96/9OK9Szfev/TivasQ1lgDAAAAEWDGGgAAAIgAwXoUzKzfzB42s11mtiMcm2JmW83s8fD2rLjrRMDM7jWzQ2a2O2es6PtlZn9rZk+Y2WNmdk08VSOryPv3cTP7VfgZ3GVmb885x/uXEGZ2npn1mdmjZrbHzG4Jx/n8Jdww7x2fvRQws0lm9iMz+2n4/t0ejvPZqzCWgoyCmfVLmufuv8kZ+5SkZ9z9E2Z2m6Sz3P0jcdWIU8zsSknPS/pf7n5JOFbw/TKziyVtkvRGSa+S9ICkC919IKby616R9+/jkp539zvy7sv7lyBmNkPSDHf/sZmdIWmnpHdIukF8/hJtmPfur8RnL/HMzCSd7u7Pm9kESd+XdIukd4rPXkUxYx2dxZI2hN9vUPAXEBLA3R+S9EzecLH3a7Gkr7j7H9z9l5KeUPAXDWJS5P0rhvcvQdz9gLv/OPz+OUmPSjpXfP4Sb5j3rhjeuwTxwPPh4YTwy8Vnr+II1qPjkv7NzHaaWUc4Nt3dD0jBX0iSpsVWHUpR7P06V9JTOffbr+H/Z4L43GxmPwuXimT/OZP3L6HMrFnSZZJ+KD5/qZL33kl89lLBzBrMbJekQ5K2ujufvSogWI/OFe7+Bklvk3RT+E/VqA1WYIz1UsmzTtKrJc2VdEDSneE4718CmdnLJP2rpA+6+++Hu2uBMd6/GBV47/jspYS7D7j7XEkzJb3RzC4Z5u68fxEhWI+Cuz8d3h6S9A0F/1xyMFyTll2bdii+ClGCYu/Xfknn5dxvpqSnq1wbRuDuB8P/aZyQ9EWd+idL3r+ECdd3/qukbnf/ejjM5y8FCr13fPbSx92flfSgpLeKz17FEazLZGanhxdyyMxOl/TnknZL2iJpaXi3pZI2x1MhSlTs/doi6TozO83Mzpd0gaQfxVAfhpH9H0PoLxV8BiXev0QJL6D6kqRH3X1tzik+fwlX7L3js5cOZjbVzM4Mv58s6WpJPxefvYobH3cBKTRd0jeCv3M0XtJGd7/fzP5T0tfM7K8l7ZP07hhrRA4z2yTpLZLOMbP9kj4m6RMq8H65+x4z+5qkRyQdl3QTV0XHq8j79xYzm6vgnyr7Jf2NxPuXQFdIul7Sw+FaT0n6qPj8pUGx966Nz14qzJC0wcwaFEyifs3de8zsB+KzV1G02wMAAAAiwFIQAAAAIAIEawAAACACBGsAAAAgAgRrAAAAIAIEawAAACACBGsAqHFm9nze8Q1mdnf4/cfN7FdmtsvMdpvZoniqBID0I1gDAD4Tbn38bkn3mhn/bwCAUeAvTwCAJMndH1WwOcQ5cdcCAGnEzosAUPsm5+yeJ0lTFGxhPIiZvUnSCUmHq1UYANQSgjUA1L6j4VIPScEaa0nzcs7/X2b2HknPSbrW2ZIXAEaFYA0A+Iy73xF3EQCQdqyxBgAAACJAsAYAAAAiYCylAwAAAMaOGWsAAAAgAgRrAAAAIAIEawAAACACBGsAAAAgAgRrAAAAIAIEawAAACACBGsAAAAgAgRrAAAAIAL/P02La6GvJnyMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting results\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12, 8))\n",
    "fig = sm.graphics.plot_fit(results, 1, ax=ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exactly the same exercise, but fit on a different response variable\n",
    "\n",
    "Y = np.log(data['MPG']).rename('log MPG')\n",
    "X = data[['HP']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P &gt; |t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>const</th>\n",
       "      <td>4.013229</td>\n",
       "      <td>0.040124</td>\n",
       "      <td>100.021194</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HP</th>\n",
       "      <td>-0.004589</td>\n",
       "      <td>0.000309</td>\n",
       "      <td>-14.873129</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           coef   std err           t  P > |t|\n",
       "const  4.013229  0.040124  100.021194      0.0\n",
       "HP    -0.004589  0.000309  -14.873129      0.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using manually coded solution\n",
    "get_regression(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            OLS Regression Results                            \n",
      "==============================================================================\n",
      "Dep. Variable:                log MPG   R-squared:                       0.734\n",
      "Model:                            OLS   Adj. R-squared:                  0.731\n",
      "Method:                 Least Squares   F-statistic:                     221.2\n",
      "Date:                Sun, 15 Mar 2020   Prob (F-statistic):           9.62e-25\n",
      "Time:                        17:06:26   Log-Likelihood:                 36.047\n",
      "No. Observations:                  82   AIC:                            -68.09\n",
      "Df Residuals:                      80   BIC:                            -63.28\n",
      "Df Model:                           1                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "==============================================================================\n",
      "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
      "------------------------------------------------------------------------------\n",
      "const          4.0132      0.040    100.021      0.000       3.933       4.093\n",
      "HP            -0.0046      0.000    -14.873      0.000      -0.005      -0.004\n",
      "==============================================================================\n",
      "Omnibus:                        4.454   Durbin-Watson:                   1.026\n",
      "Prob(Omnibus):                  0.108   Jarque-Bera (JB):                3.827\n",
      "Skew:                           0.516   Prob(JB):                        0.148\n",
      "Kurtosis:                       3.236   Cond. No.                         299.\n",
      "==============================================================================\n",
      "\n",
      "Warnings:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n"
     ]
    }
   ],
   "source": [
    "# Using statsmodels\n",
    "results = sm.OLS(Y, sm.add_constant(X)).fit()\n",
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAHwCAYAAABtz0NOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdfZhcdX3//9d7dxOThcidCSAhu1hpCyEmYEQhFVwDFu2aeIMtyZRilK7JgtUmP9B+821ttbmuil9SS2GDK/1G+u1m0XKXuAJK4iJCEEk0EAK1pTiBCE3CrcAGwu6+f3+cM8ns7MxkZnfOnLl5Pq5rrt3zOWfOvLNEr9d+8jnvj7m7AAAAAIxfQ9wFAAAAALWCcA0AAACUCOEaAAAAKBHCNQAAAFAihGsAAACgRAjXAAAAQIkQrgHUPTObYWavmlljhJ/xHTP7+wjue4+ZXVrq+wIAxoZwDaBumFnSzPaFQTr1eru7P+Xuh7v7UHjdqMBqZm5m74yncoyVmX3azO7LMp40s/PSrhkK/z781sy2mVl7+asFUAsI1wDqzUfDIJ16PRN3QfUqyn8pGIMH3P1wSUdK+hdJ3zOzo2OuCUAVIlwDqHtm1hrOTDeZ2SpJ75d0bTiTea2Z3Rte+nA49ifh+9rDWc6XzGyzmb0r7Z6nm9kvzOwVM/uupEk5Pvst4ftPSxubGs6wTzOzo8ysz8z2mtmL4ffTc9zrb83s37L9ucLjI8zsX8zsWTP7jZn9fSrgmtk7zewnZvaymT0X1pztM+4ys8szxh42s0+E3/++md1tZi+Y2a/M7I/TrvuOma0xszvM7DVJbWb2ETN7LPw5/cbM/r/w2lEzzun/epDrfePl7sOS/q+kyZLeUYp7AqgvhGsASOPuKyX9VNLl4cz25e5+Tnh6djj2XTM7Q0EI+5ykYyR9S9KGMCxPlHS7pP8n6WhJ/y7pkzk+7w1Jt0palDb8x5J+4u57FPz/9FpJLZJmSNon6dox/vFulDQo6Z2STpf0IUmp5S9fk/QjSUdJmi7pn3PcY116rWZ2aljbD8zsMEl3h9dMC6/rMrOZae9fLGmVpCmS7lMwS/w5d58i6TRJPy7wzzLW9+UV/iJyqaRXJf1XKe4JoL4QrgHUm9vDmeKXzOz2cdznzyV9y90fdPchd79R0huS3he+Jkj6pru/6e43S3ooz71GBFYFAXSdJLn78+5+i7sPuPsrCoLpucUWa2bHSvqwpC+6+2thcP9HSReFl7ypICS/3d1fd/dR65RDt0maY2Yt4XFC0q3hLwntkpLuvtbdB939F5JukXRh2vvXu/v97j7s7q+Hn3uqmb3V3V8M31OIYt73vrT/5i+Z2UsKflEZdY2k/1Hw3+Lj7v5ygbUAwAGEawD15mPufmT4+tg47tMiaUVGYDtR0tvD12/c3dOu35nnXj+WNNnM3huG1jkKQqzMrNnMvmVmO83st5LulXTkGNYrtygI/M+m1fstBTPMknSlJJP0czPbYWafyXaTMOD/QAdD+UWSetI+470ZP5OEpOPSbvF0xi0/KekjknaGy1LOKvDPU8z7fpb23/xIdz9S0lM5rnmbu7/P3TcWWAcAjNAUdwEAUIH80JfoaUmr3H1V5gkzO1fSCWZmaQF7hqT/zvph7sNm9j0FM6a7JfWFIVaSVkj6PUnvdff/MbM5kn6pIAhnek1Sc9pxZqh9Q9Lb3H0wSw3/o2A2Xmb2B5I2mtm97v5Els/plfSVcC36ZEn9aZ/xE3c/P9ufM/VRGZ/7kKSFZjZB0uWSvqfgl5QRfxYzO67A9wFArJi5BoDRdmv0w2yZY9+WtDScbTYzO8zM/sjMpkh6QMHa5r+w4CHJT0g68xCfuU7SnyiY6V2XNj5FwTrrlyzoXvGVPPfYJukcC/p2HyHpr1In3P1ZBWuqrzazt5pZg5n9TviLgMzsU2kPSr6oIAQP5ficOxTMUn9V0nfDhwAlqU/S75rZxWY2IXy9x8xOyXYTM5toZgkzO8Ld35T027TPfFjSTDObY2aTJP1tge8DgFgRrgFgtH+SdKEF3TmuCcf+VtKN4XKHP3b3LQpmeq9VEEafkPRpSXL3/ZI+ER6/qCA035rvA939QQWztW+XdGfaqW8qmB1+TtLPJN2V5x53S/qupEckbVUQdtP9maSJkh4L67pZ0vHhufdIetDMXpW0QdIX3P3XOT4n9RDmeUr7RSCcbf+QgqUizyhYv/x1SW/J80e/WFIyXPKyVNKfhvf6TwXhfaOCBwsz14BnfR8AxM1GLgkEAAAAMFbMXAMAAAAlQrgGAAAASoRwDQAAAJQI4RoAAAAoEcI1AAAAUCI1tYnM2972Nm9tbY27DAAAANSwrVu3PufuU7Odq6lw3draqi1btsRdBgAAAGqYme3MdS7yZSFm1mhmvzSzzM0MFO6w9Uj42mxms9POJc1su5ltMzMSMwAAACpeOWauvyDpcUlvzXLu15LOdfcXzezDkrolvTftfJu7P1eGGgEAAIBxi3Tm2symS/ojSTdkO+/um939xfDwZ5KmR1kPAAAAEKWoZ66/KelKSVMKuPazku5MO3ZJPzIzl/Qtd++OoD4AAIC68eabb2rXrl16/fXX4y6lKkyaNEnTp0/XhAkTCn5PZOHazNol7XH3rWb2gUNc26YgXP9B2vA8d3/GzKZJutvM/sPd783y3g5JHZI0Y8aMktUPAABQa3bt2qUpU6aotbVVZhZ3ORXN3fX8889r165dOumkkwp+X5TLQuZJWmBmSUk3Sfqgmf1b5kVm9i4Fy0YWuvvzqXF3fyb8ukfSbZLOzPYh7t7t7nPdfe7UqVk7ogAAAEDS66+/rmOOOYZgXQAz0zHHHFP0LH9k4drd/8rdp7t7q6SLJP3Y3f80/RozmyHpVkkXu/t/po0fZmZTUt9L+pCkR6OqFQAAoF4QrAs3lp9V2XdoNLOlZrY0PPwbScdI6spouXespPvM7GFJP5f0A3e/q9y1AgAAoLQOP/zwktzn05/+tJqbm/XKK68cGPvCF74gM9NzzwXN5hobGzVnzhyddtpp+tSnPqWBgQFJ0u7du7V48WK94x3v0Lvf/W6dddZZuu2220pSV1nCtbvf4+7t4ffXu/v14feXuvtR7j4nfM0Nx59099nha6a7rypHnQAAADiop0dqbZUaGoKvPT1xVzTSO9/5Tq1fv16SNDw8rP7+fp1wwgkHzk+ePFnbtm3To48+qokTJ+r666+Xu+tjH/uYzjnnHD355JPaunWrbrrpJu3ataskNZV95hoAAACVr6dH6uiQdu6U3IOvHR2lC9juriuuuEKnnXaaZs2ape9+97uSgpDc2dmpmTNnqr29XR/5yEd08803Z73HokWLDrzvnnvu0bx589TUlL1fx/vf/3498cQT+vGPf6yJEydq6dKlB861tLTo85//fEn+XIRrAAAAjLJypRSuojhgYCAYL4Vbb71V27Zt08MPP6yNGzfqiiuu0LPPPqtbb71VyWRS27dv1w033KAHHngg5z1OPvlk7d27Vy+++KJ6e3t10UUXZb1ucHBQd955p2bNmqUdO3bojDPOKM0fIgvCNQAAAEZ56qnixot13333adGiRWpsbNSxxx6rc889Vw899JDuu+8+fepTn1JDQ4OOO+44tbW15b3PJz7xCd1000168MEH9f73v3/EuX379mnOnDmaO3euZsyYoc9+9rOj3n/ZZZdp9uzZes973lOSP1c5tj8HAABAlZkxI1gKkm28FNy9qPFcLrroIp1xxhm65JJL1NAwct44teY63cyZM3XLLbccOL7uuuv03HPPae7cuUV9bi7MXAMAAGCUVauk5uaRY83NwXgpnHPOOfrud7+roaEh7d27V/fee6/OPPNM/cEf/IFuueUWDQ8Pa/fu3brnnnvy3mfGjBlatWqVOjs7C/rcD37wg3r99de1Zs2aA2MDmetfxoGZawAAAIySSARfV64MloLMmBEE69T4eH384x/XAw88oNmzZ8vMdNVVV+m4447TJz/5SW3atEmnnXaafvd3f1fvfe97dcQRR+S91+c+97mCP9fMdPvtt+sv//IvddVVV2nq1Kk67LDD9PWvf328f6Tg/sVOvVeyuXPn+pYtWw59YYn09ET3Fw4AAKDUHn/8cZ1yyilxl3FIr776qg4//HA9//zzOvPMM3X//ffruOOOi6WWbD8zM9uaaiGdiZnrMUq1p0n9K0KqPY1EwAYAABiP9vZ2vfTSS9q/f7/++q//OrZgPRaE6zHK156GcA0AADB2h1pnXcl4oHGMom5PAwAAgOpDuB6jXG1oStWeBgAAANWHcD1GUbenAQAAQPUhXI9RIiF1d0stLZJZ8LW7m/XWAAAA9YxwPQ6JhJRMSsPDwVeCNQAAqDn9/VJra/C1RK655hqdcsopOuqoo/QP//APkqTbb79djz322IFrvvOd7+iZZ54p6r7JZFKnnXZayeocC8I1AAAAsuvvl9rbg57D7e0lC9hdXV2644479OKLL+rLX/6ypNKE60pAuAYAAMBoqWCd6j08MFCSgL106VI9+eSTWrBggf7xH/9Rl19+uTZv3qwNGzboiiuu0Jw5c/T1r39dW7ZsUSKR0Jw5c7Rv3z5t3bpV5557rt797nfrD//wD/Xss89KkrZu3arZs2frrLPO0nXXXTfeP/W4Ea4BAAAwUmawTilBwL7++uv19re/Xf39/TrqqKMkSWeffbYWLFigb3zjG9q2bZu+9KUvae7cuerp6dG2bdvU1NSkz3/+87r55pu1detWfeYzn9HKlSslSUuWLNE111yjBx54YMw1lRKbyAAAAGCkJUtGB+uUgYHgfDJZtnJ+9atf6dFHH9X5558vSRoaGtLxxx+vl19+WS+99JLOPfdcSdLFF1+sO++8s2x1ZUO4BgAAwEhr12afuZaC3sNr15a1HHfXzJkzR81Ov/TSSzKzstZyKCwLAQAAwEhtbVJfX/ZNPfr6gvMlNmXKFL3yyitZj3/v935Pe/fuPRCu33zzTe3YsUNHHnmkjjjiCN13332SpJ6enpLXVSzCNQAAAEbLDNgRBmtJuuiii/SNb3xDp59+uv77v/9bn/70p7V06VLNmTNHQ0NDuvnmm/WlL31Js2fP1pw5c7R582ZJ0tq1a3XZZZfprLPO0uTJkyOprRjm7nHXUDJz5871LVu2xF0GAABARXr88cd1yimnFPem/v5gjfXatZEF60qW7WdmZlvdfW6261lzDQAAgNza2sr68GK1Y1lICaxbty7uEgAAAFABCNcl0NvbG3cJAAAAqACEawAAgDpSS8/bRW0sPyvCdcRKvWSkp0dqbZUaGoKvFdBxBgAAVIlJkybp+eefJ2AXwN31/PPPa9KkSUW9jwcaI9bb26vFixdr3bp1Wrx48bju1dMjdXQc7Oe+c2dwLEmJxDgLBQAANW/69OnatWuX9u7dG3cpVWHSpEmaPn16Ue8hXJdJKmSPx8qVozdKGhgIxgnXAADgUCZMmKCTTjop7jJqGstCYnL++ecX/Z6nnipuHAAAAOVFuI7JvffeW/R7ZswobhwAAADlRbiuIqtWHdyBNKW5ORgHAABA/AjXVSSRkLq7pZYWySz42t3NemsAAIBKwQONVSaRIEwDAABUKmauAQAAgBIhXFepyy67LO4SAAAAkIFwXaVuvvnmuEsAAABABsJ1mXV2Sk1N0v79r6upKTgGAABAbeCBxjLq7JTWrEkdmYaGDh53dcVVFQAAAEqFmesy6u4ubny81q1bF82NAQAAkBXhuoyGhoobH6/e3t5obgwAAICsIg/XZtZoZr80s74s58zMrjGzJ8zsETM7I+3cBWb2q/Dcl6OusxwaG4sbBwAAQHUpx8z1FyQ9nuPchyWdHL46JK2RgkAu6brw/KmSFpnZqdGXGq2OjuLGAQAAUF0iDddmNl3SH0m6IcclCyX9qwd+JulIMzte0pmSnnD3J919v6SbwmurWleXtGxZaqba1dgYHMf5MCP9sgEAAEon6pnrb0q6UtJwjvMnSHo67XhXOJZrvOp1dUmDg9LEiZM0OBh/lxD6ZQMAAJROZOHazNol7XH3rfkuyzLmecazfU6HmW0xsy179+4dQ6UAAABAaUQ5cz1P0gIzSypY1vFBM/u3jGt2STox7Xi6pGfyjI/i7t3uPtfd506dOrVUtVesnh6ptVXas+dZtbYGx1E5//zzJdHSDwAAoFCRhWt3/yt3n+7urZIukvRjd//TjMs2SPqzsGvI+yS97O7PSnpI0slmdpKZTQzfvyGqWqtFT0/w8OPOnZLUoJ07g+OoAva9994riZZ+AAAAhSp7n2szW2pmS8PDOyQ9KekJSd+W1ClJ7j4o6XJJP1TQaeR77r6j3LVWmpUrpYGBkWMDA8E4AAAA4leW7c/d/R5J94TfX5827pKytqtw9zsUhG+EnnqquHEAAACUFzs0VpEZM4obLyda+gEAABCuq8qqVVJz88ix5uZgPG609AMAACBcV5VEQurullpaJGlYLS3BcSIRd2Wj0WEEAADUI8J1lUkkpGRSmjbteCWTlRmspYMdRlguAgAA6gnhGpFiuQgAAKgnhGsAAACgRAjXiA1LRgAAQK0hXCM2LBkBAAC1hnANAAAAlAjhGgAAACgRwjUAAABQIoRrVJVZs2bFXQIAAEBOhGtUlR07dsRdAgAAQE6E6zrV0yO1tkp79jyr1tbguFodfvjhcZcAAAAgiXBdl3p6pI4OaedOSWrQzp3BcbUG7Ndee02StG7dupgrAQAA9Y5wXYdWrpQGBkaODQwE49Wst7c37hIAAECdI1zXoaeeKm68mp144olxlwAAAOoI4boOzZhR3Hg127VrlySWjAAAgPIgXNehVauk5uaRY83NwXitYskIAAAoB8J1HUokpO5uqaVFkobV0hIcJxJxV1Y+zGQDAIAoEK7rVCIhJZPStGnHK5msr2AtZZ/JZoMaAAAwXoRrIMQGNQAAYLwI1xFJbdLS17dera3Srl3nxl0SikSnEQAAUCzCdQSybdKyffvlVbtJS72i0wgAACgW4ToC2TZpGRqapEsuqd5dEOsZnUYAAEChCNcRyLUZy9BQ5Wwznlq2smfPs2ptrYyaAAAAqh3hOgL5NmOphG3Gsy1bqZTQDwAAUM0I1xHItklLuri3Gc+2bKUSQj8AAEC1I1xHILVJS2Nj9vMNDdL+/ftiW46RK9zHHfoBAACqHeE6IomEdOONUmPj66PODQ1JcS7HyLVsJd9yFgAAABwa4TpCiYQ0a9a1B7YZzzaTHcdyjGzLVpqbg3EAAACMHeE6YtOn/0TJpNTevlDDw9mvKfdyjNSylVTob2kJjuttC3QAAIBSI1yXUSUtx0gkpGRSmjbteCWTBGsAAIBSIFyPV3+/bti0ServP+SlLMcAAACobYTr8ejvl9rbdey+fVJ7+yEDNssxcCgnnnhi3CUAAIBxIFyPVRisDzSMHhgoOGAnk9LEiZNZjoFRdu3aFXcJAABgHAjXY5EZrFMyA3Zqycjq1QUvHQEAAED1IlyPxZIlo4N1ysBAcD59yciKFQUvHal2PT1Sa6vU17dera3Svn0fj7skAACAsiFcj8Xatbn3N29ulv7iL/LPbC9cqNf375c6O6OvtYx6eoJNcXbulFKb5LzyyupYdqEEAACIA+F6LNrapL6+7K0/vvY16a//Ov/M9oYNMklas6amAvbKldn+2M1l3ySnXqxbty7uEgAAQAbC9VhlBuzm5uD4mmtyB+tsaihg59oMp9yb5NSL3t7euEsAAAAZIgvXZjbJzH5uZg+b2Q4z+7ss11xhZtvC16NmNmRmR4fnkma2PTy3Jao6xyUM2LsnTw6CdVtb/iUjudRIwK6kTXIAAADiEOXM9RuSPujusyXNkXSBmb0v/QJ3/4a7z3H3OZL+StJP3P2FtEvawvNzI6xzfNradOn8+UGwDo+zLhk5lDVrSl9bmWXbJEcaYJMcAABQNyIL1x54NTycEL48z1sWSaqNf+ceS8Betiy6esok2yY5U6YsL0kv71QXEvdBtbaKhyQBAEBFinTNtZk1mtk2SXsk3e3uD+a4rlnSBZJuSRt2ST8ys61m1pHnMzrMbIuZbdm7d28pyx+f9CUjV18dfF2wIPu1y5ZJXV3lrS8iqU1y2tsXKpmUJk++bdz3zNaFpKODgA0AACpPpOHa3YfCJR/TJZ1pZqfluPSjku7PWBIyz93PkPRhSZeZ2Tk5PqPb3ee6+9ypU6eWtP5xSy0ZWb48+Lp+/egZ6rEE69Wr9Zs9e6TVq0tXawXL1oVkYEB0IQEAABWnLN1C3P0lSfcomJ3O5iJlLAlx92fCr3sk3SbpzAhLLJ+uLmnZsmB9zBiDtVasUJMkrVhRFwGbLiQAAKBaRNktZKqZHRl+P1nSeZL+I8t1R0g6V9L6tLHDzGxK6ntJH5L0aFS1ll1XlyZNnDjmYD1CHQRsupAAAIBqEeXM9fGS+s3sEUkPKVhz3WdmS81sadp1H5f0I3d/LW3sWEn3mdnDkn4u6QfufleEtVa+bME6JVvA7u/XDZs2jWm79dTDg/v371Nrq7Rr17lF36OUsnUhaW4WXUgidP7558ddAgAAVSnKbiGPuPvp7v4udz/N3b8ajl/v7tenXfcdd78o471Puvvs8DXT3YlRV15Z+Pn+fqm9Xcfu2xdst15EwM728OD27ZfH+vBgti4k3d1BS3GzYMxM2rx5VCt1jNG9994bdwkAAFQldmisFlddVdj5MFgfeAJwYKCogJ3t4cGhoUm65JJ4u3OkupCYNSmZDIL1pk2psyZJeuGF03XeeTEVCAAAIMJ19Vi+XLr66uznrr46OJ8ZrFOKCNi5HhIcGqqs9ncHg3U6yzEOAABQHoTrapItYKeCtSQtWTI6WKcMDATn02Vp6ZfvIUHa3wEAAORHuK42YcAelEYGaylYK5FrV8jm5uB8So6Wfh/5SP6Pp/0dAABAboTrarR8uU6YNm1ksJZyb7ve3ByMt7UFx3la+t1xR/6PrpT2d/PnZxv1A+OpjifSUEV0PAEAAPWBcF1rMgN2IcE6ZcUKfWJnMIN9rTo1LNO16jxwupLa323cmB6wXZJ09NG/1MaNldnxBAAA1AfCdQ3qeaZNi6b0KakZWjSlTz3PtB08eYiWflfpSl2rTnVqjUxSp9boZU3R/IZ+dXcHXTsqxcaNkrskNchdOvvsr0jK3fGE9eIAACBqhOsak5q1vWl3m07STt20u21kl49DtPR7/p3vPRCspaDJ3Vv1qn6kDynx9sL7Ze/ada5aW6U9e55Va2t5u4ywXXrlmjVrVtwlAAAQKcJ1jck2azuiy0e+ln5nn61pT2w+EKzTNQwPSh/6UEHt/Hp6gmUY6csyytnGj+3SK9eOHTviLgEAgEgRrmtMQbO2uVr6bc4erA8YHBzVL7unR/ra0av1mz3P62tHr1ZPTxDkh4YmjXhrOdv4ZdsuvbHx9YpZLw4AAGoX4brGFDxrm62l37Jl4aOBeaT1y+7pkR5Zslr/+8UVatKQ/veLK/TIktXhjPVopV6WkasjSCIhXXKJ1NgoSa7GRumEEzZW1HpxAABQmwjXNSbbrG3OLh+ZLf26uvT/Dj9EwE7rl/3k51frH95cMWJ99j+8uULXqVO/Vqs+oJFLSEq5LCNfR5CeHunGG4NdJSXT0JD0m9+cR7eQKrNu3bq4SwAAoGiE6xqTSEjd3VJLiyQNq6VFRXX5aLy+S92NOQJ2elu/1cGMdeYyEpO0TGvUqp26S3+oQTXqi1pd8jZ++TqC0C2kNvT29sZdAgAARSNc16BEQkompfb2hUomi2ufl0hIh994cAZ7MNjDcXS/7CuvzLk+OzX+Fr2pRg1rtVbo3o+tLumyjFxLT3bupFsIAACID+EaoyQS0p+90qVJEyeq6cc/0u7Jk0cGa0m66qpDr88OmaR3rzu4xXrU6BYCAADiQrhGfm1tunT+/JHBWpKWL9ffH3V1wQFb0oEt1iVJq1frN3v2RBK46RYCAADiQrjGmH3lpeVaroMBu6CgfeWVB7Zgb5JGBu4SybbufNasa+kWUsN4+BEAUCkI1xizGTOkbyoI2INqVJeW6Q1NyP+mP/qjIFCnG0PAnj8/26gfGE+tO5calUxK06f/pKj7o7rw8CMAoFIQrjFmqeUX39RyTdCgLleXPvaWH2qoIUfAXrBA2rAh+7kiA/bGjekBO5gzP/roX2rjxsLrBwAAKDXCNcYs2/KLP/2XNjVu/KF+29Q08uKrr5Z+8IP8N7zyyqI+f+NGyV2SGuQunX32V4p6PwAAQKkRrjEuqeUX06Ydf7DtX1ubEhdcMHoHyKuuyn+zQ51fuFBD7tLChSWpHQAAoNQI14hO5g6Q4ZbrWaUCeC4LF0obNgQ9tDdsIGADAICKRLhGeWUL2AUG6xFSAfukkzQsSSedVOpKAQAAika4RvmFAXvEkpFcsgXrlA0bpGQymM1OJnV7X5/U31/CQheptVXq61uv1lapp6eEtwYAADWJcI2y6+yUmq5crgkaVtOVy9XZmefiXME6i0YpaCFSgoAdBOlvh9usN2jnTqmjg4ANAADyI1yjrDo7pTVrpKEhSTINDQXHOQP2ggXFfYD7wYA9b16wZGTevKLrXLlSkg4bMTYwkBpHPWGDGgBAMQjXKKvu7uLGtX792AP25s3BkpHNm3XLD35Q1Iz2U08VN47axQY1AIBiEK5RVsGMdeHjksYesNNMdJfOO6/ggD1jRnHjqB+XXXZZ3CUAACoY4Rpl1dhY3PgBYcB2KQjara3Ff/jwcBCwV6/WDZs25QzanZ2pGeqRAb25OdiVEvXt5ptvjrsEAEAFI1yjrDo6ihsfYf16NZoFQfvXvx57wF6xQsfu2ye1tx8M2OEGNQ+ftFBr1qQmvi18k+vww4OlK4lE8R8JAADqB+Eakdi161y1tkp79jw7oo1dV5e0bFlqptrV2Bgcd3Xlv19np9TUJLkPqakpfAAyDNhBDg6DcOprIQYGgoA9b96BDWreldyg25S5QY1p3z6CNQ5t1qxZcZcAAIgZ4Rol19Mjbd9+ec42dl1d0uCgNG3acRocLCxY5+ww8utfB3+JNyedQ3IAACAASURBVG3S7smTpU2bpLPPLrzYgQFp8+YDhyZpoTZor47WsEw/VdBpJO+acCC0Y8eOuEsAAMSMcI2SW7lSGhqaNGJsPG3sCuow0tamS+fPl9rapPvvLy5gZzBJx+hFmaR52qx9mqT5DaXcnAYAANQqwjVKrtRt7MbUYSQM2C5JDcX/Nbe0r5P0hn40XHinEQAAUL8I1yi5UrexG3OHkfvvD/6Cb9wYLBm5+uqg5ccYNGi4qFZ+AACgPhGuUXKrVkmNja+PGBtPG7txdRiRDi4ZWb5c6us7GLCbm4tbPjI8PLLDCAAAQAbCNUoukZBmzbpWLS2SNKyWlvG1sRtrh5Gs2tqkvr5gJruvL1g+UswGNQMD0pIlY/hgAABQDwjXiMT06T9RMilNm3a8ksnxt7FLdRgxayyow0he6Q8/SiM3qDnqqPzvbW6W1q4dx4cDAIBaRrgGpIMb1LzwQu6lIs3NwWx3KpSXUE9Pak+coRF9wQEAQHUhXAOZsnUaiThYd3QoZ19wYN26dXGXAAAoEOEayCaz00hEwVoK+n8PDIwcG09fcNSe3t7euEsAABQosnBtZpPM7Odm9rCZ7TCzv8tyzQfM7GUz2xa+/ibt3AVm9isze8LMvhxVnUCmEUs0lrTpgt/7amTBWip9X3AAABCfKGeu35D0QXefLWmOpAvM7H1Zrvupu88JX1+VJDNrlHSdpA9LOlXSIjM7NcJaAUnZl2hs3355pEs0St0XHAAAxCeycO2BV8PDCeHLC3z7mZKecPcn3X2/pJskLYygTGCEbEs0hoYmRbpEY9WqYEn3TzVPwzL9VPPG1Rcc9Y312QAQr0jXXJtZo5ltk7RH0t3u/mCWy84Kl47caWYzw7ETJD2dds2ucCzbZ3SY2RYz27J3796S1o/6E8xYFz5eComE9PCUeZqnzTJJ87RZD0+ZN+72hahPrM8GgHhFGq7dfcjd50iaLulMMzst45JfSGoJl478s6Tbw3HLdrscn9Ht7nPdfe7UqVNLVTrq1Ji3Wh+H3SfP0+/s3nzgL71J+p3dm7X75HnRfSgAAIhEWbqFuPtLku6RdEHG+G9TS0fc/Q5JE8zsbQpmqk9Mu3S6pGfKUSvq29BQcePjNm+epj2xedRvkyZp2hObpXnlDdiphzn3799Hv20AAMYgym4hU83syPD7yZLOk/QfGdccZ2YWfn9mWM/zkh6SdLKZnWRmEyVdJGlDVLWi9px3nhT8zRqWmbR586hmNVkFW7YXPj5um0cH6xQLz5cL/bYBABi/KGeuj5fUb2aPKAjLd7t7n5ktNbOl4TUXSnrUzB6WdI2ki8IHIQclXS7ph5Iel/Q9d98RYa2oIeedJ23alDoKousLL5yu88479Hs/8pFso55jvARSm9Vk/VTl3i2yv183bNok9feXrBT6bQMAMH5NUd3Y3R+RdHqW8evTvr9W0rU53n+HpDuiqg+162CwTmc5xke6I+vfOMsxXgL33689J49eGuKS9rzzbB17//2j39PfL7W369h9+6T29pJtcEO/bQAAxo8dGktg0aJFcZeAEokjYB77X/drzzsPzmAfCNb/lTtYH5hiHhgIjkswg02/bQAAxo9wXQKLFy+Ou4Q6t0itrVJf3/pwZ8XipB7icx9UQ47/RUQdMI/9r/tl4RIRO7vAYJ1SooCd6redjn7bAAAUh3CNqhY8bPftEQ/hZeeaP/9gkE4F8c5O6TOfOfgQX7auII2Nr0ceMDs7paYH71eDhtX04P3q7Mxy0ZIlo4N1ysBAcH4cEgmpuzv18OawWlqCY/ptAwBQOMJ1xFgyEq3gYbvDRo1PnJj6LlhscfTRv9SSJaO7YaxZI+3fP/q+wQx2EDBnzbo20oDZ2RnUEQR709BQcDwqYK9dO3pqOaW5OTgPAABiRbiOWGrJCCE7GrnWQr/5puQuSQ1yl84++ytZu2HkMjwsSY1KJqXp039Sklpz6e4ucLytLXh4MdvajRI81EgrPgAAxo9wXSasy45GMQ/hVWrXi6I2rskM2CUK1hKt+AAAKAXCdUzOOeecuEuoCcFa6NdGjOV6CK+YhxKPOWZcZRWl6C3Xw4C9e/LkkgVriVZ8AACUAuE6JnfffXfcJdSEYC30nxf0EF62bhgTJowOsRMmSP/0T9HUm01HR3HjkqS2Nl06f37JgrV08JePa9WpYZmuVeeIcQAAcGiEa9SAXiWTUnv7QiWTubtbZOuGsXatdOONo8fK2SGjq0tatiwV8l2NjcFxV1f5apCCXz6ub+xUp9bIJHVqja5v7KQVX5XI7ITDWnkAiEdkOzQClSiRCF4f/ehCff/73x8x3tDQpGRyOJa6urqCl1mDBgdzbYgercT9nfKhNQd2ijRJHUNrZPdLSpQ56aMoqYdRgzXzBx9GlWilCADlxsw1gAP9AC1j2KQcfQFRSXgYFQAqB+EaQBCgx3MeseJhVACoHITrKnXhhRfGXQJqybJl4zuPWBXTkhIAEC3CdZW67rrr4i4BtST1VGU2cTxdiaJk64STqyUlACBahGtUhVQnBPdBOiFEJVvAJlhXhWydcHK1pAQARItwjYrHttxlFAZslwjWVSaRUEEtKQEA0SJc17BFixbFXUJJ0AmhzLq6NGnixNIH64ULNeQuLVxY2vsCAFBBCNc1bPHixXGXUBJ0QqgBCxdKGzYErf02bCBgAwBqFuEaFY9OCFUuDNYjVEHAZsdDAMBYEK7rXDW09KMTQhXLFqxTKjhgs84fADBWhOs6Vw0t/eiEUMVyBetCz8eEdf4AgLEiXKMqpDohmDXRCaGaLFgwtvP9/bph0yapv7/0NRWAdf4AgLEiXCOnc845J+4SUO3Wr88doBcsCM5n6u+X2tt17L59Unt7LAGbdf4AgLEiXCOnu+++W1LttPRDTLIF7EME6wNrMgYGYgnYrPMHAIwV4RqHVCst/RCjMGC7VHiwTokhYLPOHwAwVoRr1LhFh2yn1tkpNTVJ7kNqagqOyy3V9k0airXtW6qO/fv3lb6O9evVaJY9WEvSkiWjg3XKwEBwPpuI1mez4yEAYCya4i4AtaESW/oFwfDbo9qppevslNasSR2ZhobSj8sj1fYtyJUj6yxnoIu9jrVrs89cS8GajLVrR49nrs/u65Pa2qKvFQCAHJi5RklUYku/oG3aYSPGMtupdXdnf2+u8ShUStu32Otoa5P6+jQ4ceRi58GJzdlDc4WszwYAIB3hGpGohIcgC2mnNjSU/Zpc41GolLZvlVBHzzNt+qj16TUFAfs1Neuj1qeeZw4RrFMI2ACAmBGuEYnUQ5BxLhcppJ1aY2P2a3KNR6FS2r5VQh0rV0p3vdGmdvUpqRa1q093vdE2evZ8rOuzAQCIGOEakYpzuUjQNu21EWOZ7dQy12AfajwKldL2rRLqSM2S36M2naSk7lHbiPED1q4dXWxKrvXZAACUAeEaNSt4CO/P87ZT6+qSli1LzVS7GhuD466u8tZZCW3fKqGOgmfPw/XZWX8b4KFGAECMCNeITXmWjPQesp1aV5c0OCiZNWpwsLzBOiXV9k1qjLXtW6qOiRMnx1JHUbPnmQGbYA0AqACEa8SmEjuMIF5Fz56HAXv35MkEawBARSBcA6goRW/e0tamS+fPjyZYR7RBDQCgdhGuASCbzA1qCNgAgAIQrgEgU64Najo79Zs9e6TVq+OtDwBQsQjXqCozZ86MuwTUunwb1KxZoyZJWrGCgA0AyCpnuDazhWZ2Wdrxg2b2ZPiKb2cQ1LXt27fHXQJqXb4NatIRsAEAWeSbub5S0oa047dIeo+kD0haFmFNQFEOO+ywuEtALcm3QU2m9IC9cKGG3KWFC6OrDQBQ8fKF64nu/nTa8X3u/ry7PyXpkGnGzCaZ2c/N7GEz22Fmf5flmoSZPRK+NpvZ7LRzSTPbbmbbzGxLUX8q1JVXX31VkrRo0aKYK0FNyLVBTS5XXhkE6g0bZJK0YQMBGwDqWL5wfVT6gbtfnnY4tYB7vyHpg+4+W9IcSReY2fsyrvm1pHPd/V2SviapO+N8m7vPcfe5BXwe6tzixYvjLgG1opiA/fu/HwTqdARsAKhb+cL1g2b255mDZvY5ST8/1I098Gp4OCF8ecY1m939xfDwZ5KmF1Q1UKDp02vvr1RPj9TaKvX1rVdra3CMCKRvULMsx0q4mTOlHTuynyNgA0Bdyheu/1LSEjPrN7Orw9c9kj4t6YuF3NzMGs1sm6Q9ku529wfzXP5ZSXemHbukH5nZVjPrKOTzgExPPx2sbKqVJSM9PVJHh7RzpyQ1aOfO4JiAHZHUBjVdXdLVV488d/XVuYN1SuaMNhCBdevWxV0CgDQ5w7W773H3sxUs10iGr6+6+1nuvruQm7v7kLvPUTAjfaaZnZbtOjNrUxCuv5Q2PM/dz5D0YUmXmdk5Od7bYWZbzGzL3r17CykLdahWloysXJm9Q9zKlfHUU1eWL5euvlqDUhCsly+XFizI/55DnQdKoLe3N+4SAKTJ14pvkpl9UdInJO2XtMbdfzyWD3H3lyTdI+mCLJ/zLkk3SFro7s+nveeZ8OseSbdJOjPHvbvdfa67z506tZCl4ED1zmQ/9VRx45UitZTFfbC6l7IsX64Tpk0LgrUkrV+fO0AvWBCcB8qMmWwgXvmWhdwoaa6k7Qpmj/9PMTc2s6lmdmT4/WRJ50n6j4xrZki6VdLF7v6faeOHmdmU1PeSPiTp0WI+H8gn20x2NWxQM2NGceOVoOaXsmQL2ARrxIiZbCBe+cL1qe7+p+7+LUkXSsq6LCOP4yX1m9kjkh5SsOa6z8yWmtnS8Jq/kXSMpK6MlnvHSrrPzB5W8PDkD9z9riI/HyhKNWxQs2rV6AYWzc3BeKWqi6UsYcB2iWANAHWuKc+5N1PfuPugmRV1Y3d/RNLpWcavT/v+UkmXZrnmSUmzM8eBcqrETiOJRPB15Upp585htbQ0aNWqg+OVqFqXshRt/Xo1NjRouNTBur9fN2zaFGzL3tZW2nsDAEou38z1bDP7bfh6RdK7Ut+b2W/LVSAQl0rtNJJISMmk1N6+UMlkZQdrqTqXslSM/n6pvV3H7tsntbcHxwCAipavW0iju781fE1x96a0799aziKBONVKp5G4VONSlooQBusDa2oGBgjYAFAF8nULOTrfq5xFAqheiYTU3S21tEjSsFpaguNKn3GPVWawTiFgowzoNgKMT741189J2iUFbV0lpS+6dknviKooALUlkQheDQ1NSiaH4y6n8i1ZMjpYpwwMBOeTybKWhPrR29vLv9gB45BvzfU/S3pR0l2SLpH0Dnc/KXwRrAEgKmvXjl5Lk9LcHJzPJv3hRwBALPKtuf6CpDmS/l3SxZJ+aWZXmdlJ5SoOAOpSW5vU15d9sXpfX/auITz8CAAVId/MtTzQL+lKSddLWqJgMxgAQJQyA3YBwZqHHwEgfvkeaDzMzBab2XpJd0g6XNIZ7v7tslUHoCbUzPbn5RYG7N2TJxcerFMI2AAQi3wPNO6R9F+SeiU9oeAhxveY2Xskyd1vjb48ANUutf15kP0Obn8u0TGkIG1tunT+fH0/1wYyPPwIABUl37KQf5f0S0m/L6ld0kfTXu3RlwagFtTF9udxGuvDjwCASOScuXb3T5exDgA1Ksrtz3t6UlvBr1drq2LbCn7mTOmxxyRpSGbSqadKO3aU6cNTa7Mzl4bkW6MNAIhM3gcaAZTX9OnT4y6h5KLa/jy13GTnTil9uUlU67lT68b37Hl2xLrxg8FaSm0H8NhjwXjZFPPwI5BD6u94X996no0oQurn1tAgfm6QRLgGKsrTTz8ddwklF9X25+VcbpIvyB8M1iPlGo9MIQ8/AjmU+5fVWpH+c3MXPzdIIlwDiFhU259HudwkU9WsGw8ffiRYo1hV83e8wvBzQzb5uoVIkszsE1mGX5a03d33lL4kALUmiu3PZ8xIzbKNHi+1cgZ5IA78HR8bfm7IppCZ689KukFSInx9W9JySfeb2cUR1gYAOUW13CSbo4/OPX7qqdnP5RoHKlFUz0bUOn5uyKaQcD0s6RR3/6S7f1LSqZLekPReSV+KsjgAuS1atCjuEgrW2Sk1NUnuQ2pqCo7HK6rlJsXasSM9SLukMncLAUqgnL+s1hJ+bsimkHDd6u670473SPpdd39B0pvRlAXgUBYvXhx3CQXp7JTWrJGGhiTJNDQUHJcqYCeTUnv7QiWT0QXrF17IP75jR/Awk1mj3AnWqD6V8stqtUn/uZmJnxskFRauf2pmfWZ2iZldImmDpHvN7DBJL0VbHoBq191d3Hgl4p9+UQ/K9cvqeK1bty7uEkZI/dyGh1XRPzeUTyHh+jJJayXNkXS6pBslXebur7k7j6QDyCuYsS58vBLxT79A5ejt7Y27BCCvQ3YLcXc3s/sk7VewoPDn7u6RVwagJjQ2Zg/SjY3lr2WsUjNRwW6Qw2ppaYhtN0gAQGU75My1mf2xpJ9LulDSH0t60MwujLowALWho6O48UqV+qffadOO559+AQA5HXLmWtJKSe9J9bQ2s6mSNkq6OcrCANSGrq7ga3e3NDTkamw0dXQcHAcAoJYUEq4bMjaLeV7s7AigCF1dwauhoVGDg6XZRAYAgEpUSLi+y8x+KCn1BMGfSLojupIAAACA6lTIA41XmNknJc2TZJK63f22yCsDEJtzzjkn7hIAAKhKhcxcy91vkXRLxLUAqBB333133CUAAFCVcoZrM3tFqb18M04p6ND31siqAgAAAKpQznDt7lPKWQgAAABQ7ej6AQAAAJQI4RoAAAAoEcI1AAAAUCKEawAAAKBECNcAAADjtG7durhLQIUgXAMom5kzZ8ZdAgBEore399AXoS4QrgGUzfbt2+MuAQCASBGuAQAAgBIhXAMAAAAlQrgGAAAASoRwDaAiLVq0KO4SAAAoGuEaQEVavHhx3CUAAFC0yMK1mU0ys5+b2cNmtsPM/i7LNWZm15jZE2b2iJmdkXbuAjP7VXjuy1HVCQAAAJRKlDPXb0j6oLvPljRH0gVm9r6Maz4s6eTw1SFpjSSZWaOk68Lzp0paZGanRlgrAAAAMG6RhWsPvBoeTghfnnHZQkn/Gl77M0lHmtnxks6U9IS7P+nu+yXdFF4LAKOwPhsAUCkiXXNtZo1mtk3SHkl3u/uDGZecIOnptONd4ViucQAYhfXZAIBKEWm4dvchd58jabqkM83stIxLLNvb8oyPYmYdZrbFzLbs3bt3fAUDAAAA41CWbiHu/pKkeyRdkHFql6QT046nS3omz3i2e3e7+1x3nzt16tSS1QwAAFBt1q1bF3cJdS/KbiFTzezI8PvJks6T9B8Zl22Q9Gdh15D3SXrZ3Z+V9JCkk83sJDObKOmi8FoAAADk0NvbG3cJdS/KmevjJfWb2SMKwvLd7t5nZkvNbGl4zR2SnpT0hKRvS+qUJHcflHS5pB9KelzS99x9R4S1AkDF6emRWlulvr71am0NjgEAla0pqhu7+yOSTs8yfn3a9y7pshzvv0NB+AaAutPTI3V0SAMDktSgnTuDY0lKJOKsDACQDzs0AkAFWrkyFawPGhgIxgEAlYtwDQAV6KmnihsHAFQGwjUA5BHXBjUzZhQ3DgCoDIRrAMgjrg1qVq2SmptHjjU3B+MAgMpFuAaAIlx44YVl+ZxEQurullpaJGlYLS3BMQ8zAkBlI1wDQBGuu+66sn1WIiElk1J7+0IlkwRrZNHfrxs2bZL6++OuBECIcA0AQDXq75fa23Xsvn1SezsBG6gQhGsAKJGZM2fGXQLqRRisD/RrHBgIjlevZiYbiBnhGgBKZPv27XGXgHqQGaxTBgakFSuYyQZiRrgGAKCaLFkyOlhnSs1kE7CBsiNcAwBQTdauHd2nMRsCNhALwjUAANWkrU3q6ys8YC9ZEn1NAA4gXAMAUG0KDdjNzcFMN4CyIVwDAFCNwoC9e/Jk6eqrs2/p2dcXXFco+majSvT0SK2tUkND8LWnJ+6KDiJcAwBQrdradOn8+dLy5SNnsscYrOmbjWrQ0yN1dEg7d0ruwdeOjsoJ2IRrAABqQfpM9hiD9ai+2QRsVKCVK7N3oly5Mp56MhGuAaDCLVq0KO4SUC1SM9njCdYpBGxUqKeeKm683AjXAFDhFi9eHHcJqGX5+mbTbQQVaMaM4sbLjXANAEA9y9c3m24jqECrVmV/fnfVqnjqyUS4BgCgnuVq6zeWhyKBMkgkpO5uqaVFMgu+dncH45WAcA0AQL3LDNgEa1S4REJKJqV/+7d1SiYrJ1hLhGsAqCk8/IgxG0+3ESAmvb29cZcwCuEaAGoIDz9iXMbSbQTACIRrAABqQGrHur6+9RW3Yx1QT5riLgAAAIxPase6oKNew4Ed66TKWosK1ANmrgEAqHKVvmNdKTAzj2rBzDUAAFWu0nesGy9m5lFNmLkGAKDKVfqOdeNVDzPzqB2EawAAqlyl71g3XrU+M4/aQrgGAKDKpe9YJw1X3I5145Wagf+A+vVrteoD6h8xDtakVxLCNQAANSC1Y117+8KK27FuvFatki54S7/61K5W7VSf2nXBW/prZmZ+vFJr0nfulNLXpBOw40G4BgAAFS3x9n5939t1mIKF14dpQN/3diXe3h9zZZWBNemVhXANAAAqV3+/1N6upv0j02PT/gGpvT04X+dYk15ZCNcAAKByLVkyelo2ZWAgOF/nar1bTLUhXAMAgMq1du3oVigpzc3B+TpX691iqg3hGgAAVK62NqmvL3t67OsLzte5Wu8WU20I1wAAoLJlBmyC9Si13C2m2hCuAQBA5QsD9u7JkwnWqGiEawAAUB3a2nTp/PkEa1Q0wjUAAABQIoRrAAAAoESaorqxmZ0o6V8lHSdpWFK3u/9TxjVXSEotuW+SdIqkqe7+gpklJb0iaUjSoLvPjapWAAAAoBQiC9eSBiWtcPdfmNkUSVvN7G53fyx1gbt/Q9I3JMnMPirpL939hbR7tLn7cxHWCAAAAJRMZMtC3P1Zd/9F+P0rkh6XdEKetyyS1BtVPQAAAEDUyrLm2sxaJZ0u6cEc55slXSDplrRhl/QjM9tqZh1R1wgAAACMV+Th2swOVxCav+juv81x2Ucl3Z+xJGSeu58h6cOSLjOzc3Lcv8PMtpjZlr1795a0dgAAgEPq79cNmzZJ/f1xV4IKEGm4NrMJCoJ1j7vfmufSi5SxJMTdnwm/7pF0m6Qzs73R3bvdfa67z506dWppCgcAAChEf7/U3q5j9+2T2tsJ2IguXJuZSfoXSY+7++o81x0h6VxJ69PGDgsfgpSZHSbpQ5IejapWAACAooXBWgMDwfHAAAEbkXYLmSfpYknbzWxbOPa/JM2QJHe/Phz7uKQfuftrae89VtJtQT5Xk6R17n5XhLUCAAAULjNYp6QCNlu0163IwrW73yfJCrjuO5K+kzH2pKTZkRQGAAAwXkuWjA7WKQMDwflksqwloTKwQyMAAECx1q6Vmpuzn2tuDs6jLhGuAQAAitXWFiz9yAzYzc0sCalzhGsAAICxyAzYBGuIcA0AADB2YcDePXkywRqSCNcAAADj09amS+fPJ1hDEuEaAAAAKBnCNQAAAFAihGsAAACgRAjXAAAAQIkQrgEAAGpBf79u2LQp2JodsSFcAwAAVLv+fqm9Xcfu2ye1txOwY0S4BgAAqGZhsNbAQHA8MEDAjhHhGgAAoFplBusUAnZsCNcAAADVasmS0cE6ZWAgOI+yIlwDAABUq7Vrpebm7Oeam4PzKCvCNQAAQLVqa5P6+kYH7ObmYJwt2cuOcA0AAFDNMgM2wTpWhGsAAIBqFwbs3ZMnE6xjRrgGAACoBW1tunT+fIJ1zAjXAAAAQIkQrgEAAIASIVwDAAAAJUK4BgAAAEqEcA0AQBVbtGhR3CUASEO4BgCgii1evDjuEgCkIVwDAFCFenqk1lapoSH42tMTjDOTDcSrKe4CAABAcXp6pI4OaWAgON65MziWpESCmWwgTsxcAwBQZVauPBisUwYGgnEA8SJcAwBQZZ56qrhxAOVDuAYAoMrMmFHcOIDyIVwDAFBlVq2SmptHjjU3B+MA4kW4BgCgyiQSUne31NIimQVfu7uDcQDxolsIAABVKJGIJkzTyg8YH2auAQDAAWxKg6rR368bNm2S+vvjrmQEwjUAAACqS3+/1N6uY/ftk9rbKypgE64BAABQPcJgfaDZ+8BARQVswjUAAACqQ2awTqmggE24BgAAQHVYsmR0sE4ZGAjOx4xwDQAAgOqwdu3oJu8pzc3B+ZgRrgEAAFAd2tqkvr7suyj19QXnY0a4BgAAQPXIDNgVFKylCMO1mZ1oZv1m9riZ7TCzL2S55gNm9rKZbQtff5N27gIz+5WZPWFmX46qTgAAAFSZMGDvnjy5ooK1FO0OjYOSVrj7L8xsiqStZna3uz+Wcd1P3b09fcDMGiVdJ+l8SbskPWRmG7K8FwAAAPWorU2Xzp+v71dQsJYinLl292fd/Rfh969IelzSCQW+/UxJT7j7k+6+X9JNkhZGUykAAABQGmVZc21mrZJOl/RgltNnmdnDZnanmc0Mx06Q9HTaNbuUI5ibWYeZbTGzLXv37i1h1QAAAEBxIg/XZna4pFskfdHdf5tx+heSWtx9tqR/lnR76m1ZbuXZ7u/u3e4+193nTp06tVRlAwAAAEWLNFyb2QQFwbrH3W/NPO/uv3X3V8Pv75A0wczepmCm+sS0S6dLeibKWgEAAIDxirJbiEn6F0mPu/vqHNccF14nMzszrOd5SQ9JOtnMTjKziZIukrQhqloBAACAUoiyW8g8SRdL2m5m28Kx/yVphiS5+/WSLpS0zMwGJe2TdJG7u6RBM7tc0g8lNUr6v+6+I8JaAQAAgHGLLFy7+33KvnY6/ZprJV2b49wdku6IoDQAAFClFi1aFHcJQF7s0AgAAKrG4sWLrNtPBQAACHtJREFU4y4ByItwDQAAAJQI4RoAAAAoEcI1AAAAUCKEawAAAKBECNcAAABAiRCuAQAAgBIhXAMAAAAlQrgGAAAASoRwDQAAAJQI4RoAAAAoEcI1AAAAUCKEawAAAKBECNcAAABAiRCuAQAAgBIhXAMAAIzTokWL4i4BFYJwDQAAME6LFy+OuwRUCMI1AAAAUCKEawAAAKBECNcAAADA/9/e/YVKXpZxAP8+qJSsQpqrmLoZURchqbHohVEKYdlFVmC1W6Kg2IWBRheVF2V3EqY3UWEpGKghqCRBooEhYmkqm//WStJKXVzNQhciUJ8uzm/pqOesf/Yd54z7+cBhZt7fO+c8h4d35svMO/MbRLgGAIBBhGsAABhEuAYAgEGEawAAGES4BgCAQYRrAAAYRLgGAIBBhGsAABhEuAYAgEGEawCAt4lNmzbNu4Q9nnANAPA2sXnz5nmXsMcTrgEAYBDhGgAABhGuAQBgEOEaAAAGEa4BAGAQ4RoAAAYRrgEAYBDhGgAABhGuAQBgkJmF66o6oqpuraqtVfVgVZ23wpwvV9V9088dVXX0smOPVdX9VbWlqu6eVZ0AADDK3jP83S8k+UZ331tV+ye5p6pu6e6Hls15NMnHu/tfVXVKksuSHL/s+End/cwMawQAgGFmFq67e1uSbdP156tqa5LDkjy0bM4dy+7y+ySHz6oeAACYtbdkz3VVHZnk2CR37mLaWUl+vex2J7m5qu6pqnNmVx0AAIwxy20hSZKq2i/JdUnO7+7nVplzUpbC9UeXDZ/Q3U9W1cFJbqmqh7v7thXue06Sc5Jkw4YNw+sHAIDXa6avXFfVPlkK1ld19/WrzPlwkp8lObW7/7lzvLufnC63J7khyXEr3b+7L+vujd29cf369aP/BQAAeN1m+W0hleTyJFu7+5JV5mxIcn2S07v7z8vG100fgkxVrUtycpIHZlUrAACMMMttISckOT3J/VW1ZRq7IMmGJOnunyT5TpJ3J/nRUhbPC929MckhSW6YxvZOcnV33zTDWgEAYLfN8ttCbk9SrzHn7CRnrzD+1yRHv/oeAACwdjlDIwAADCJcAwDAIMI1AAAMIlwDAMAgwjUAAAwiXAMAwCDCNQAAC2nTpk3zLuFVhGsAABbS5s2b513CqwjXAAAwiHANAACDCNcAADCIcA0AAIMI1wAAMIhwDQAAgwjXAAAwiHANAACDCNcAADCIcA0AAIMI1wAAMIhwDQAAgwjXAAAwiHANAACDCNcAADCIcA0AAIMI1wAAMIhwDQAAg1R3z7uGYarq6SR/m2MJByV5Zo5/n92jf4tL7xab/i0uvVts+vfmvbe716904G0Vruetqu7u7o3zroM3R/8Wl94tNv1bXHq32PRvNmwLAQCAQYRrAAAYRLge67J5F8Bu0b/FpXeLTf8Wl94tNv2bAXuuAQBgEK9cAwDAIML1bqiqx6rq/qraUlV3T2MHVtUtVfWX6fKAeddJUlVXVNX2qnpg2diqvaqqb1fVI1X1p6r65HyqZqdV+ndhVT0xrb8tVfXpZcf0b42oqiOq6taq2lpVD1bVedO49bfG7aJ31t4CqKp3VtVdVfXHqX/fm8atvRmzLWQ3VNVjSTZ29zPLxr6f5NnuvqiqvpXkgO7+5rxqZElVfSzJjiQ/7+6jprEVe1VVH0pyTZLjkrwnyW+SfLC7X5xT+Xu8Vfp3YZId3X3xK+bq3xpSVYcmObS7762q/ZPck+SzSc6M9bem7aJ3X4i1t+ZVVSVZ1907qmqfJLcnOS/J52PtzZRXrsc7NcmV0/Urs/RAxJx1921Jnn3F8Gq9OjXJL7r7v939aJJHsvRgw5ys0r/V6N8a0t3buvve6frzSbYmOSzW35q3i96tRu/WkF6yY7q5z/TTsfZmTrjePZ3k5qq6p6rOmcYO6e5tydIDU5KD51Ydr2W1Xh2W5B/L5j2eXT+hMD9fq6r7pm0jO9/a1L81qqqOTHJskjtj/S2UV/QusfYWQlXtVVVbkmxPckt3W3tvAeF695zQ3R9JckqSc6e3rll8tcKY/VNrz4+TvD/JMUm2JfnBNK5/a1BV7ZfkuiTnd/dzu5q6wpj+zdEKvbP2FkR3v9jdxyQ5PMlxVXXULqbr3yDC9W7o7ieny+1JbsjS2ydPTfvUdu5X2z6/CnkNq/Xq8SRHLJt3eJIn3+LaeA3d/dT0xPFSkp/m/29f6t8aM+33vC7JVd19/TRs/S2AlXpn7S2e7v53kt8m+VSsvZkTrt+kqlo3fcAjVbUuyclJHkhyY5IzpmlnJPnlfCrkdVitVzcm+VJVvaOq3pfkA0numkN97MLOJ4fJ57K0/hL9W1OmD1VdnmRrd1+y7JD1t8at1jtrbzFU1fqqetd0fd8kn0jycKy9mdt73gUssEOS3LD02JO9k1zd3TdV1R+SXFtVZyX5e5LT5lgjk6q6JsmJSQ6qqseTfDfJRVmhV939YFVdm+ShJC8kOdenpedrlf6dWFXHZOlty8eSfDXRvzXohCSnJ7l/2vuZJBfE+lsEq/Vuk7W3EA5NcmVV7ZWlF1Ov7e5fVdXvYu3NlK/iAwCAQWwLAQCAQYRrAAAYRLgGAIBBhGsAABhEuAYAgEGEa4A9RFXteMXtM6vqh9P1C6vqiaraUlUPVNVn5lMlwGITrgHY6dLpVMmnJbmiqjxHALxBHjgBeJnu3pqlk0gcNO9aABaNMzQC7Dn2XXamvSQ5MEunPH6Zqjo+yUtJnn6rCgN4uxCuAfYc/5m2fSRZ2nOdZOOy41+vqq8keT7JF9spfAHeMOEagJ0u7e6L510EwCKz5xoAAAYRrgEAYJCypQ4AAMbwyjUAAAwiXAMAwCDCNQAADCJcAwDAIMI1AAAMIlwDAMAgwjUAAAwiXAMAwCD/A6JoEEH4Uz/GAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting results\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12, 8))\n",
    "fig = sm.graphics.plot_fit(results, 1, ax=ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.7**.  Get the passenger car mileage data from \n",
    "\n",
    "http://lib.stat.cmu.edu/DASL/Datafiles/carmpgdat.html\n",
    "\n",
    "**(a)** Fit a multiple linear regression model to predict MPG (miles per gallon) from HP (horsepower).  Summarize your analysis including a plot of the data with the fitted line.\n",
    "\n",
    "**(b)** Use Mallow's $C_p$ to select a best sub-model.  To search through all models try (i) all possible models, (ii) forward stepwise, (iii) backward stepwise.  Summarize your findings.\n",
    "\n",
    "**(c)** Repeat (b) but use BIC.  Compare the results.\n",
    "\n",
    "**(d)** Now use Lasso and compare the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "The exercise wording is unclear -- if we specify that HP is the only covariate, the multiple linear regression and the simple linear regression are the same, and (a) is like the previous exercise.\n",
    "\n",
    "We will elect to use VOL, HP, SP and WT as the covariates, and MPG as the response variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(a)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use full model\n",
    "\n",
    "features = ['HP', 'SP', 'WT', 'VOL']\n",
    "\n",
    "Y = data['MPG']\n",
    "X = data[features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P &gt; |t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>const</th>\n",
       "      <td>192.437753</td>\n",
       "      <td>23.531613</td>\n",
       "      <td>8.177839</td>\n",
       "      <td>3.351097e-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HP</th>\n",
       "      <td>0.392212</td>\n",
       "      <td>0.081412</td>\n",
       "      <td>4.817602</td>\n",
       "      <td>6.671547e-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SP</th>\n",
       "      <td>-1.294818</td>\n",
       "      <td>0.244773</td>\n",
       "      <td>-5.289864</td>\n",
       "      <td>1.019119e-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WT</th>\n",
       "      <td>-1.859804</td>\n",
       "      <td>0.213363</td>\n",
       "      <td>-8.716617</td>\n",
       "      <td>2.888800e-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VOL</th>\n",
       "      <td>-0.015645</td>\n",
       "      <td>0.022825</td>\n",
       "      <td>-0.685425</td>\n",
       "      <td>4.950327e-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             coef    std err         t       P > |t|\n",
       "const  192.437753  23.531613  8.177839  3.351097e-12\n",
       "HP       0.392212   0.081412  4.817602  6.671547e-06\n",
       "SP      -1.294818   0.244773 -5.289864  1.019119e-06\n",
       "WT      -1.859804   0.213363 -8.716617  2.888800e-13\n",
       "VOL     -0.015645   0.022825 -0.685425  4.950327e-01"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using manually coded solution\n",
    "get_regression(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            OLS Regression Results                            \n",
      "==============================================================================\n",
      "Dep. Variable:                    MPG   R-squared:                       0.873\n",
      "Model:                            OLS   Adj. R-squared:                  0.867\n",
      "Method:                 Least Squares   F-statistic:                     132.7\n",
      "Date:                Sun, 15 Mar 2020   Prob (F-statistic):           9.98e-34\n",
      "Time:                        17:06:27   Log-Likelihood:                -220.00\n",
      "No. Observations:                  82   AIC:                             450.0\n",
      "Df Residuals:                      77   BIC:                             462.0\n",
      "Df Model:                           4                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "==============================================================================\n",
      "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
      "------------------------------------------------------------------------------\n",
      "const        192.4378     23.532      8.178      0.000     145.580     239.295\n",
      "HP             0.3922      0.081      4.818      0.000       0.230       0.554\n",
      "SP            -1.2948      0.245     -5.290      0.000      -1.782      -0.807\n",
      "WT            -1.8598      0.213     -8.717      0.000      -2.285      -1.435\n",
      "VOL           -0.0156      0.023     -0.685      0.495      -0.061       0.030\n",
      "==============================================================================\n",
      "Omnibus:                       14.205   Durbin-Watson:                   1.148\n",
      "Prob(Omnibus):                  0.001   Jarque-Bera (JB):               18.605\n",
      "Skew:                           0.784   Prob(JB):                     9.12e-05\n",
      "Kurtosis:                       4.729   Cond. No.                     1.16e+04\n",
      "==============================================================================\n",
      "\n",
      "Warnings:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
      "[2] The condition number is large, 1.16e+04. This might indicate that there are\n",
      "strong multicollinearity or other numerical problems.\n"
     ]
    }
   ],
   "source": [
    "# Using statsmodels\n",
    "results = sm.OLS(Y, sm.add_constant(X)).fit()\n",
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs0AAAHwCAYAAABdQ1JvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdf5hcdX3//dc7u8luNlL5kR8GYnYQfxQ1EDWiaL+QaUIhdC5i+6XFzdTiKt+tEBWvRBDd67ba772tP0gqVIKN1CXfslltYy10NFjIPdHb0tsaNBUweKE4GwOYhCAKTLLJbj73H+dMdnd2ZmdmZ86cMzvPx3XtNXs+Z+bMe3eT8NoPn/P+mHNOAAAAAIqbFXYBAAAAQNQRmgEAAIASCM0AAABACYRmAAAAoARCMwAAAFACoRkAAAAogdAMAFMws/9hZj8Nu46ZwMyWmtmLZtYSdi0AUClCMwBIMrOMma3OH3fO/b/OudeFUVM+M/uUmZ3wg+fzZvaQmV0cdl3lcs7td869zDk3GnYtAFApQjMARJCZtRY59TXn3MskzZeUlvTPdX5/AGhKhGYAmIKZrTSzA+OOM2b2UTP7sZn9xsy+Zmbt484nzGzvuJngC8adu8XMfm5mL5jZT8zsj8ade6+Z/YeZ/a2ZPSfpU1PV5ZwbkTQg6RwzW1Dm+7/ZzH7kv/8/+7X/3+O/TjP7mJn9SlJ/Gdf7mJk95V/vp2a2yh+/yMz2mNlvzeygmW32x2Nm5nKB3MzONrP7zOw5M/uZmf2vcdf+lJn9k5n9H//6j5nZirJ/cABQY4RmAKjcn0q6QtK5ki6Q9F7JC6WSviLpLySdJenvJd1nZm3+634u6X9IermkT0u6x8wWj7vu2yQ9KWmhpL6pCjCzOZL+XNIRSb8u9f7+878h6W5JZ0oalPRHeZd9hX+uU1JPieu9TtIHJb3VOXeapMslZfzr3CbpNufc70g6T9I/FfkyBiUdkHS2pKsl/XUuePuukvRVSadLuk/SF6f6ngBAkAjNAFC5251zTzvnnpP0b5KW++P/S9LfO+e+75wbdc5tkzQs6e2S5Jz7Z/91J51zX5P0hKSLxl33aefc3znnRpxzR4u895+a2fOSjvrvd7U/61zq/d8uqdWv/YRz7l8k/VfetU9K+kvn3LD//lNdb1RSm6TXm9ls51zGOfdz/zonJL3azOY75150zv1/+V+Emb1S0u9J+phz7phzbq+kuyS9Z9zTvuec+5a/BvofJV1Y5HsCAIEjNANA5X417vOspJf5n3dK2ugvZXjeD7evlDeTKjP783FLHZ6X9EZ5a5NzflnGe/+Tc+50SYskPSrpLePOTfX+Z0t6yjnnpni/w865Y+Vczzn3M0kfkbeM5JCZfdXMzvZf935Jr5X0uJn9wMwSBb6OsyU955x7YdzYkKRzxh3nf5/bWWsNICyEZgConV9K6nPOnT7uo8M5N2hmnZK+LG9Jw1l+8H1Uko17vStwzYKcc8/KWzbxqXFLPIq+v6Rn5K1/Hv9+r8y/bLlfj1/Ddufc78kL107SZ/3xJ5xzXfKWmXxW0g4zm5d37aclnWlmp40bWyrpqXK/BwBQT4RmABgz28zax31UOqv5ZUkfMLO3mWeemf2hHwznyQuWhyXJzLrlzTRPm3PucUnflnRzGe//n/KWVHzQzFrNbK0mLg2p6Osxs9eZ2e/767WPyVsuMup/bX9mZguccyclPe9fa0KbOefcLyU9JOlv/O/1BfJmqAeq+Z4AQFAIzQAw5lvywl/u41OVvNg5t0feOuAvyrs572fybxJ0zv1E0iZ54fWgpGWS/qMGNX9e3k17C0u8/3FJfywvmD4v6c8kpeStUa7465G3nvkzkp6Vt4xioaRP+OeukPSYmb0o76bAd+ct+8jpkhSTN+v8DXnrqR+o8OsHgLqwicvbAADNwsy+L+lLzrn+sGsBgKhjphkAmoSZXWpmr/CXZ1wrr13e/WHXBQCNgLuQAaB5vE5ez+SXyesZfbVz7plwSwKAxsDyDAAAAKAElmcAAAAAJRCaAQAAgBIaYk3z/PnzXSwWC7sMAAAAzHAPP/zws865BfnjDRGaY7GY9uzZE3YZAAAAmOHMbKjQOMszAAAAgBIIzQAAAEAJhGYAAACghIZY0wwAAIBgnDhxQgcOHNCxY8fCLqWu2tvbtWTJEs2ePbus5xOaAQAAmtiBAwd02mmnKRaLyczCLqcunHM6cuSIDhw4oHPPPbes17A8AwAAoIkdO3ZMZ511VtMEZkkyM5111lkVza4TmgEAAJpcMwXmnEq/ZkIzAAAAQmVmes973nPqeGRkRAsWLFAikZAk3X333VqwYIGWL1+u17/+9fryl7986rn333+/LrroIv3u7/6uli9frmuuuUb79++veY2EZgAAAJRtYECKxaRZs7zHgYHqrzlv3jw9+uijOnr0qCTpgQce0DnnnDPhOddcc4327t2r3bt36xOf+IQOHjyoRx99VB/60Ie0bds2Pf7449q7d6+SyaQymUz1ReUhNAMAAKAsAwNST480NCQ55z329NQmOK9Zs0bf/OY3JUmDg4Pq6uoq+LyFCxfqvPPO09DQkD772c/qE5/4hM4///xT56+66ipdcskl1ReUh9AMAACAsvT2StnsxLFs1huv1rvf/W599atf1bFjx/TjH/9Yb3vb2wo+78knn9STTz6pV7/61Xrsscf05je/ufo3LwOhGQAAAGUptlS4FkuIL7jgAmUyGQ0ODurKK6+cdP5rX/uali9frq6uLv393/+9zjzzzAnnjxw5ouXLl+u1r32tbr311uoLykOfZgAAAJRl6VJvSUah8Vq46qqr9NGPflS7d+/WkSNHJpy75ppr9MUvfnHC2Bve8Ab98Ic/1IUXXqizzjpLe/fu1a233qoXX3yxNgWNw0wzAAAAytLXJ3V0TBzr6PDGa+F973ufPvnJT2rZsmVlPf/mm29WX1+f9u3bd2osm79+pEaYaQYAAEBZkknvsbfXW5KxdKkXmHPj1VqyZIluvPHGsp+/bNky3XbbbfrzP/9zvfDCCzrrrLO0dOlSffrTn65NQeOYc67mF621FStWuD179tTt/QYGgvvDAAAAECX79u2b0H2imRT62s3sYefcivznMtOcJ9dKJTezn2ulIhGcAQAAmhVrmvME2UoFAAAAjYnQnCfIVioAAABoTITmPMVaptSqlQoAAAAaD6E5T9CtVAAAANB4CM15kklp61aps1My8x63buUmQAAAgGZGaC4gmZQyGenkSe+RwAwAADBOOi3FYt5jDdx+++06//zzdcYZZ+gzn/mMJOlf//Vf9ZOf/OTUc+6++249/fTTFV03k8nojW98Y01qJDQDAACgfOm0lEh4fXkTiZoE5y1btuhb3/qWfv3rX+uWW26RVJvQXEuEZgAAAJQnF5hz/Xmz2aqD8wc+8AE9+eSTuuqqq/S3f/u3+uAHP6iHHnpI9913n2666SYtX75cn/3sZ7Vnzx4lk0ktX75cR48e1cMPP6xLL71Ub3nLW3T55ZfrmWeekSQ9/PDDuvDCC3XxxRfrjjvuqMVXLYnQDAAAgHLkB+acKoPzl770JZ199tlKp9M644wzJEnveMc7dNVVV+nzn/+89u7dq4997GNasWKFBgYGtHfvXrW2tupDH/qQduzYoYcffljve9/71OtvqtHd3a3bb79d//mf/1nVl5uPHQEBAABQWnf35MCck8165zOZupTy05/+VI8++qguu+wySdLo6KgWL16s3/zmN3r++ed16aWXSpLe8573aOfOnTV5T0IzAAAASuvvLzzTLHn9efv761aKc05veMMbJs0mP//88zKzQN6T5RkAAAAoLR6XUqnCG1qkUt75GjrttNP0wgsvFDx+3etep8OHD58KzSdOnNBjjz2m008/XS9/+cv1ve99T5I0MDBQs3oIzQAAAChPfnAOKDBL0rvf/W59/vOf15ve9Cb9/Oc/13vf+1594AMf0PLlyzU6OqodO3boYx/7mC688EItX75cDz30kCSpv79f69ev18UXX6y5c+fWrB5zztXsYkFZsWKF27NnT9hlAAAAzDj79u3T+eefX9mL0mlvDXN/fyCBuV4Kfe1m9rBzbkX+c1nTDAAAgMrE43W76S8qWJ4BAAAAlEBoBgAAAEogNAMAADS5RrjHrdYq/ZoJzQAAAE2svb1dR44caarg7JzTkSNH1N7eXvZruBEQAACgiS1ZskQHDhzQ4cOHwy6lrtrb27VkyZKyn09oBgAAaGKzZ8/WueeeG3YZkcfyDAAAAKAEQjMAAABQAqEZAAAAKIHQDAAAAJRAaAYAAABKIDQDAAAAJRCaAQAAgBIIzQAAAEAJhOYpbN++PewSAAAAEAGE5ikMDg6GXQIAAAAigNAMAAAAlBBoaDaz081sh5k9bmb7zOxiMzvTzB4wsyf8xzOCrAEAAACoVtAzzbdJut8597uSLpS0T9ItknY5514jaZd/DAAAAERWYKHZzH5H0iWS/kGSnHPHnXPPS1oraZv/tG2S3hVUDQAAAEAtBDnT/CpJhyX1m9mPzOwuM5snaZFz7hlJ8h8XBlgDAAAAULUgQ3OrpDdLutM59yZJL6mCpRhm1mNme8xsz+HDh4OqEQAAACgpyNB8QNIB59z3/eMd8kL0QTNbLEn+46FCL3bObXXOrXDOrViwYEGAZQIAAABTCyw0O+d+JemXZvY6f2iVpJ9Iuk/Stf7YtZLuDaqGRjIwIMVi0qxZ3uPAQNgVAQAAIKc14Ot/SNKAmc2R9KSkbnlB/Z/M7P2S9kv6k4BriLyBAamnR8pmveOhIe9YkpLJ8OoCAACAJ9DQ7JzbK2lFgVOrgnzfRtPbOxaYc7JZb5zQDAAAED52BIyA/fsrGwcAAEB9EZojYOnSysYBAABQX4TmCOjrkzo6Jo51dHjjAAAACB+hOQKSSWnrVqmzUzLzHrduZT0zAABAVATdPQNlSiYJyQAAAFHFTDMAAABQAqEZAAAAKIHQDAAAAJRAaAYAAABKIDRXafv27WGXAAAAgIARmqs0ODgYdgkAAAAIGKEZAAAAKIHQHIDLLrss7BIAAABQQ4TmAHz3u98NuwQAAADUEKEZAAAAKIHQDAAAAJRAaAYAAABKIDQDAAAAJRCaa2hgQIrFpOPHjyoW844BAADQ+FrDLmCmGBiQurulEyckaZaGhrxjSUomw6wMAAAA1WKmuUZuvDEXmMecOOGNAwAAoLERmmvkyJHKxgEAANA4CM0AAABACYTmGjnrrMrGAQAA0DgIzTVy223SnDkTx+bM8cYBAADQ2AjNNZJMSl/5itTZKUkn1dnpHdM5AwAAoPHRcq6Gkknvo61trjKZ4bDLAQAAQI0w0wwAAACUQGiOmPXr14ddAgAAAPIQmiNmx44dYZcAAACAPIRmAAAAoARCMwAAAFACoRkAAAAogdAMAAAAlEBoLiad1l27dknpdNiVAAAAIGSE5kLSaSmR0KKjR6VEguAMAADQ5AjN+fzArGzWO85m6xKcBwakWEw6dOgZxWLeMQAAAKKB0DxefmDOCTg4DwxIPT3S0JAkzdLQkHdMcAYAAIgGQvN43d2TA3NONuudD0Bvb+Gc3tsbyNsBAACgQoTm8fr7pY6Owuc6OrzzAdi/v7JxAAAA1Behebx4XEqlJgfnjg5vPB4P5G2XLq1sfPv27YHUAQAAgMIIzfnyg3PAgVmS+voK5/S+vsLPHxwcDKwWAAAATEZoLsQPzgfnzg08MEtSMilt3Sp1dkrSSXV2esfJ5PSvuX79+lqVBwAA0PQIzcXE47pu1arAA3NOMillMtLChYuVyVQXmCVpx44dtSgLAAAAIjQDAAAAJRGaAQAAgBIIzQAAAEAJhGYAAACgBEJzE7rsssvCLgEAAKChEJqb0He/+92wSwAAAGgohGYAAACgBEIzAAAAUAKheZq2b98edgkAAACoE0LzNA0ODoZdAgAAAOqE0FyNdFp37dolpdNhVwIAAIAAEZqnadmzz0qJhBYdPSolEgRnAACAGSzQ0GxmGTN7xMz2mtkef+xMM3vAzJ7wH88IsoZApNP65A9+IGWz3nE2S3AGAACYweox0xx3zi13zq3wj2+RtMs59xpJu/zjxpFOS4mE2kdHJ44TnAEAAGasMJZnrJW0zf98m6R3hVDD9HV3j80w58tmpauu0rHjx6UbbqhvXQAAAAhM0KHZSfp3M3vYzHr8sUXOuWckyX9cGHANtdXfL3V0FD7X0iK9+KJMku68k+AMAAAwQwQdmt/pnHuzpDWS1pvZJeW+0Mx6zGyPme05fPhwcBVWKh6XUikda2mZON7SIuUv2Yh4cKbXNAAAQHkCDc3Ouaf9x0OSviHpIkkHzWyxJPmPh4q8dqtzboVzbsWCBQuCLLNy8bj+6q1vHZtxLhSYcyIcnMvpNb1+/fo6VAIAABBtgYVmM5tnZqflPpf0B5IelXSfpGv9p10r6d6gagjSI/PnS6mUDs6dWzww59x5Z32KCsCOHTvCLgEAACB0Qc40L5L0PTP7b0n/Jembzrn7JX1G0mVm9oSky/zjxhSP67pVq6Trr5/6eaXOAwAAINICC83OuSedcxf6H29wzvX540ecc6ucc6/xH58Lqoa62bKleDC+/nrvfJ0MDEixmHTo0DOKxbzjWmMtNAAAaDbsCFgrhYJzCIG5p0caGpKkWRoa8o5rHZzLWQsNAAAwkxCaa8kPzk6qe2CWpN7eyS2ks1lvHAAAANNHaK61LVvUPmdO3QOzJO3fX9k4AAAAykNonkGWLq1sHAAAAOUhNM8gfX2TNyvs6PDGAQAAMH2E5hkkmZS2bpU6OyXppDo7veNkMuzKAAAAGltr2AWgtpJJ72PRosXKZA6GXQ4AAMCMwEwzAAAAUAKhGQAAACiB0BwlmzfrqUOHpM2bw64EAAAA4xCao2LzZmnjRm+R+caNBGcAAIAIITRHgR+YJ2iA4Lx9+/awSwAAAKgLQnOFBgakWExKpe5VLCYdOHBpdRcsFJhzIh6cBwcHwy4BAACgLgjNFRgYkHp6pKEhSZqloSHpkUc+qIGBKi56883VnZ/B1q9fH3YJAAAAkgjNFentlbLZiWOjo+3q7a3iop/7XHXnZ7AdO3aEXQIAAIAkQnNF9u8vPD40pOnPNm/YIG3aVPjcpk3eeQAAAISK0FyBpUuLn+vpqXFwboDA/NRTT4VdAgAAQF0QmivQ1yd1dBQ+l82qumUafnAekaoKzLkbFQ8dekaxWBVBvgzFQjNdNQAAwExDaK5AMilt3Vr8fLHlG2XbsEHnLFxYVWDOv1Gxqhnwabr11ltrcp1jx47V5DoAAADVIjRXKJmUOjsLn5tq+UY9FLpRseoZ8Gmo1bKNbP4XAwAAEBJC8zT09UktLRNnQTs6vPHApdO6a9cuKZ2edKrYTHfVM+AhGR0dDbsEAAAASYTmaUkmpWXLvujPOJ9UZ6e3bCOZDPiN02kpkdCio0elRGJScM7NdK9UWr9QTCuVnjAOAACA6SE0T9OSJd9RJiMlEmuVyXiBOXcT3vHjR2t/E54fmE+tv8hmJwXnvj7pira0dmqNYhrSTq3RFW3p+syA1wCbmQAAgKgiNE+hq6ur7OcGehNefmDOyQvOybPTSo2uUbuGJUntGlZqdI2SZ09eyhFFbGYCAACiitA8hXXr1pX93EBvwuvunnzx8W/S3e0F5zVr1DIyPOF0y8iwtGZNwTXQAAAAKA+huUYCvQmvv794g+iODu98V5c0PFz4OcPD3vkG45wreo5e0AAAoJ4IzTVS7Ga7mtyEF49LqdTk4NzR4Y3H4zV4k/BV0pe5Vr2gAQAAykForpFCuwXWtA1dfnDOD8yDg1JbW+HXtrV55yXphht07Phx6YYbalRY7VTSl5ktvAEAQD0Rmmskt1tgoG3o/OB8cO7cyTPM8bi0c+fk4NzW5o3H415QvvNOmSTdeac+8OMf17C46tGXGQAARBWhuYaSSSmTkebMmXuqDV3NxeO6btWqwksy8oNzgcA83pX790dyxhkAACBqCM0zjR+cfzlr1pSBWdKpGWeCMwAAwNQIzTNRPK4V8+ePzUYXCMwTlDoPAADQ5AjNzeD666s7DwAA0OQIzdORTuuuXbsaZ8OQLVsKBmMneeNbttS9JAAAgEZCaK6Uv6X1oqNHpURCy559NuyKylMgOH9r6VICMwAAQBkIzZXwA/OpLa2zWX3yBz9ouBnn3Azzly64oKKXDwxIsZiUSt2rWEw6evSPAigSAAAgegjN5coPzL720VFvvIGCc/ucORXPMA8MSD090tCQJM3S0JD0wgubNTAQSJUAAACRQmguV3f3pMB8SjbrnZ/BensLffkd6u0NoxoAAID6IjSXq79/8j7ZOR0d3vkZbP/+ysajbvv27WGXAAAAGgihuVz+Ftb5wflYS8vkLa1noKVLKxuPultvvTXsEgAAQAMhNFciPzh3dOiv3vrWxgzM5bTN27xZTx06JG3erL6+QhPtWfX1BVlkcJ566qmwSwAAAA2E0FwpPzgfnDtXSqX0yPz5YVdUsUtPnpzQNq9gcN68Wdq4Ua2StHGjkgc3a+tWqbNTkk6qs1M67bQNSiaDr5elFAAAIGyE5umIx3XdqlUNO8P8jZGRCW3zJgVnPzBP4AfnTEZKJNYqk5Hmzv1GXUoeHBysy/sAAAAUQ2iuUldXV9gllM9vmzcvf3x8cC4UmHM2bvTO11iu/7NzI4rFVFYbu2PHjtW8DgAAgGJawy6g0a1bty7sEspXTtu8AwemvsZNN+mutraa9aXO9X/2yvL6P/f0lH4doRkAANQTM83NpJy2eZ/73NTXaGk5tRb6ncePV11Sof7P2axK9n8+efJk1e8tsV4aAACUh9AcgEsuuSTsEgrzb2J8KX+8o2Osbd6GDdKmTYVfP3u2dOKE93k2q3uef77qGefp9n8eGRkpeq6SIMx6aQAAUA5CcwAeeOCBsEsoLh7XH7W2TmibN6nPdKHgPD4w+zqkqrcQD6L/M0EYAADUGqG5CX1n1qwJbfMKdgHxg/OIJJ1xxqTAfEqVW4gX6v/c0aGa9n9mCQYAAKgWoblZldM2b8MGnbNwofT1rwe2hXgyqUn9n7duVU37PzPzDAAAqkVobiK51m7Hjx9VLCYdOHBpeS8ssoV4VqrJFuLJpJTJSGatymRqG5ilwrv/vfDCC7V9EwAAMKMRmptErrXb0JCUa+32yCMfLKsnsqSCW4j/2emnB7LBy+rVkpkknZSZ9NBDn67qeoVCMy3rAABAJQjNTaJQa7fR0faSrd0myNtC/D/mzKlpjZIXmHftyh2ZJOm5596k1atr+z7OudpeEAAAzGiE5iZRrIXb0FB5O/CdEvAW4mOBeTwrMg4AAFAfhOYmMVULt56eCoMzAABAkyE0N4m+Pq/VciHl7MA3STqtPc8+W7PttAEAAKKM0NxEvJvrCiu1A994y559Vkok9MqTJ6ve3CTfqlWFRl2RcQAAgPoIPDSbWYuZ/cjMUv7xmWb2gJk94T+eEXQN8GaSjx8vfr7sHfjSaX3yBz8Yu6swm61pcH7wwfHB2btZ78wzf6QHH/RGcm3zpFHFYiwrAQAA9VGPmeYbJe0bd3yLpF3OuddI2uUfI2BTzSSXvQNfOi0lEmofHZ04HkBw9ppbzJJz0jve8ZeSCrfNYz02AACoh0BDs5ktkfSHku4aN7xW0jb/822S3hVkDfAUm0luaalgB77u7sl963Kq3E57ks2bdcJ/zCnUNm9a67EBAAAqFPRM8xck3Szp5LixRc65ZyTJf1xY6IVm1mNme8xsz+HDhwMuszHkliakUvdWvDShr2/yTtgtLce0bVsFO/D19we2nfYEmzdLGzeqVZI2btTan/9cUvHZ8krWYwMAAExHYKHZzBKSDjnnHp7O651zW51zK5xzKxYsWFDj6hpPtUsTkklvRrmzU5JOqrNTWrbsi5VtWe1vbnJ8Vt4fm7a2se20q+2q4Qfm8d6/b5+0eXPR2fKy12MDAABMU5Azze+UdJWZZSR9VdLvm9k9kg6a2WJJ8h8PBVjDjBGlpQnOTfxjc2qJczotrVnjddVYs6bi4PwR5yYFZsnfF3DjRn39nZsnTXSXvR4bAACgCoGFZufcx51zS5xzMUnvlvT/OOf+TNJ9kq71n3atpHuDqqERXX311QXHq12aUGim+pFHPljZTXTptEauSKjNjUwYbhkZ1ugfrJEuv1waHvYGh4crDs6fK3H+LV+7edJsednrsQEAAKoQRp/mz0i6zMyekHSZfwzfHXfcUXC82qUJhWaqR0fbi85U59ZPHzr0zNj66e5utR4vfCNgy8iwdOLExMEKg/PNpZ7wuc8pmZQyGUlqUSZDYAYAAPVRl9DsnNvtnEv4nx9xzq1yzr3Gf3yuHjU0ukI38lWyNKGSmepi66cfTPbrJRW5EbCY4WGpq6usp37BTNq0ye/OPMZJ0qZN0oYN9GkGAAChYEfABlHoRr5KliZUMlNdbP30dQNxXbcoNSk4n1QNbN6s485p/z3fKXj64Yfp0wwAAMJDaG4guaUJicTaipcmFGs5V2imeqpZ6cSmuNbO+pdTwfkldejjszdptLWt8Iva2qTBwamLG9di7pU/uk/5u32bpDdv36gnP7Q5MjdDAgCA5kJobhKVtJybalY6mZSOXPCIrluUUkZLdd2ilC7o36CWf9/pBeTx2tqknTu9VnTF5LWYyw/M48c//uvCq57p0wwAAIJGaG4iuZnqOXPmKpORliwpvBSir0+aPXvi2OzZY+unlyz5jgZ/FdfbFh7T4K/iXvCOx72AnAvO0wjMU3GS/uYMr7/GSqX1C8W0Ut4NhvRpBgAAQSM0oyCzqY8L8oPzL2fNKh2YJenmkv0yJHmBeYM26VV/t0FXtKW1U2sU05B2ao1Wzfo2fZoBAEDgCM2YpLdXOn584tjx42WuHY7HtWL+/NKBWZI+V6oz80TJs9NKja5Ru7xe0O0a1jddQsmzp7n7YBNav3592CUAANCQCM0NqKvMFm7TVe1GKmV705smrwMpwCRt1kbpD/7A6wc9Tpsbmdbug81qx44dYZcAAEBDIjQ3oJqO5t4AACAASURBVHXr1gV6/Wo3Uilbd/fkDVGKMEkaGSl8soJe0AAAANNBaJ6him3HXY5qN1Ip24c/XOMLFnbs2LG6vA8AAJi5CM0zVLHtuMtR7UYqZbv99tpcp0QvaEIzAACoFqEZBeXa0y1cuLjijVTK1t8/eUq7mI4ObyvtvF7Qx8vt1AEAAFAFQjPCE49LqVTJ4HyspcV73oYNk3pBf+qiiwjMAAAgcIRmhCs/OOdmlMcd/9Vb3zoWjP1e0BlJ2rlTj8yfH0LRAACg2RCam9All1wSdgkT+cE5I43NKI87nhSM43Gdm3tdA1m2bFnYJQAAgGkiNDehBx54IOwSJovH9SqzCTPKjRiMp/LYY4+FXQIAAJgmQjOaVJdiMSmVulexmDQwEPw7OueCfxMAABAIQjOazoO9af1C39O5Q2lJszQ0JPX01Cc4AwCAxkRoRs3ccIPU2iodOvQrtbZ6x5GTTusdf5NQTL9USgmtlLf9djYr9faGXBsAAIisKUOzmS0xs98bd7zBzD7pf7w6+PLQKG64QbrzTml0VJJMo6PecaSCczotJRLqcFlJ0jxlJwTn/fvDLA4AAERZqZnmz0s6fdzxX0h6SZKT9OmgikLj2bq1svGypdO6a9cuL/BO08mTJ08FZmWzE86ND85Ll1ZZKwAAmLFKhebXOedS446zzrlNzrn/LYmIgVO8Gebyx8uxUpISCS06etQLvNMMzqOjo1J396TAnDNPWd1t3errm3apAABghisVmtvzjleN+/ysGtfSULq6usIuIVJaWiobLymdVkoaC7rZ7FhwTqf1C/85Zfvwh4uecpKe7fpwMFuFAwCAGaFUaH7BzF6bO3DOPSdJZva7kl4MsrCoW7duXdglREpPT2XjU/KXUszLH89mpTVrpDVrFJOkRELLnn22vGvefnvRUybpLf9R/DwAAECp0PyXklJmdq2ZLfM/3ivpPv8cmtCBA5cqFpMOHXrmVI/jLVuk66/PzSw7tbR4x1u2lL7ewIAUi0nOjSgWk178k+JLKTQ87H1IUjarT/7gBxNnnIutge7vV1YdBS95TG1Sf3/pQgEAQNOaMjQ75+6X9MfylmXc7X/EJf2xc25n0MUhegYGpEce+aCGhqT8HsdbtkgjI9LCha/QyEj5gbm7WxOu90fP92tkTuGAm699dHTCso2ia6Djcf2hUnqpSHAGAACYSjl9mg9Kul3SKufcW5xz1zrnHg24LkRUb680OjpxqXs1PY5vvFE6cWLi2IOjcV3dntJL5V4km5W6uiZ2xxi/Btq3W3EllNJRtU14ebuGq7rREAAAzHyl+jRfJ+kxSX8n6XEzu6ouVaEuyrmZ8eqrr55wXKyX8XR7HB85Unj83t/GlZCkDn9mePbsqS/03HOTl3T4a6Cfdk5Kp3WWf+uqFXp9gZANAACQU2qm+SOS3uCcu1jSOyR9PPiSUC/l3Mx4xx13TDgu1ss4iB7HuyUpldLBuXOll71s6ifnT1fnDA/rFZK0Zo2++hdp3a1ub2a5kGzWWysCAACQp1RoPu6cOyxJzrknpbz/r42m09cntbQcmzDW0aFp9zg+q0jjwlPj8biuW7VKmjNnem+QMzys1ZvW6Nl1H9ZxFZm17uiY8TcEHjt2rPSTAADAJKVC8xIzuz33UeAYTSaZlJYt+6I6OyXppDo7vV3/ptvj+LbbJufhOXO88QkGB6W2Ir+ztbVJmzaNLeUoZnhYb/nazZqjArPSHR1SKiXF42XX3ogIzQAATE+p0HyTpIfHfeQfownddNPZymSkhQsXK5OZfmCWvNd+5SuaEMK/8pUC14zHpZ07JwXn47NmeeMbNniht1RwLrRF4ezZTRGYo2L79u1hlwAAQMVKtZzbNtVHvYpEtNR6Y5dkUspkJLPWqUN4fnBua9OnLrpoLOzG41IqpYp37j5xQvrRj6ZVOyo3ODgYdgkAAFSsVPeM+6b6qFeRwCl+cM5I0s6demT+/Enn/6+3v734Uo5ibrqpRgVOltu8RRo9tRkMAABoLK0lzl8s6ZeSBiV9X0W6dQF1FY/rXEkuHpc2b550+pH5870Z6TVrxnYPLOXlL69tjb6BAW/zF68b3thmMFJ1y1oAAEB9lVrT/ApJn5D0Rkm3SbpM0rPOue84574TdHHAtOUv5Zg9W2ot8jvi7NnS178eSBm9vYXbR093MxjUDmurAQCVKLWmedQ5d79z7lpJb5f0M0m7zexDdakOqMb4pRzf/rb07/8+aZOUE2beuYBuAqz1ZjCoHdZWAwAqUXIbbTNrM7M/lnSPpPXyttT+l6ALA2oiHterzLxQHI/rwZu+fapP83HN1g2v6gm0a0Y9N4MBAADBKXUj4DZJD0l6s6RPO+fe6pz73865p+pSHVBDAwPS2i/Edbm+rYw6dbm+rf7MFwK9Ma+vb3IXvGo2gwEAAOEoNdP8HkmvlXSjpIfM7Lf+xwtm9tvgywNqJ7e+eLfiOlcZ7VZco6Ptga4vTia9zV9qtRkMAAAIx5TdM5xzJZdvAI1iaKiycQAAgJxSLeeAos4555ywS6hIS0vhDQFbWoJ7T1rOAQAwMzCTjGlrtNBcKDBPNV4LUWk5l9tg5fjxo2ywAgDANBCa0VBWr5bMJOmkzKSHHvp02a/11hWXP14LUWg5l5vt9pahjM12E5wBACgfoRkNY/Vqadeu3JG3OeVzz71Jq1eX9/orryw06oqM10YUWs5FZbYbAIBGRmhGwxgLzONZkfHJvvWtwq8vPF4bUWg5F4XZbgAAGh2hGU0jjPA43ZZztdziOQqz3QAANDpCM6atq6sr1PdfqbR+oZhWKl3W88MKj8mklMlIUosymfK6ZtRyi+cozHYDANDoCM2YtnXr1hUcr11XjS7FYlIqda9iMen1rx87s1JppZRQTEPaOSshpdOnOkTknr96tdTaKjk3qtZW6dWvnhweW1qOzfjwyAYr1avlzD8AoDERmlFzH/3oR6u+htfZ4csTOj5kMl5wzgXmefLubms/mdXIFQnd8/70hOfv2pVrJ2caHfWOL754YnhctuyLgYfHG27wwrt0Uq2t3nG95Wa758yZW/ZsN8bUcuYfANCYCM2ouWIz0JXwOjvMmzCWzUoXHEkr3TEWmHNaj2e1YzhRcqnG7t0Tl0osWfKdqmudyg03SHfeOTG833ln/YPzwIDU9Yq0fnr8Fep6RZp2cw2I2W4ACBehGZFU7Oa8vznYPbl/mm+esupX95TXDXIjk0K2bq1sPAgDA9I970+r/+AaxbRf/QfX6J73E5wbDbPdABAuQjMiqdjNeR9f1D95YbLvJXWoW/1TXjfILbMLCWMXwnypjWl9Y3iN2jUsSWrXsL4xvEapjeXdQAkAAAjNiJD29vZTn3s357004XxHh5TYFJdSqbwz0sicDl3dltJuxad8j56e2tRarmIhvW7hPe3NMOcCc067htV/cI2UJjgDAFAOQjMi47TTTjv1eTIptbRcX7jjQzyuhDQ249zRodb7U/qzf4hPeP6qVdKqWV5bulWz0rr+emnLlvp+TcVCet3Ce1fXpMCc065hKeS2gQAANApCMyKrpeVrymSkRGLtpI4PuyUpldLBuXOlVEqKx091iMg9/8HetB5s99rSPdie0JY/qf+s6pYt0vXX52aWnVpaFEp4D1qQN6k1+w1w+a0UWYsOAOEgNCMy8vs7L168eOoXxOO6btUqKV5gSUY6LSUSYzcNZrPecQjLEbZskUZGJGmWRkbqHJgHB6W2tsLn2tq88zVw66231uQ6hTTzDXADA97/lRjfSrGnh+AMAGEILDSbWbuZ/ZeZ/beZPWZmn/bHzzSzB8zsCf/xjKBqQGPJ7++8bNmyos9t9RofF7Ts2WcnBuacEINzaOJxaefOycG5rc0bL/QLxzQ89dRTNbkOJurtLfzH2GvJCACopyBnmocl/b5z7kJJyyVdYWZvl3SLpF3OuddI2uUfA5P6O0+1TfesWcX/6N743/9dtC2dslmpe+q2dIFYu1Yn/ce6yw/ONQ7MzSCsXwqKtV4sNg4ACE5godl5XvQPZ/sfTtJaSdv88W2S3hVUDWhs090k5bYLLyzalk4dHVL/1G3pam7tWum++2SSdN99oQbnIYnAXMT69euLnnv88cfrWMmYYq0Xi40DAIIT6JpmM2sxs72SDkl6wDn3fUmLnHPPSJL/uLDIa3vMbI+Z7Tl8+HCQZWKGeWT+fO/mwPzg3NFx6qbBuvED8wQhBufXzplDYC5ix44dRc8dO3asjpWM6esr/MfYa8kIAKinQEOzc27UObdc0hJJF5nZGyt47Vbn3Arn3IoFCxYEVyRCl+sOMGuWatcdIB6fGJyjEphzwgrOaCjJpNdqsWDrRQBAXdWle4Zz7nl5XcKukHTQzBZLkv94qB41IJrGdwdwTrXtDuAH54xU/8AsFQ/M5Z4HpEmtFAnMABCOILtnLDCz0/3P50paLelxSfdJutZ/2rWS7g2qBkRf4N0B4nG9yiycJQnveEd15wEAQGQEOdO8WFLazH4s6Qfy1jSnJH1G0mVm9oSky/xjNKnAuwOk03rSuXDazJXquECbNgAAGkaQ3TN+7Jx7k3PuAufcG51zf+WPH3HOrXLOvcZ/fC6oGhB9gXYH8Dc4iUnh9Gfu749WFw9IYoc9AMD0sCMgQhVYd4Ao7AiYfzNiThg3JUISO+wBAKaP0IxQje8OYKaadAeI1I6AUejigVPYYQ8AGsP27dvDLmESQjNCl+sOcPKkatIdIHI7AobdxQOnsMMeADSGwcHBsEuYhNCMGSdyOwJKUjyuc/1HhIcd9oBoi+LsIpBDaEZDGh29pujNXLkdAYdbJwbn4daxpRFmVrdaczeeSaOh3XiWq+H48aNNffMbO+yhmTVCII3i7CKQQ2hGwxkYkEZH75zyZq4b/jmuK0ZSekleQnpJHbpiJKUb/tmb6Z01qz5/9KNw41kUaogKdthDMyOQAtUhNCPyurq6Jhx7N23NmzCWfzPX1q3SbsWVUEoZdSqhlHYrrq1bvfMtLS3BFj2u1rBvPItCDVJ0Wr2xwx4AYDoIzYi8devWTTgu52au0VHvcbfiOlcZ7VZ8wni9ROHGs9x7rVRav1BMK5Wuew3MdgMAGh2hGQ2nnJu5ik0k12mC+ZQo3Hi2dKkXmFNKKKYhpZTQSqXrWkNUZrsBoFKNsBYc9UFoRsPxbtqamMDyb+bq6Sn82mLjQYnCjWd3Jb3APM//ns1TVikldFeyfv2qozDbDQDTwVpw5BCa0XCSSWn16q8VvZmrq6tLW7ZI11+fm1l2amnxjrdsmXit/PXSQdQa6o1n6bRWf2EsMOfMU1arv1C/jV6iMNsNAEA1CM1oSA880F30Zq7cGugtW6SREcmsRSMjkwPz+OcGKXfjmdRS/xvPursjsdFLFGa7AaCRsUwkfIRmYCbr7w9/o5eIzHYDQCNjmUj4CM2Y8erVkzmS/C28Cy6srteW3hGZ7Z4gndZdu3YR2AEAZWviNIFmMW/evNJPmsnyg3M9A7MUjdnu8dJpKZHQoqNHpQQz3QCA8hCaMeO1t7dPGlu8eHEIlYTID85DUiCB+dixYyXfO9TZ7hw/MJ+a+c5mCc4AgLIQmtGU/vqv/zrsEuovHtdr58wJJKROGZr99w51tluaHJhz/OD8zuPH61cLAKDhEJrRlOrRNQN5/OB8cO7c+gdmqeTa6tt++9v61gNMAx0UgPAQmgFU7eTJk+U9MR7XdatW1T8wSyXXVt/4O79T33qAaaCDAhAeQjOAqo2OjoZdQmkl1lb/x5w5dS+JWUMAaByEZiBCBgakWExKpe5VLOYdo4aisLZ6HGYNAaBxEJqBiBgYkHp6pKEhSZqloSHvmOBcY2GvrQYANCRCMxARvb2FGzv09oZTTzlyM+POjTTWzHiYa6sBAA2pNewCAHj2769sPGy5mXEv6I/NjEtSMhlmZQAA1B4zzUBELF1a2XjYGnFmHACA6SI0AxHR1+fdl7ZSaf1CMa1UWh0d3ngUNdrMOAAA1SA0AxGRTEr3fiStb1pCMQ3pm5bQvR9JR3apQ6PNjAMAUA1CMxAV6bRWfyGhDueteehwWa3+QsLb/jmCcjPj40V5ZhwAgGoQmoEoSKelRKLwIuFENINzMilt3Sp1dkrSSXV2esdRnRkHAKAahGYgCrq7JwfmnGzWOx9ByaSUyUhmrcpkZmhgTqf1pHOR/MUFAFA/hGYgCvr7J691yOno8M6j/vz/AxCTwp3xT6d1165dBHcACBGhGYiC/O2dc0Le5rmUht3cpBz5S2bCWirj17Ho6NHILtUBgGZAaAaiIj84N0BgnrHbfkdljXlUgjsAgNAMRIofnA/OnRvpwCyNbW4yvq90LTc3GRiQul6R1t+lfqyuV6TrG8ajsMY8KsEdACCJ0IwG19XVFXYJtReP67pVqyIdmCVvE5OVSmun1iimIe3UGq1UuiabmwwMSPe8P627DiYU037ddTChe94fTHDOLTE5dOiZsSUm/f06aoXXmB+1Oq0xj0Jwx4yQ+zOeSt0785ZRBYjvG/IRmtHQ1q1bF3YJTeuahV5gbtewJKldw9qpNbpmYfUzoKmNae0YTmievNA4T1ntGE4otbG2s6tFl5g8HdeVLqWXNDE4v6QOXenq9H8AuDkUNTCjl1EFiO8bCiE0A6hcOq17jowF5px2DeueI2uqWzqQ9maYc4E5Z56yuutgbZcl5JaYjJdbYrJbcSWU0jG1SZKOqU0JpbRbdfo/AA16cyiiZao/4yiO7xsKITQDTaS9vb02F+rqUsvIcMFTLSPDUjXLZrq7JwXmnHmq7bIEbxZJ+og264Ra9RFtnjAuSS0amfBYVw12cyiip9hyqVoso5rJ+L6hEEIz0ETOO++8sEsorb9fI3MKL0sYmVPbZQktLV5g3qyNatWoNmujPqLNammR3tuZ1rd1uWZrVJI0W6P6ti7XezvrfANeA90ciuhZurSycXj4vqEQQjPQRM4555zaXGhwUMdntRU8dXxWmzQ4OP1rx+NqvT81KTiPzOlQ6/21DY0fGvUCs/nHJmmzNuq20RvU//TlmqMTE54/RyfU//Tl9e9c0SA3hyJ6+voKr/Dp6wunnkbB9w2FEJox49UsKGJMPK4rtVNHNTE4H1WbrtTO6sOdH5zHL0uodWBe+/OfTwjMOSbpBt0pnThR6GXe+P/8nzWrAwhSMilt3Sp1dkrSSXV2esczcsv7GuL7hkIIzZjxCM3B2HVyYnDOBeZdJ2sUbANeltC9b9+kwJxTbPyU3/ymprUAQUompUxGSiTWKpMh+JWL7xvyEZoBTEtLi9dh4krtVEadulI7tVtxtbTU8E0CXJbQf/7503/x5z9fu0IAAA2B0AxgWnp6vMfdiutcZU61YsuNR929550nbdpU+OSmTVOf27AhuMIAAJHUGnYBQLN42cteFnYJNbVli/e4das0OurU0mLq6Rkbbwi58Ltx49hYfiie6hwAoGkw0wzUycqVK8Muoea2bJFGRiSzFo2MNFhgztmwQdq0yevCnB+KpzoHAGgqhGYA2LBB5yxcWDgUb9igOWYEZgBocoRmoE66qtklDwAAhIrQDNTJunXrwi4BAABME6EZAAAAKIHQDAAAAJRAaMaMx1piAE0vndZdu3ZJ6XTYlRTXCDWiqRGaMeOxlhhAU0unpURCi44elRKJaIbSRqgR9RPRX6AIzQAAzFR+GFU26x1ns9ELpY1QI+onwr9AEZoBAJiJ8sNoTpRCaSPUiPqJ+C9QhGYAACJo+/bt1V2gu3tyGM3JZr3zYWuEGlEfDfALVGCh2cxeaWZpM9tnZo+Z2Y3++Jlm9oCZPeE/nhFUDQAANKrBwcHqLtDfL3V0FD7X0eGdD1vUa4zo2toZqQF+gQpypnlE0kbn3PmS3i5pvZm9XtItknY5514jaZd/DAAAaikel1KpyaG0o8Mbj8fDqWu8KNcY4bW1M1LUf4FSgKHZOfeMc+6H/ucvSNon6RxJayVt85+2TdK7gqoBAICmlh9KoxBG80WxxoivrZ2RovwLlK8ua5rNLCbpTZK+L2mRc+4ZyQvWkhbWowYA9KwGmpIfRg7OnRuZ8DFJlGpsgLW1M1YUf4EaJ/DQbGYvk/R1SR9xzv22gtf1mNkeM9tz+PDh4AoEmgg9q6NhYECKxaRU6l7FYt4xEKh4XNetWhWZ8FFQVGqM4traZlpbHaVfoPIEGprNbLa8wDzgnPsXf/igmS32zy+WdKjQa51zW51zK5xzKxYsWBBkmQCqkU7rSefK/se82We7Bwaknh5paEiSZmloyDsmOAMREbW1tc24tjoqv0DlCbJ7hkn6B0n7nHObx526T9K1/ufXSro3qBoABMz/xzwmlf2PebPPdvf2Fv6/vr294dSDBtJMs41hitLaWtZWR0qQM83vlPQeSb9vZnv9jyslfUbSZWb2hKTL/GMAjYZ/zKdl//7KxgFJzTnbGKYorK1lbXXkBNk943vOOXPOXeCcW+5/fMs5d8Q5t8o59xr/8bmgagAQEP4xn7alS73HlUrrF4pppdITxoFJ+AU1HGGvrY3i2uomx46AACrHP+bT1tcnXdGWVkoJxTSklBK6oi2tvr6wK0Mk8QtquMJcWxu1tdUgNAOYhv5+qa2t8Lm2tob4xzysGxKTZ6f1by6hefJC0Dxl9W8uoeTZhB8UwC+ozStKa6shidAMoEmFckOiP2vYenxiCGo9zqwhimC2sblFYW01TiE0A6hcd7c0PFz43PAws1/FMGuISjHbiLDXVuMUQjOAyjH7NT183zAdzDYion2LgxDlzZ8IzQAqx+zX9PB9w3Qx24gmEPXNnwjNAKaH2a/p4fuG6Wqi2UY0p6hv/kRoBjB9fgDMSAS/SjBrCACTRH3zJ0IzgOrE43qVGcGvUswaAsAExTZ5isrmT4RmAAAAhK6vr/AtH1HZ/InQDABAxEW5owBQK8mktHWr1NkpSSfV2ekdJ5NhV+YhNAMAEGFR7yhQC/xSgJxkUspkpERirTKZ6ARmidAMRFJYWzwDiJ6odxSoVjP8UoCZgdAMRFAoWzwDiKSodxSo1kz/paBazMJHB6EZAIAIi3pHgWrN9F8KqsEsfLQQmgEAiLCodxSo1kz/paAazMJHC6EZAIAIi3pHgWrN9F8KqsEsfLQQmgEAiLgodxSo1kz/paAazMJHC6EZABoAHVUwk83kXwqqwSx8tBCaAaAB0FEFaD7MwkcLoRlApEVhhrWlpSXsEgA0KWbho4PQDKBqs2YF909JFGZYg/z6AACNgf8SAKjavHnzwi4BAIBAEZoBVK29vT3sEgI1078+AEBphGYAKOG8884LuwQAQMgIzQAg6eqrry567qMf/WgdKwEARBGhGQAk3XHHHUXPReFmRABAuAjNAAAAQAmEZgAAAKAEQjMAAABQAqEZAAAAKIHQDAAAAJRAaAYAAABKIDQDAAAAJRCaAQAAgBIIzQAAAEAJhGYAACJkYECKxaRU6l7FYt4xgPC1hl0AAADwDAxIPT1SNitJszQ05B1LUjIZZmUAmGkGACAienu9wLxSaf1CMa1UWtmsNw4gXIRmAAAiYv9+LzCnlFBMQ0opoZVKa//+sCsDQGgGACAirlnoBeZ5ykqS5imrlBK6ZmG66mt3dXVVfQ2gmRGaAQCIgnRa//jrscCcM09Z/eOvE1K6uuC8bt26ql5fDwR7RBmhGQCAKOjuVuvxbMFTrcezUnd3nQuqv0YI9mhehGYAAKKgv1/q6Ch8rqPDOw8gNIRmAACiIB6XUqnJwbmjwxuPx8OpC4AkQjMAANGRH5wJzGhSUVzfTmgGACBK/OB8cO5cAjOaVhTXtxOaAQCImnhc161aRWCOgCjOeCIchGYAAIAiojjjiXAQmgFU7Zxzzgm7BAAAAkVoBlA1QjNQHywVAMJDaAYAoEGwVAAID6EZAELCrCGAcvHvRfgIzQCqxj/m08OsIYBy8e9F+AjNAKrGP+YAgJmO0AwAAACUEFhoNrOvmNkhM3t03NiZZvaAmT3hP54R1PsDAAAAtRLkTPPdkq7IG7tF0i7n3Gsk7fKPAQAAgEgLLDQ7574r6bm84bWStvmfb5P0rqDeHwAAAKiVeq9pXuSce0aS/MeFdX5/AAAAoGKRvRHQzHrMbI+Z7Tl8+HDY5QAAAKCJ1Ts0HzSzxZLkPx4q9kTn3Fbn3Arn3IoFCxbUrUAAAAAgX71D832SrvU/v1bSvXV+fwAAAKBiQbacG5T0n5JeZ2YHzOz9kj4j6TIze0LSZf4xAAAAEGmtQV3YOVdsX91VQb0nAAAAEITI3ggIAAAARAWhGQAAACiB0AwAAACUQGgGAAAASiA0AwAAACUQmgEAAIASCM0AAABACYRmAAAAoARCMwAAAFACoRkAAAAogdAMAAAAlEBoBgAAAEogNAMAAAAlEJoBAACAEgjNAAAAQAmEZgAAAKAEQjMAAABQAqEZAAAAKIHQDAAAAJRAaAYAAABKIDQDAAAAJRCaAQAAgBIIzQAAAEAJhGYAAACgBEIzAAAAUAKhGQAAACiB0AwAAACUQGgGAAAASiA0AwAAACUQmgEAiKCurq6wSwAwDqEZAIAIWrduXdglABiH0AwAAACUQGgGAAAASiA0AwAAACUQmgEAAIASCM0AAABACYRmAAAAoARCMwAAAFACoRkAAAAogdAMAAAAlEBoBgAAAEogNAMAAAAlEJoBAACAEgjNAAAAQAmEZgAAAKAEQjMAAABQAqEZAAAAKIHQDAAAAJRAaAYAAABKMOdc2DWUZGaHJQ2FXMZ8Sc+GXAOmh59dY+Pn17j42TU2fn6Ni59ddTqdcwvyBxsiNEeBme1xzq0Iuw5Ujp9dY+Pn17j42TU2fn6Ni59dMFieAQAAAJRAaAYAAABKIDSXb2vYBWDa+Nk1Nn5+jYufXWPj59e4+NkFgDXNAAAAIls3cgAABJZJREFUQAnMNAMAAAAlEJoLMLOMmT1iZnvNbI8/dqaZPWBmT/iPZ4RdJzxm9hUzO2Rmj44bK/rzMrOPm9nPzOynZnZ5OFVDKvqz+5SZPeX//dtrZleOO8fPLiLM7JVmljazfWb2mJnd6I/zd68BTPHz4+9fxJlZu5n9l5n9t/+z+7Q/zt+9gLE8owAzy0ha4Zx7dtzY5yQ955z7jJndIukM59zHwqoRY8zsEkkvSvo/zrk3+mMFf15m9npJg5IuknS2pAclvdY5NxpS+U2tyM/uU5JedM7dmvdcfnYRYmaLJS12zv3QzE6T9LCkd0l6r/i7F3lT/Pz+VPz9izQzM0nznHMvmtlsSd+TdKOkPxZ/9wLFTHP51kra5n++Td4/LogA59x3JT2XN1zs57VW0ledc8POuV9I+pm8f0gQgiI/u2L42UWIc+4Z59wP/c9fkLRP0v/f3h2DyFFHcRz//khEQixEokESQRE7i1NEizQpRLQRLIIKSgRBi1hoa6OlhZpGsBAFCxUOYjBYiDZ2oiIEjcZS9EzI2YgGbDTPYv6H67G7k4B3M3N+P7Ds7uy/ePB4s4+dN/s/gLU3CUvyt4j5G4nqXGxvr2qPwtrbcjbN8xXwcZKvkjzVju2vqvPQnWyAGwaLTpdjUb4OAD/NrFtj+ReFhvFMkq/b+MbGJUZzN1JJbgbuAD7H2pucTfkD62/0kuxKchpYBz6pKmtvG9g0z3eoqu4EHgCOtUvI2hky55gzSuPyOnArsAKcB15px83dCCW5BjgBPFtVvy1bOueY+RvYnPxZfxNQVX9V1QpwELg7ye1Llpu7/4hN8xxVda49rwMn6S5jXGgzYBuzYOvDRajLsChfa8BNM+sOAue2OTYtUVUX2hfCJeAN/rmMaO5Gps1TngDeqar322FrbyLm5c/6m5aq+hX4FLgfa2/L2TRvkmRvuymCJHuB+4AzwCngaFt2FPhgmAh1mRbl6xTwSJKrk9wC3AZ8MUB8WmDjpN88RFd/YO5Gpd2M9CZwtqpenfnI2puARfmz/sYvyfVJrm2v9wD3At9j7W253UMHMEL7gZPd+YTdwLtV9VGSL4HVJE8CPwJHBoxRM5K8BxwG9iVZA14AXmJOvqrq2ySrwHfAn8Ax7yAezoLcHU6yQnf58AfgaTB3I3QIeBz4ps1WAjyPtTcVi/L3qPU3ejcCbyfZRffj52pVfZjkM6y9LeVfzkmSJEk9HM+QJEmSetg0S5IkST1smiVJkqQeNs2SJElSD5tmSZIkqYdNsyRNXJKLm94/keS19vrFJD8nOZ3kTJIHh4lSkqbNplmSdr7jbcvdI8BbSTz3S9IV8sQpSf8TVXWWbnODfUPHIklT446AkjR9e2Z2dQO4jm7r3H9Jcg9wCfhluwKTpJ3CplmSpu+PNn4BdDPNwF0znz+X5DHgd+DhcitYSbpiNs2StPMdr6qXhw5CkqbMmWZJkiSph02zJEmS1COOtkmSJEnL+UuzJEmS1MOmWZIkSeph0yxJkiT1sGmWJEmSetg0S5IkST1smiVJkqQeNs2SJElSD5tmSZIkqcffw1MTBEMDPjkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting results\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12, 8))\n",
    "fig = sm.graphics.plot_fit(results, 1, ax=ax)\n",
    "ax.set_ylabel(\"MPG\")\n",
    "ax.set_xlabel(\"HP\")\n",
    "ax.set_title(\"Linear Regression\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b)**\n",
    "\n",
    "First, let's create helper functions to calculate the model variance and Mallow's $C_p$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model_variance(X, Y):\n",
    "    X = X.copy()\n",
    "    \n",
    "    # Create new column with all 1s for intercept at start\n",
    "    X.insert(0, 'const', 1)\n",
    "    \n",
    "    # Least squares solution\n",
    "    beta_hat = (np.linalg.inv(X.T @ X) @ X.T @ Y).to_numpy()\n",
    "\n",
    "    # Predicted solutions\n",
    "    Y_pred = X @ beta_hat\n",
    "\n",
    "    # Prediction errors\n",
    "    epsilon_hat = Y_pred - Y\n",
    "\n",
    "    # Error on training data\n",
    "    training_error = epsilon_hat.T @ epsilon_hat\n",
    "    \n",
    "    # Estimated error variance\n",
    "    return (training_error / (Y.shape[0] - X.shape[1]))\n",
    "    \n",
    "\n",
    "def get_mallow_cp(X, Y, S, full_model_variance):\n",
    "    if len(S) > 0:\n",
    "        X = X[list(S)].copy()\n",
    "        # Create new column with all 1s for intercept at start\n",
    "        X.insert(0, 'const', 1)\n",
    "    else:\n",
    "        X = pd.DataFrame({'const': np.ones_like(Y)})\n",
    "    \n",
    "    # Least squares solution\n",
    "    beta_hat = (np.linalg.inv(X.T @ X) @ X.T @ Y).to_numpy()\n",
    "\n",
    "    # Predicted solutions\n",
    "    Y_pred = X @ beta_hat\n",
    "\n",
    "    # Prediction errors\n",
    "    epsilon_hat = Y_pred - Y\n",
    "\n",
    "    # Error on training data\n",
    "    partial_training_error = epsilon_hat.T @ epsilon_hat\n",
    "    \n",
    "    # Increase size of S by to account for constant covariate\n",
    "    return partial_training_error + 2 * (len(S) + 1) * full_model_variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's calculate and save the full model variance **once**, sine it's used for every candidate model -- and use it to define our custom score function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_model_variance = get_model_variance(X, Y)\n",
    "\n",
    "def score_mallow_cp(S):\n",
    "    return get_mallow_cp(X, Y, S, full_model_variance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's do the submodel search.\n",
    "\n",
    "First approach is an explicit search through all features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recipe from itertools documentation, https://docs.python.org/2.7/library/itertools.html#recipes\n",
    "\n",
    "from itertools import chain, combinations\n",
    "\n",
    "def powerset(iterable):\n",
    "    \"powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)\"\n",
    "    s = list(iterable)\n",
    "    return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterate through the powerset and calculate the score for each value\n",
    "results = [(S, score_mallow_cp(S)) for S in powerset(features)]\n",
    "    \n",
    "# Format as dataframe for ease of presentation\n",
    "results = pd.DataFrame(results, columns=['S', 'score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>()</td>\n",
       "      <td>8134.147794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(HP,)</td>\n",
       "      <td>3102.805578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(SP,)</td>\n",
       "      <td>4318.234609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(WT,)</td>\n",
       "      <td>1519.372094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(VOL,)</td>\n",
       "      <td>7059.223052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(HP, SP)</td>\n",
       "      <td>2436.578842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(HP, WT)</td>\n",
       "      <td>1510.677511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(HP, VOL)</td>\n",
       "      <td>2354.823180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(SP, WT)</td>\n",
       "      <td>1463.076513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(SP, VOL)</td>\n",
       "      <td>3056.598857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>(WT, VOL)</td>\n",
       "      <td>1542.174631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(HP, SP, WT)</td>\n",
       "      <td>1140.390871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>(HP, SP, VOL)</td>\n",
       "      <td>2147.886521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>(HP, WT, VOL)</td>\n",
       "      <td>1507.484328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>(SP, WT, VOL)</td>\n",
       "      <td>1443.795004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>(HP, SP, WT, VOL)</td>\n",
       "      <td>1160.807643</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    S        score\n",
       "0                  ()  8134.147794\n",
       "1               (HP,)  3102.805578\n",
       "2               (SP,)  4318.234609\n",
       "3               (WT,)  1519.372094\n",
       "4              (VOL,)  7059.223052\n",
       "5            (HP, SP)  2436.578842\n",
       "6            (HP, WT)  1510.677511\n",
       "7           (HP, VOL)  2354.823180\n",
       "8            (SP, WT)  1463.076513\n",
       "9           (SP, VOL)  3056.598857\n",
       "10          (WT, VOL)  1542.174631\n",
       "11       (HP, SP, WT)  1140.390871\n",
       "12      (HP, SP, VOL)  2147.886521\n",
       "13      (HP, WT, VOL)  1507.484328\n",
       "14      (SP, WT, VOL)  1443.795004\n",
       "15  (HP, SP, WT, VOL)  1160.807643"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(HP, SP, WT)</td>\n",
       "      <td>1140.390871</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               S        score\n",
       "11  (HP, SP, WT)  1140.390871"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[results.score == results.score.min()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This approach recommends the features HP, SP, and WT."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's do forward stepwise feature selection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(WT, SP, HP)</td>\n",
       "      <td>1140.390871</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              S        score\n",
       "0  (WT, SP, HP)  1140.390871"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "current_subset = []\n",
    "current_score = score_mallow_cp(current_subset)\n",
    "\n",
    "while len(current_subset) < len(features):\n",
    "    best_score, best_subset = current_score, current_subset\n",
    "    updated = False\n",
    "    for f in features:\n",
    "        if f not in current_subset:\n",
    "            candidate_subset = current_subset + [f]\n",
    "            candidate_score = score_mallow_cp(candidate_subset)\n",
    "            if candidate_score < best_score:\n",
    "                best_score, best_subset = candidate_score, candidate_subset\n",
    "                updated = True              \n",
    "    if not updated:\n",
    "        break\n",
    "        \n",
    "    current_score, current_subset = best_score, best_subset\n",
    "    \n",
    "pd.DataFrame([[tuple(current_subset), current_score]], columns=['S', 'score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This approach also recommends selecting these 3 features (order is irrelevant): WT, SP, HP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's so backward stepwise feature selection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(HP, SP, WT)</td>\n",
       "      <td>1140.390871</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              S        score\n",
       "0  (HP, SP, WT)  1140.390871"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "current_subset = features\n",
    "current_score = score_mallow_cp(current_subset)\n",
    "\n",
    "while len(current_subset) > 0:\n",
    "    best_score, best_subset = current_score, current_subset\n",
    "    updated = False\n",
    "    for f in features:\n",
    "        if f in current_subset:\n",
    "            candidate_subset = [a for a in current_subset if a != f]\n",
    "            candidate_score = score_mallow_cp(candidate_subset)\n",
    "            if candidate_score < best_score:\n",
    "                best_score, best_subset = candidate_score, candidate_subset\n",
    "                updated = True              \n",
    "    if not updated:\n",
    "        break\n",
    "        \n",
    "    current_score, current_subset = best_score, best_subset\n",
    "    \n",
    "pd.DataFrame([[tuple(current_subset), current_score]], columns=['S', 'score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This approach also recommends selecting the same 3 features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(c)**  This is analogous to (b), using the new scoring function.\n",
    "\n",
    "Note that the book's definition of BIC is:\n",
    "\n",
    "$$ \\text{BIC}(S) = \\text{RSS}(S) + 2 |S| \\hat{\\sigma}^2 $$\n",
    "\n",
    "while other sources apply a log to this quantity and scale it by the number of observations $n$, e.g.\n",
    "\n",
    "$$ \\text{BIC} = n \\log \\left( \\text{RSS} / n \\right) + |S| \\log n $$\n",
    "\n",
    "We will use the later definition for this exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_bic(X, Y, S):\n",
    "    if len(S) > 0:\n",
    "        X = X[list(S)].copy()\n",
    "        # Create new column with all 1s for intercept at start\n",
    "        X.insert(0, 'const', 1)\n",
    "    else:\n",
    "        X = pd.DataFrame({'const': np.ones_like(Y)})\n",
    "    \n",
    "    # Least squares solution\n",
    "    beta_hat = (np.linalg.inv(X.T @ X) @ X.T @ Y).to_numpy()\n",
    "\n",
    "    # Predicted solutions\n",
    "    Y_pred = X @ beta_hat\n",
    "\n",
    "    # Prediction errors\n",
    "    epsilon_hat = Y_pred - Y\n",
    "\n",
    "    # Error on training data\n",
    "    rss = epsilon_hat.T @ epsilon_hat\n",
    "    \n",
    "    n = Y.shape[0]\n",
    "    k = X.shape[1]\n",
    "    \n",
    "    return n * np.log(rss / n) + k * np.log(n)\n",
    "\n",
    "def score_bic(S):\n",
    "    return get_bic(X, Y, S)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Full search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterate through the powerset and calculate the score for each value\n",
    "results = [(S, score_bic(S)) for S in powerset(features)]\n",
    "    \n",
    "# Format as dataframe for ease of presentation\n",
    "results = pd.DataFrame(results, columns=['S', 'score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>()</td>\n",
       "      <td>381.100039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(HP,)</td>\n",
       "      <td>305.324815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(SP,)</td>\n",
       "      <td>332.832040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(WT,)</td>\n",
       "      <td>245.266568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(VOL,)</td>\n",
       "      <td>373.531556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(HP, SP)</td>\n",
       "      <td>288.594470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(HP, WT)</td>\n",
       "      <td>247.670065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(HP, VOL)</td>\n",
       "      <td>285.699095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(SP, WT)</td>\n",
       "      <td>244.895261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(SP, VOL)</td>\n",
       "      <td>307.747647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>(WT, VOL)</td>\n",
       "      <td>249.455822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(HP, SP, WT)</td>\n",
       "      <td>225.425717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>(HP, SP, VOL)</td>\n",
       "      <td>281.219751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>(HP, WT, VOL)</td>\n",
       "      <td>250.346085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>(SP, WT, VOL)</td>\n",
       "      <td>246.530268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>(HP, SP, WT, VOL)</td>\n",
       "      <td>229.333642</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    S       score\n",
       "0                  ()  381.100039\n",
       "1               (HP,)  305.324815\n",
       "2               (SP,)  332.832040\n",
       "3               (WT,)  245.266568\n",
       "4              (VOL,)  373.531556\n",
       "5            (HP, SP)  288.594470\n",
       "6            (HP, WT)  247.670065\n",
       "7           (HP, VOL)  285.699095\n",
       "8            (SP, WT)  244.895261\n",
       "9           (SP, VOL)  307.747647\n",
       "10          (WT, VOL)  249.455822\n",
       "11       (HP, SP, WT)  225.425717\n",
       "12      (HP, SP, VOL)  281.219751\n",
       "13      (HP, WT, VOL)  250.346085\n",
       "14      (SP, WT, VOL)  246.530268\n",
       "15  (HP, SP, WT, VOL)  229.333642"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(HP, SP, WT)</td>\n",
       "      <td>225.425717</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               S       score\n",
       "11  (HP, SP, WT)  225.425717"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[results.score == results.score.min()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Forward stepwise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(WT, SP, HP)</td>\n",
       "      <td>225.425717</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              S       score\n",
       "0  (WT, SP, HP)  225.425717"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "current_subset = []\n",
    "current_score = score_bic(current_subset)\n",
    "\n",
    "while len(current_subset) < len(features):\n",
    "    best_score, best_subset = current_score, current_subset\n",
    "    updated = False\n",
    "    for f in features:\n",
    "        if f not in current_subset:\n",
    "            candidate_subset = current_subset + [f]\n",
    "            candidate_score = score_bic(candidate_subset)\n",
    "            if candidate_score < best_score:\n",
    "                best_score, best_subset = candidate_score, candidate_subset\n",
    "                updated = True              \n",
    "    if not updated:\n",
    "        break\n",
    "        \n",
    "    current_score, current_subset = best_score, best_subset\n",
    "    \n",
    "pd.DataFrame([[tuple(current_subset), current_score]], columns=['S', 'score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Backward stepwise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(HP, SP, WT)</td>\n",
       "      <td>225.425717</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              S       score\n",
       "0  (HP, SP, WT)  225.425717"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "current_subset = features\n",
    "current_score = score_bic(current_subset)\n",
    "\n",
    "while len(current_subset) > 0:\n",
    "    best_score, best_subset = current_score, current_subset\n",
    "    updated = False\n",
    "    for f in features:\n",
    "        if f in current_subset:\n",
    "            candidate_subset = [a for a in current_subset if a != f]\n",
    "            candidate_score = score_bic(candidate_subset)\n",
    "            if candidate_score < best_score:\n",
    "                best_score, best_subset = candidate_score, candidate_subset\n",
    "                updated = True              \n",
    "    if not updated:\n",
    "        break\n",
    "        \n",
    "    current_score, current_subset = best_score, best_subset\n",
    "    \n",
    "pd.DataFrame([[tuple(current_subset), current_score]], columns=['S', 'score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All approaches recommend the same feature selection as the previous method -- select the 3 features HP, SP, and WT."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(d)**  To use Lasso, we will need to minimize the L1 loss function for an arbitrary penalty parameter $\\lambda$:\n",
    "\n",
    "$$ \\sum_{i=1}^n (Y_i - \\hat{Y}_i)^2 + \\lambda \\sum_{j=1}^k | \\beta_j | $$\n",
    "\n",
    "Since we are including a penalty parameter that affects both estimation and model selection, we will need to calculate this loss function on some test data distinct from the training data -- the recommended approach being to use leave-one-out cross validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.optimize import minimize\n",
    "\n",
    "# Lasso loss function\n",
    "def lasso_loss(Y, Y_pred, beta, l1_penalty):\n",
    "    error = Y - Y_pred\n",
    "    return error.T @ error + l1_penalty * sum(abs(beta))\n",
    "\n",
    "# Regularized fit\n",
    "def fit_regularized(X, Y, l1_penalty):\n",
    "    def loss_function(beta):\n",
    "        return lasso_loss(Y, X @ beta, beta, l1_penalty)\n",
    "    \n",
    "    # Use the solution without penalties as an initial guess\n",
    "    beta_initial_guess = (np.linalg.inv(X.T @ X) @ X.T @ Y).to_numpy()\n",
    "    return minimize(loss_function, beta_initial_guess, method = 'Powell',\n",
    "                    options={'xtol': 1e-8, 'disp': False, 'maxiter': 10000 }) \n",
    "\n",
    "# Leave-one-out cross-validation\n",
    "def leave_one_out_cv_risk(X, Y, fitting_function):\n",
    "    n = X.shape[1]\n",
    "    total_risk = 0\n",
    "    for i in range(n):\n",
    "        XX = pd.concat([X.iloc[:i], X.iloc[i + 1:]])\n",
    "        YY = pd.concat([Y.iloc[:i], Y.iloc[i + 1:]])\n",
    "        beta = fitting_function(XX, YY).x\n",
    "        validation_error = Y.iloc[i] - X.iloc[i] @ beta\n",
    "        total_risk += validation_error * validation_error\n",
    "    return total_risk / n\n",
    "\n",
    "# Optimize over penalty parameter with best cross-validation risk\n",
    "def optimize_l1_penalty(X, Y):\n",
    "    def loss_function(l1_penalty_signed):\n",
    "        # Ensure l1_penalty >= 0\n",
    "        l1_penalty = abs(l1_penalty_signed)\n",
    "        return leave_one_out_cv_risk(X, Y, lambda xx, yy: fit_regularized(xx, yy, l1_penalty))\n",
    "    \n",
    "    l1_penalty_initial_guess = 0.0\n",
    "    return minimize(loss_function, l1_penalty_initial_guess, method = 'Powell',\n",
    "                   options={'xtol': 1e-8, 'disp': True, 'maxiter': 10000 })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new dimension with constants, so the regressions have an intercept\n",
    "X_c = X.copy()\n",
    "X_c.insert(0, 'const', 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 61.554007\n",
      "         Iterations: 1\n",
      "         Function evaluations: 45\n",
      "Selected penalty:  4.965056196086726e-07\n"
     ]
    }
   ],
   "source": [
    "# Optimize cross validation risk over penalties\n",
    "best_penalty_res = optimize_l1_penalty(X_c, Y)\n",
    "selected_l1_penalty = abs(best_penalty_res.x)\n",
    "\n",
    "print(\"Selected penalty: \", selected_l1_penalty)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>const</th>\n",
       "      <td>192.437753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HP</th>\n",
       "      <td>0.392212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SP</th>\n",
       "      <td>-1.294818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WT</th>\n",
       "      <td>-1.859804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VOL</th>\n",
       "      <td>-0.015645</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             coef\n",
       "const  192.437753\n",
       "HP       0.392212\n",
       "SP      -1.294818\n",
       "WT      -1.859804\n",
       "VOL     -0.015645"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Re-fit with selected penalty over the whole dataset\n",
    "selected_fit = fit_regularized(X_c, Y, selected_l1_penalty)\n",
    "beta = selected_fit.x\n",
    "pd.DataFrame(beta.reshape(-1, 1), index=X_c.columns, columns=['coef'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best leave-one-out cross validation for the Lasso procedure is achieved (according to our optimizer) at $\\lambda \\approx$  0, where all covariants have a non-zero coefficient (i.e. $\\beta_i \\neq 0$ for all $i$).  \n",
    "\n",
    "In other words, Lasso with leave-one-out cross validation selects the full model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.8**.  Assume that the errors are Normal.  Show that the model with highest AIC is the model with lowest Mallows $C_p$ statistic.\n",
    "\n",
    "Mallows' $C_p$:\n",
    "\n",
    "$$\\hat{R}(S) = \\hat{R}_\\text{tr}(S) + 2 |S| \\hat{\\sigma}^2$$\n",
    "\n",
    "AIC:\n",
    "\n",
    "$$  \\text{AIC} = \\ell_S - |S|$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "For the given definition, note that $-2 \\text{AIC} \\hat{\\sigma}^2 = -2 \\ell_S \\hat{\\sigma}^2 + 2|S| \\hat{\\sigma}^2 = \\hat{R}_\\text{tr}(S) + 2 |S| \\hat{\\sigma}^2 = \\hat{R}(S)$, given the log likelihood of a Normal model -- so maximizing AIC is equivalent to minimizing Mallows $C_p$.\n",
    "\n",
    "Note that a more general result holds: rather than assuming the errors are Normal, one can assume that the distributions are part of a spherically symmetric family, i.e., modifying the distributions under an orthogonal transformation (and potentially removing invariance).  See: Boisbunon, Aurélie, et al. \"AIC, Cp and estimators of loss for elliptically symmetric distributions.\" arXiv preprint arXiv:1308.2766 (2013)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14.9.9**.  In this question we will take a closer look at the AIC method.  Let $X_1, \\dots, X_n$ be iid observations.  Consider two models $\\mathcal{M}_0$ and $\\mathcal{M}_1$.  Under $\\mathcal{M}_0$ the data are assumed to be $N(0, 1)$ while under $M_1$ the data are assumed to be $N(\\theta, 1)$ for some unknown $\\theta \\in \\mathbb{R}$:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathcal{M}_0 : X_1, \\dots, X_n &\\sim N(0, 1) \\\\\n",
    "\\mathcal{M}_1 : X_1, \\dots, X_n &\\sim N(\\theta, 1), \\; \\theta \\in \\mathbb{R}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "This is just another way of viewing the hypothesis testing problem: $H_0: \\theta = 0$ versus $H_1: \\theta \\neq 0$.  Let $\\ell_n(\\theta)$ be the log-likelihood function.  The AIC score for a model is the log-likelihood at the MLE minus the number of parameters.  (Some people multiply this score by 2 but that is irrelevant).  Thus, the AIC score for $\\mathcal{M}_0$ is $\\text{AIC}_0 = \\ell_n(0)$ and the AIC score for $\\mathcal{M}_1$ is $\\text{AIC}_1 = \\ell_n(\\hat{\\theta}) - 1$.  Suppose we choose the model with highest AIC score.  Let $J_n$ denote the selected model:\n",
    "\n",
    "$$J_n = \\begin{cases}\n",
    "0 & \\text{if } \\text{AIC}_0 > \\text{AIC}_1 \\\\\n",
    "1 & \\text{if } \\text{AIC}_1 > \\text{AIC}_0\n",
    "\\end{cases}$$\n",
    "\n",
    "**(a)** Suppose that $\\mathcal{M}_0$ is the true model, i.e. $\\theta = 0$.  Find\n",
    "\n",
    "$$ \\lim_{n \\rightarrow \\infty} \\mathbb{P}(J_n = 0) $$\n",
    "\n",
    "Now compute $\\lim_{n \\rightarrow \\infty} \\mathbb{P}(J_n = 0)$ when $\\theta \\neq 0$.\n",
    "\n",
    "**(b)** The fact that $\\lim_{n \\rightarrow \\infty} \\mathbb{P}(J_n = 0) \\neq 1$ when $\\theta = 0$ is why some people say that AIC \"overfits\". But this is not quite true as we shall now see.  Let $\\phi_\\theta(x)$ denote a Normal density function with mean $\\theta$ and variance 1.  Define\n",
    "\n",
    "$$ \\hat{f}_n(x) = \\begin{cases}\n",
    "\\phi_0(x) & \\text{if } J_n = 0 \\\\\n",
    "\\phi_\\overline{\\theta}(x) & \\text{if } J_n = 1\n",
    "\\end{cases}$$\n",
    "\n",
    "If $\\theta = 0$, show that $D(\\phi_0, \\hat{f}_n) \\xrightarrow{\\text{P}} 0$ as $n \\rightarrow \\infty$ where\n",
    "\n",
    "$$ D(f, g) = \\int f(x) \\log \\left( \\frac{f(x)}{g(x)} \\right) dx $$\n",
    "\n",
    "is the Kullback-Leibler distance.  Show that $D(\\phi_\\theta, \\hat{f}_n) \\xrightarrow{\\text{P}} 0$ if $\\theta \\neq 0$.  Hence, AIC consistently estimates the true density even if it \"overshoots\" the correct model.\n",
    "\n",
    "REMARK: If you are feeling ambitious, repeat the analysis for BIC which is the log-likelihood minus $(p / 2) \\log n$ where $p$ is the number of parameters and $n$ is the sample size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "**(a)**  Note that the log-likelihood of the distribution $N(\\mu, \\sigma^2)$ is:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\ell_n(\\mu, \\sigma^2) &= \\log \\prod_{i=1}^n f(X_i | \\mu, \\sigma^2) \\\\\n",
    "&= \\sum_{i=1}^n \\log f(X_i | \\mu, \\sigma^2) \\\\\n",
    "&= - \\frac{n}{2} \\log 2\\pi - \\frac{n}{2} \\log \\sigma^2 - \\frac{1}{2\\sigma^2} \\sum_{i=1}^n (X_i - \\mu)^2\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "\n",
    "Then, we have:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{P}(J_n = 0) &= \\mathbb{P}(\\text{AIC}_0 > \\text{AIC}_1) \\\\\n",
    "&= \\mathbb{P}(\\ell_n(0) > \\ell_n(\\hat{\\theta}) - 1) \\\\\n",
    "&= \\mathbb{P} \\left(-\\frac{n}{2} \\log 2\\pi - 0 - \\frac{1}{2} \\sum_{i=1}^n (X_i - 0)^2 > -\\frac{n}{2} \\log 2\\pi - 0 - \\frac{1}{2} \\sum_{i=1}^n (X_i - \\hat{\\theta})^2 - 1 \\right) \\\\\n",
    "&= \\mathbb{P}\\left( -\\frac{1}{2} \\sum_{i=1}^n X_i^2 > -\\frac{1}{2} \\sum_{i=1}^n (X_i - \\hat{\\theta})^2 - 1\\right) \\\\\n",
    "&= \\mathbb{P}\\left( \\sum_{i=1}^n \\left((X_i - \\hat{\\theta})^2 - X_i^2 \\right) > -2 \\right) \\\\\n",
    "&= \\mathbb{P}\\left( n \\hat{\\theta}^2 - 2 \\hat{\\theta} \\sum_{i=1}^n X_i > -2 \\right) \\\\\n",
    "&= \\mathbb{P}\\left( n \\overline{X}_n^2 - 2 \\overline{X}_n n \\overline{X}_n > -2\\right) \\\\\n",
    "&= \\mathbb{P}\\left( -n \\overline{X}_n^2 > -2 \\right) \\\\\n",
    "&= \\mathbb{P}\\left(-\\sqrt{\\frac{2}{n}} < \\overline{X}_n < \\sqrt{\\frac{2}{n}} \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "But $X_i \\sim N(\\theta, 1)$, so $n \\overline{X}_n = \\sum_i X_i \\sim N(n\\theta, n)$ and $\\overline{X}_n \\sim N(\\theta, 1/n)$.  Then:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{P}(J_n = 0) &= \\mathbb{P}\\left(-\\sqrt{\\frac{2}{n}} < \\overline{X}_n < \\sqrt{\\frac{2}{n}} \\right) \\\\\n",
    "&= \\mathbb{P}\\left(\\frac{-\\sqrt{\\frac{2}{n}} - \\theta}{\\sqrt{1/n}} < \\frac{\\overline{X}_n - \\theta}{\\sqrt{1/n}} < \\frac{\\sqrt{\\frac{2}{n}} - \\theta}{\\sqrt{1/n}} \\right) \\\\\n",
    "&= \\mathbb{P}\\left(-\\sqrt{2} - \\sqrt{n}\\theta < Z < \\sqrt{2} - \\sqrt{n}\\theta \\right) \\\\\n",
    "&= \\Phi(\\sqrt{2} - \\sqrt{n}\\theta) - \\Phi(-\\sqrt{2} - \\sqrt{n}\\theta)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "where $\\Phi$ is the CDF of the standard normal distribution $N(0, 1)$.\n",
    "\n",
    "When $\\theta = 0$,\n",
    "\n",
    "$$\\mathbb{P}(J_n = 0) = \\Phi(\\sqrt{2}) - \\Phi(-\\sqrt{2}) \\approx 0.8427 \\neq 0$$\n",
    "\n",
    "When $\\theta \\neq 0$,\n",
    "\n",
    "$$\\lim_{n \\rightarrow \\infty} \\mathbb{P}(J_n = 0) = \\lim_{n \\rightarrow \\infty}\\Phi(\\sqrt{2} - \\sqrt{n}\\theta) - \\Phi(-\\sqrt{2} - \\sqrt{n}\\theta) = \\lim_{n \\rightarrow \\infty}\\Phi(\\sqrt{n}\\theta) - \\lim_{n \\rightarrow \\infty}\\Phi(-\\sqrt{n}\\theta) = 0$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b)** We have:\n",
    "\n",
    "$$ D(\\phi_\\theta, \\hat{f}_n) = \\int \\phi_\\theta(x) \\log \\left(\\frac{\\phi_\\theta(x)}{\\hat{f}_n(x)} \\right) dx = \\int \\left[\\phi_\\theta(x) \\log \\phi_\\theta(x) - \\phi_\\theta(x) \\log \\hat{f}_n(x) \\right] dx $$\n",
    "\n",
    "But $\\lim_{n \\rightarrow \\infty} \\hat{f}_n(x) = \\phi_\\theta(x)$, so the integrand goes to 0 at each x, and so $D(\\phi_\\theta, \\hat{f}_n) \\xrightarrow{\\text{P}} 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**REMARK: I am feeling ambitious.**  Let $K_n$ denote the selected model:\n",
    "\n",
    "$$ K_n = \\begin{cases}\n",
    "0 & \\text{if } \\text{BIC}_0 > \\text{BIC}_1 \\\\\n",
    "1 & \\text{if } \\text{BIC}_1 > \\text{BIC}_0\n",
    "\\end{cases}$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\n",
    "\\text{BIC}_0 = \\ell_n(0) \\\\\n",
    "\\text{BIC}_1 = \\ell_n(\\hat{\\theta}) - \\frac{1}{2} \\log n\n",
    "$$\n",
    "\n",
    "So,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{P}(K_n = 0) &= \\mathbb{P}(\\text{BIC}_0 > \\text{BIC}_1) \\\\\n",
    "&= \\mathbb{P}\\left(\\ell_n(0) > \\ell_n(\\hat{\\theta}) - \\frac{1}{2} \\log n \\right) \\\\\n",
    "&= \\mathbb{P} \\left(-\\frac{n}{2} \\log 2\\pi - 0 - \\frac{1}{2} \\sum_{i=1}^n (X_i - 0)^2 > -\\frac{n}{2} \\log 2\\pi - 0 - \\frac{1}{2} \\sum_{i=1}^n (X_i - \\hat{\\theta})^2 - \\frac{1}{2} \\log n \\right) \\\\\n",
    "&= \\mathbb{P}\\left( -\\frac{1}{2} \\sum_{i=1}^n X_i^2 > -\\frac{1}{2} \\sum_{i=1}^n (X_i - \\hat{\\theta})^2 - \\frac{1}{2} \\log n\\right) \\\\\n",
    "&= \\mathbb{P}\\left( \\sum_{i=1}^n \\left((X_i - \\hat{\\theta})^2 - X_i^2 \\right) > -\\log n \\right) \\\\\n",
    "&= \\mathbb{P}\\left( n \\hat{\\theta}^2 - 2 \\hat{\\theta} \\sum_{i=1}^n X_i > -\\log n \\right) \\\\\n",
    "&= \\mathbb{P}\\left( n \\overline{X}_n^2 - 2 \\overline{X}_n n \\overline{X}_n > -\\log n\\right) \\\\\n",
    "&= \\mathbb{P}\\left( -n \\overline{X}_n^2 > -\\log n \\right) \\\\\n",
    "&= \\mathbb{P}\\left(-\\sqrt{\\frac{\\log n}{n}} < \\overline{X}_n < \\sqrt{\\frac{\\log n}{n}} \\right) \\\\\n",
    "&= \\mathbb{P}\\left(\\frac{-\\sqrt{\\frac{\\log n}{n}} - \\theta}{\\sqrt{1/n}} < \\frac{\\overline{X}_n - \\theta}{\\sqrt{1/n}} < \\frac{\\sqrt{\\frac{\\log n}{n}} - \\theta}{\\sqrt{1/n}} \\right) \\\\ \n",
    "&=  \\mathbb{P}\\left(-\\sqrt{\\log n} - \\sqrt{n}\\theta < Z < \\sqrt{\\log n} - \\sqrt{n}\\theta \\right) \\\\\n",
    "&= \\Phi(\\sqrt{\\log n} - \\sqrt{n}\\theta) - \\Phi(-\\sqrt{\\log n} - \\sqrt{n}\\theta)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "As $O(\\sqrt{log n}) < O(\\sqrt{n})$, we get the result:\n",
    "\n",
    "$$ \\lim_{n \\rightarrow \\infty} \\mathbb{P}(K_n = 0) = \\begin{cases}\n",
    "1 & \\text{if } \\theta = 0 \\\\\n",
    "0 & \\text{if } \\theta \\neq 0\n",
    "\\end{cases}$$\n",
    "\n",
    "Also, if we define:\n",
    "\n",
    "$$\\hat{g}_n(x) = \\begin{cases}\n",
    "\\phi_0(x) & \\text{if } K_n = 0 \\\\\n",
    "\\phi_\\overline{\\theta}(x) & \\text{if } K_n = 1\n",
    "\\end{cases}$$\n",
    "\n",
    "then, again, \n",
    "\n",
    "$$ D(\\phi_\\theta, \\hat{g}_n) = \\int \\phi_\\theta(x) \\log \\left(\\frac{\\phi_\\theta(x)}{\\hat{g}_n(x)} \\right) dx = \\int \\left[\\phi_\\theta(x) \\log \\phi_\\theta(x) - \\phi_\\theta(x) \\log \\hat{g}_n(x) \\right] dx $$\n",
    "\n",
    "But $\\lim_{n \\rightarrow \\infty} \\hat{g}_n(x) = \\phi_\\theta(x)$, so the integrand goes to 0 at each x, and so $D(\\phi_\\theta, \\hat{g}_n) \\xrightarrow{\\text{P}} 0$."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}