Linear Regression Step by Step.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ccf0bf87",
   "metadata": {},
   "source": [
    "# In Depth Analysis of Linear Regression\n",
    "- **ML Overview**\n",
    "    - Example, Algorithms vs Model\n",
    "- **Supervised Learning**\n",
    "    - Definition, Examples\n",
    "- **Supervised Learning Setup**\n",
    "    - Nomenclature, Formulation(`Regression` & `Classification`),  Example,  Learning,  Hypothesis Class.\n",
    "    - Performance Evaluation\n",
    "        - Loss Function, 0/1 Loss Function, Squared Loss, Root Mean squared error, Absolute Loss\n",
    "    - Generalization: The Train-Test Split, Generalization loss.\n",
    "    \n",
    "[Linear Regression](#Linear-Regression)\n",
    "\n",
    "- **Single Feature**\n",
    "- **Multiple Feature**\n",
    "- **Model Formulation and Setup**\n",
    "- **Loss Function**\n",
    "   - How to solve?\n",
    "   - Reformulation\n",
    "   - Consequently\n",
    "- **Solve Optimization Problem (Analytical Solution employing Calculus)**\n",
    "- **Model Evaluation Techniques**\n",
    "- **Polynomial Regression**\n",
    "- **How to Handle Overfitting?**\n",
    "- **Regularization (Ridge Regression and Lasso Regression)**\n",
    "- **Gradient Descent Algorithm**\n",
    "   - Formulation\n",
    "   - Algorithm\n",
    "   - Types \n",
    "- **Linear Regression Implementation in Python**\n",
    "- **Linear Regression Implementation using sklearn**\n",
    "- **Project:** Medical Insurance Cost Prediction\n",
    "- **Interview Questions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf1cc6a2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "40f24ee6",
   "metadata": {},
   "source": [
    "<h2 align='center' > ML Overview</h2> \n",
    "\n",
    "### What is Machine Learning?\n",
    "- Automating the process of automation\n",
    "- Getting computers to program themselves\n",
    "\n",
    "<figure>\n",
    "<img src=\"images/p1.png\" width=300px height=300px>\n",
    "<figcaption align = \"center\" ><b> Given examples(Training data), make a machine learn system behavior or discover patterns. </b> </figcaption>\n",
    "</figure>\n",
    "\n",
    "\n",
    "> **A simple definition of machine learning is the ability of a computer or machine to improve its performance on a specific task through experience. It involves training a model on a data set, and then using that model to make predictions or take actions based on new, unseen data.**\n",
    "\n",
    "> **In other words, machine learning is a way for computers to learn and make decisions on their own, without being explicitly programmed to perform a specific task.**\n",
    "-----\n",
    "\n",
    "<img src=\"images/p2.png\">\n",
    "\n",
    "&ensp;\n",
    "\n",
    "**Note**: An algorithm is a set of steps that a model follows to learn from data and make predictions. A model, on the other hand, is a representation of the relationships between the input data and the output predictions. It is trained on a dataset and is able to make predictions on new data based on what it has learned from the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c033440f",
   "metadata": {},
   "source": [
    "### Rules vs. Learning\n",
    "- `Rules-based` approaches involve following a predetermined set of rules or instructions to solve a problem or make a decision. These rules are typically established in advance and are followed regardless of the specific circumstances or context.\n",
    "\n",
    "- On the other hand, `learning-based` approaches involve using data and experience to improve decision-making over time. In the context of machine learning, this involves training a model on a dataset, allowing the model to learn patterns and relationships in the data, and using this learned knowledge to make predictions on new data.\n",
    "\n",
    "- Both rules-based and learning-based approaches have their own strengths and weaknesses, and the most appropriate approach will depend on the specific problem or task at hand. Rules-based approaches are often simpler and easier to implement, but they may not be able to adapt to changing circumstances or new data as well as learning-based approaches. Learning-based approaches, on the other hand, may be more complex and require more data and computation, but they have the ability to improve over time and adapt to new situations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5940a3a",
   "metadata": {},
   "source": [
    "<h2 align=\"center\">Supervised Learning</h2> \n",
    "\n",
    "**Predicting the labels for unseen data based on labelled\n",
    "instances.** \n",
    "\n",
    "<img src=\"images/p4.png\" height=500px width=500px align='center'>\n",
    "\n",
    "- Each column is a feature and adds one dimension to the data\n",
    "- Number of columns define total number of features and hence data dimensionality.\n",
    "- `Inputs`: referred to as Features.\n",
    "- `Output`: referred to as Label.\n",
    "- `Training data`: (input, output) for which the output is known and is used for training a model by ML algorithm.\n",
    "- `A Loss, an objective or a cost function`: determines how well a trined model approximates the training data\n",
    "- `Test data`: (input, output) for which the output is known and is used for the evaluation of the performance of the trained model\n",
    "\n",
    "#### A Recipe for Applying Supervised Learning\n",
    "\n",
    "To apply supervised learning, we define a dataset and a learning algorithm.\n",
    "\n",
    "$$ \\text{Dataset} + \\text{Learning Algorithm} \\to \\text{Predictive Model} $$\n",
    "\n",
    "The output is a predictive model that maps inputs to targets. For instance, it can predict targets on new inputs.\n",
    "\n",
    "-----\n",
    "The learning algorithm would receive a set of inputs along with the corresponding correct outputs to train a model.\n",
    "\n",
    "<img src=\"images/p5.png\" height=600px width=600px align=\"center\">\n",
    "&ensp;\n",
    "\n",
    "<img src=\"images/p8.png\" height=600px width=600px align=\"center\">\n",
    "\n",
    "\n",
    "### Algorithms vs Model\n",
    "- `Linear regression` algorithm produces a model, that is, a vector of values of the coefficients of the model.\n",
    "- `Decision tree` algorithm produces a model comprised of a tree of if-then statements with specific values.\n",
    "- `Neural network` along with backpropagation + gradient descent: produces a model comprised of a trained (weights assigned) neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3e25535",
   "metadata": {},
   "source": [
    "<h2 align=\"center\">Supervised Learning Setup</h2>\n",
    "\n",
    "##### A Supervised Learning Dataset: Notation\n",
    "\n",
    "Using the adopted notation, we can formalize the supervised machine learning setup. We represent the entire training data as \n",
    "$$\\mathcal{D} = \\{(x^{(i)}, y^{(i)}) \\mid i = 1,2,...,n\\} \\in X^{R} \\times Y$$\n",
    "\n",
    "Where\n",
    "- D is a dataset\n",
    "- X is the d-dimensional feature space($R^d$)\n",
    "- $x_i$ is the input vector of the <i>ith</i> sample\n",
    "- Y is the label space\n",
    "\n",
    "Each $x^{(i)}$ denotes an input (e.g., the measurements for patient $i$), and each $y^{(i)} \\in \\mathcal{Y}$ is a target (e.g., the Heart Disease). \n",
    "\n",
    "<b><u>Regression</u></b>:    $\\mathcal{y} = R (prediction-on-continuous-scale)$\n",
    "\n",
    "<b><u>Classification</u></b>:    $\\mathcal{y} = {{0,1}} \\;\\;or \\;\\;{-1,1} \\;\\; or \\;\\; 1,2 \\;\\; binary-classification $\n",
    "\n",
    "$\\mathcal{y} = {1,2,3......,M} \\;\\;\\; M-class-Classification$\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Together, $(x^{(i)}, y^{(i)})$ form a *training example*.\n",
    "&emsp;\n",
    "<h3 align='center'>Example</h3>\n",
    "<img src=\"images/9.png\" height=500px width=500px >\n",
    "\n",
    "&emsp;\n",
    "\n",
    "<h3 align='center'>Learning </h3>\n",
    "\n",
    "We want to develop a model that can predict the label for the input for which label is **unknown.**\n",
    "\n",
    "We assume that the data points $(x_i,y_i)$ are drawn from some **unknown** distribution $P(X,Y)$.\n",
    "<img src=\"images/10.png\" align='center'>\n",
    "\n",
    "Our goal is to learn the machine (model, function or hypothesis) $h\\in H$, such that for a new pair/instance $(x,y)P$ , we can use *h* to obtain\n",
    "$$h(x) =y$$\n",
    "\n",
    "with high probability or \n",
    "$$h(x)\\approx y$$\n",
    "\n",
    "in some optimal sense.\n",
    "\n",
    "**Note:** Here, we do not know exact distribution but using **ML**, we try to estimate that distribution. Here, $h$ is our hypothesis, this is basically returns a mapping between input and output. $h$ is a machine. It is a sample from Hypothesis $H$. Here, we want to learn a function from dataset which takes input and produces its output.\n",
    "\n",
    "<h3 align='center'> Model/Function/Hypothesis: Notation</h3>\n",
    "\n",
    "We'll say that a model is a function\n",
    "$$ f : \\mathcal{X} \\to \\mathcal{Y} $$\n",
    "that maps inputs $x \\in \\mathcal{X}$ to targets $y \\in \\mathcal{Y}$.\n",
    "\n",
    "Often, models have *parameters* $\\theta \\in \\Theta$ living in a set $\\Theta$. We will then write the model as\n",
    "$$ f_\\theta : \\mathcal{X} \\to \\mathcal{Y} $$\n",
    "to denote that it's parametrized by $\\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e91caaf",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Feature Space</h3>  \n",
    "\n",
    "**Definition:** A feature space is a set of mathematical features or variables that are used to represent the data. It is a multi-dimensional space where each dimension corresponds to a feature or variable, and each data point is represented as a point in this space.\n",
    "\n",
    "**Example:** If you have a dataset containing three features (age, height, and weight), the feature space would be a three-dimensional space where each data point is represented by its age, height, and weight.\n",
    "\n",
    "- Student data(e.g. for predicting grades), an input $x^{(i)} \\in \\mathcal{X}$ is a $d$-dimensional vector of the form\n",
    "$$ x^{(i)} = \\begin{bmatrix}\n",
    "x^{(i)}_1 \\\\\n",
    "x^{(i)}_2 \\\\\n",
    "\\vdots \\\\\n",
    "x^{(i)}_d\n",
    "\\end{bmatrix}$$\n",
    "- Where each $x_j^i \\;$ is the value of the $ith$ feature for student $j$.\n",
    "- Examples of features, $x^i$\n",
    "    - Scores in assignments, quizzes, exams\n",
    "    - Educational records, grades in previous courses\n",
    "    - Rankings of previous educational institutes\n",
    "    - Interaction with online tools? Missed instruments?\n",
    "- Small number of features and relatively low features with 0 values: **Dense vectors**.\n",
    "\n",
    "The set $\\mathcal{X}$ is called the feature space. Often, we have, $\\mathcal{X} = \\mathbb{R}^d$.\n",
    "\n",
    "**Attributes:** We refer to the numerical variables describing the patient as *attributes*. Examples of attributes include:\n",
    "* The age of a patient.\n",
    "* The patient's gender.\n",
    "* The patient's BMI.\n",
    "\n",
    "**Features:** Often, an input object has many attributes, and we want to use these attributes to define more complex descriptions of the input.\n",
    "\n",
    "* Is the patient old and a man? (Useful if old men are at risk).\n",
    "* Is the BMI above the obesity threshold?\n",
    "\n",
    "We call these custom attributes *features*.\n",
    "\n",
    "**Feature:** We may denote features via a function $\\phi : \\mathcal{X} \\to \\mathbb{R}^p$ that takes an input $x^{(i)} \\in \\mathcal{X}$ and outputs a $p$-dimensional vector\n",
    "$$ \\phi(x^{(i)}) = \\left[\\begin{array}{@{}c@{}}\n",
    "\\phi(x^{(i)})_1 \\\\\n",
    "\\phi(x^{(i)})_2 \\\\\n",
    "\\vdots \\\\\n",
    "\\phi(x^{(i)})_p\n",
    "\\end{array} \\right]$$\n",
    "We say that $\\phi(x^{(i)})$ is a *featurized* input, and each $\\phi(x^{(i)})_j$ is a *feature*.\n",
    "\n",
    "**Features vs Attributes**:\n",
    "In practice, the terms attribute and features are often used interchangeably. Most authors refer to $x^{(i)}$ as a vector of features. We will follow this convention and use the term \"attribute\" only when there is ambiguity between features and attributes. Features can be either discrete or continuous."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "936edb6e",
   "metadata": {},
   "source": [
    "<h3 align='center'>Label Space</h3>\n",
    "Formally, when $(x^{(i)}, y^{(i)})$ form a *training example*, each $y^{(i)} \\in \\mathcal{Y}$ is a target. We call $\\mathcal{Y}$ the target space.\n",
    "\n",
    "- Binary (one-of-two) – Binary classification\n",
    "    - Sentiment: positive / negative\n",
    "    - Email: Spam / Not Spam\n",
    "    - Online Transactions: Fraudulent (Yes / No)\n",
    "    - Tumor: Malignant / Benign\n",
    "    - y ∈ 0,1 e.g. 0: Negative class, 1: Positive class\n",
    "    - y ∈ −1,1 e.g. -1: Negative class, 1: Positive class\n",
    "- **Multi-class (one-of-many, many-of-many problems) – Multi-class classification**\n",
    "    - Sentiment: Positive / negative / neutral\n",
    "    - Emotion: Happy, Sad, Surprised, Angry,...\n",
    "    - Part-of-Speech tag: Noun / verb / adjective / adverb /...\n",
    "    - Recognize a word: One of |V| tags\n",
    "    - y ∈ 0,1,2,3, ... e.g. 0: Happy, 1: Sad, 2, Angry,...\n",
    "- **Real-valued – Regression**\n",
    "    - Temperature, height, age, length, weight, duration, price...\n",
    "    \n",
    "**Regression vs. Classification**\n",
    "\n",
    "\n",
    "1. __Regression__: The target variable $y$ is continuous. We are fitting a curve in a high-dimensional feature space that approximates the shape of the dataset.\n",
    "2. __Classification__: The target variable $y$ is discrete. Each discrete value corresponds to a *class* and we are looking for a hyperplane that separates the different classes.\n",
    "\n",
    "<h3 align='center'>Feature Matrix</h3>\n",
    "\n",
    "Suppose that we have a dataset of size $n$ (e.g., $n$ patients), indexed by $i=1,2,...,n$. Each $x^{(i)}$ is a vector of $d$ features.\n",
    "\n",
    "Machine learning algorithms are most easily defined in the language of linear algebra. Therefore, it will be useful to represent the entire dataset as one matrix $X \\in \\mathbb{R}^{n \\times d}$, of the form:\n",
    "$$ X = \\begin{bmatrix}\n",
    "x^{(1)}_1 & x^{(2)}_1 & \\ldots & x^{(n)}_1 \\\\\n",
    "x^{(1)}_2 & x^{(2)}_2 & \\ldots & x^{(n)}_2 \\\\\n",
    "\\vdots \\\\\n",
    "x^{(1)}_d & x^{(2)}_d & \\ldots & x^{(n)}_d\n",
    "\\end{bmatrix}.$$\n",
    "\n",
    "Similarly, we can vectorize the target variables into a vector $y \\in \\mathbb{R}^n$ of the form\n",
    "$$ y = \\begin{bmatrix}\n",
    "y^{(1)} \\\\\n",
    "y^{(2)} \\\\\n",
    "\\vdots \\\\\n",
    "y^{(n)}\n",
    "\\end{bmatrix}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ef7cd04",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Hypothesis Space</h3>\n",
    "\n",
    "- We call the set of possible functions or candidate models `the hypothesis class`.\n",
    "- The hypothesis $h$ is sampled from a hypothesis space $H$.\n",
    "$$h \\in H$$\n",
    "\n",
    "- $H$ can be thought of to contain classes of hypotheses which share sets of assumptions like\n",
    "    - Decisions tree\n",
    "    - Perceptron\n",
    "    - Neural networks\n",
    "    - Support Vector Machines\n",
    "    \n",
    "**Example:** $h \\in H$ for $H$: Decision trees, would be instances of\n",
    "decisions trees of different height, arity, thresholds, etc.\n",
    "\n",
    "- In machine learning, the `hypothesis space` is the set of all possible hypotheses that a learning algorithm can consider when making predictions. It is an important concept because the choice of hypotheses can significantly affect the performance of a learning algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "908967ba",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<function <lambda> at 0x7f74c0c24ee0>, <function <lambda> at 0x7f74c0c24e50>, <function <lambda> at 0x7f74c0c24f70>, <function <lambda> at 0x7f74c0c24670>, <function <lambda> at 0x7f74c134e1f0>, <function <lambda> at 0x7f74c134e040>, <function <lambda> at 0x7f74c134e0d0>, <function <lambda> at 0x7f74c134e160>, <function <lambda> at 0x7f74c134e310>, <function <lambda> at 0x7f74c134e3a0>, <function <lambda> at 0x7f74c134e430>, <function <lambda> at 0x7f74c134e4c0>, <function <lambda> at 0x7f74c134e550>, <function <lambda> at 0x7f74c134e5e0>, <function <lambda> at 0x7f74c134e670>, <function <lambda> at 0x7f74c134e700>, <function <lambda> at 0x7f74c134e790>, <function <lambda> at 0x7f74c134e820>, <function <lambda> at 0x7f74c134e8b0>, <function <lambda> at 0x7f74c134e940>, <function <lambda> at 0x7f74c134e9d0>, <function <lambda> at 0x7f74c134ea60>, <function <lambda> at 0x7f74c134eaf0>, <function <lambda> at 0x7f74c134eb80>, <function <lambda> at 0x7f74c134ec10>, <function <lambda> at 0x7f74c134eca0>, <function <lambda> at 0x7f74c134ed30>, <function <lambda> at 0x7f74c134edc0>, <function <lambda> at 0x7f74c134ee50>, <function <lambda> at 0x7f74c134eee0>, <function <lambda> at 0x7f74c134ef70>, <function <lambda> at 0x7f74c136a1f0>, <function <lambda> at 0x7f74c136a040>, <function <lambda> at 0x7f74c136a0d0>, <function <lambda> at 0x7f74c136a160>, <function <lambda> at 0x7f74c136a280>, <function <lambda> at 0x7f74c136a3a0>, <function <lambda> at 0x7f74c136a430>, <function <lambda> at 0x7f74c136a4c0>, <function <lambda> at 0x7f74c136a550>, <function <lambda> at 0x7f74c136a5e0>, <function <lambda> at 0x7f74c136a670>, <function <lambda> at 0x7f74c136a700>, <function <lambda> at 0x7f74c136a790>, <function <lambda> at 0x7f74c136a820>, <function <lambda> at 0x7f74c136a8b0>, <function <lambda> at 0x7f74c136a940>, <function <lambda> at 0x7f74c136a9d0>, <function <lambda> at 0x7f74c136aa60>, <function <lambda> at 0x7f74c136aaf0>, <function <lambda> at 0x7f74c136ab80>, <function <lambda> at 0x7f74c136ac10>, <function <lambda> at 0x7f74c136aca0>, <function <lambda> at 0x7f74c136ad30>, <function <lambda> at 0x7f74c136adc0>, <function <lambda> at 0x7f74c136ae50>, <function <lambda> at 0x7f74c136aee0>, <function <lambda> at 0x7f74c136af70>, <function <lambda> at 0x7f74c141ce50>, <function <lambda> at 0x7f74c141cca0>, <function <lambda> at 0x7f74c141c040>, <function <lambda> at 0x7f74c0c85280>, <function <lambda> at 0x7f74c0c85040>, <function <lambda> at 0x7f74c0c850d0>, <function <lambda> at 0x7f74c0c85160>, <function <lambda> at 0x7f74c0c851f0>, <function <lambda> at 0x7f74c0c85310>, <function <lambda> at 0x7f74c0c85430>, <function <lambda> at 0x7f74c0c854c0>, <function <lambda> at 0x7f74c0c85550>, <function <lambda> at 0x7f74c0c855e0>, <function <lambda> at 0x7f74c0c85670>, <function <lambda> at 0x7f74c0c85700>, <function <lambda> at 0x7f74c0c85790>, <function <lambda> at 0x7f74c0c85820>, <function <lambda> at 0x7f74c0c858b0>, <function <lambda> at 0x7f74c0c85940>, <function <lambda> at 0x7f74c0c859d0>, <function <lambda> at 0x7f74c0c85a60>, <function <lambda> at 0x7f74c0c85af0>, <function <lambda> at 0x7f74c0c85b80>, <function <lambda> at 0x7f74c0c85c10>, <function <lambda> at 0x7f74c0c85ca0>, <function <lambda> at 0x7f74c0c85d30>, <function <lambda> at 0x7f74c0c85dc0>, <function <lambda> at 0x7f74c0c85e50>, <function <lambda> at 0x7f74c0c85ee0>, <function <lambda> at 0x7f74c0c85f70>, <function <lambda> at 0x7f74c0cb6280>, <function <lambda> at 0x7f74c0cb6040>, <function <lambda> at 0x7f74c0cb60d0>, <function <lambda> at 0x7f74c0cb6160>, <function <lambda> at 0x7f74c0cb61f0>, <function <lambda> at 0x7f7452c7c8b0>, <function <lambda> at 0x7f7452c7c3a0>, <function <lambda> at 0x7f7452c883a0>, <function <lambda> at 0x7f7452c88310>, <function <lambda> at 0x7f7452c88280>, <function <lambda> at 0x7f7452c880d0>, <function <lambda> at 0x7f7452c88700>, <function <lambda> at 0x7f7452c88790>, <function <lambda> at 0x7f7452c88820>, <function <lambda> at 0x7f7452c888b0>, <function <lambda> at 0x7f7452c88940>, <function <lambda> at 0x7f7452c889d0>, <function <lambda> at 0x7f7452c88a60>, <function <lambda> at 0x7f7452c88af0>, <function <lambda> at 0x7f7452c88b80>, <function <lambda> at 0x7f7452c88c10>, <function <lambda> at 0x7f7452c88ca0>, <function <lambda> at 0x7f7452c88d30>, <function <lambda> at 0x7f7452c88dc0>, <function <lambda> at 0x7f7452c88e50>, <function <lambda> at 0x7f7452c88ee0>, <function <lambda> at 0x7f7452c88f70>, <function <lambda> at 0x7f74c1359040>, <function <lambda> at 0x7f74c13590d0>, <function <lambda> at 0x7f74c1359160>, <function <lambda> at 0x7f74c13591f0>, <function <lambda> at 0x7f74c1359280>, <function <lambda> at 0x7f74c1359310>]\n"
     ]
    }
   ],
   "source": [
    "# Import necessary libraries\n",
    "import numpy as np\n",
    "\n",
    "# Define the input space X as a matrix of four rows and two columns, \n",
    "# representing four possible input examples with two features each.\n",
    "X = np.array([[0, 0], \n",
    "              [0, 1], \n",
    "              [1, 0], \n",
    "              [1, 1]])\n",
    "\n",
    "# Define the output space Y as a vector of four values, representing \n",
    "#  the corresponding output labels for each of the four input examples.\n",
    "Y = np.array([0, 1, 1, 0])\n",
    "\n",
    "# Define the hypothesis space H\n",
    "H = []\n",
    "\n",
    "# Iterate over all possible combinations of weights w1 and w2\n",
    "for w1 in range(-5, 6):\n",
    "    for w2 in range(-5, 6):\n",
    "        # Append the hypothesis h(x) = w1 * x1 + w2 * x2 to the hypothesis space H\n",
    "        result = lambda x: w1 * X[0] + w2 * X[1]\n",
    "        H.append(result)\n",
    "\n",
    "# Print the hypothesis space\n",
    "print(H)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3452a33b",
   "metadata": {},
   "source": [
    ">**Note:** The code prints the resulting `hypothesis space` $H$, which is a list of all possible hypotheses that can be considered by the learning algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb32a7a0",
   "metadata": {},
   "source": [
    "<h3><b><i>Question:</i></b> \n",
    "    For a given problem, How we can select/choose hypothesis(machine) $h \\in h$.</h3>\n",
    "    \n",
    "**Answer:**\n",
    "- **Randomly**\n",
    "    - May not work well\n",
    "    - Like using a random program to solve a sorting problem\n",
    "    - May work if $H$ is constrained enough\n",
    "\n",
    "- **Exhaustively**\n",
    "    - Would be very slow\n",
    "    - The space $H$ is usually very large (if not infinite)\n",
    "\n",
    "- $H$ is usually chosen by data scientists (you!) based on their experience!\n",
    "    - $h \\in H$ is estimated efficiently using various optimization techniques. Define hypothesis class $H$ for a given learning algorithm. Evaluate the performance of each candidate function and choose the best one."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3b4de7a",
   "metadata": {},
   "source": [
    "<h3><i>Question<i/>: How do we evaluate the performance? </h3>\n",
    "\n",
    "**Answer:** \n",
    "    \n",
    "Define a loss function to quantify/calculate the accuracy of the prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22de0490",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Loss Function</h3>\n",
    "\n",
    "- Loss function should quantify/calculate the average error in predicting $y$ using hypothesis function $h$ and input $x$. It is denoted by $L$.\n",
    "\n",
    "- Smaller is better\n",
    "    - 0 loss: No error\n",
    "    - 100% loss: Could not even get one instance right\n",
    "    - 50% loss: Your h is as informative as a coin toss\n",
    "    \n",
    "&emsp;    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2aca097d",
   "metadata": {},
   "source": [
    "\n",
    "<h3 align=\"center\"><b>0/1 Loss</b></h3>\n",
    "\n",
    "- The `0/1 loss` is a loss function that is used to evaluate the performance of a machine learning model in binary classification tasks. It is defined as the number of incorrect predictions made by the model, divided by the total number of predictions.\n",
    "\n",
    "<img src=\"https://www.baeldung.com/wp-content/ql-cache/quicklatex.com-3fc482ec51a32e213970a07a3de41d10_l3.svg\" height=600px width=600px>\n",
    "\n",
    "- Counts the average number of mistakes in predicting $y$\n",
    "- Returns the training error rate\n",
    "- Not used due to Non-continuous and non-differentiable\n",
    "    - Difficult to utilize in optimization\n",
    "- Used to evaluate classifiers in binary/multiclass settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "673466a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n"
     ]
    }
   ],
   "source": [
    "# Import necessary libraries\n",
    "import numpy as np\n",
    "\n",
    "# Define the true labels y_true as a numpy array of four values, \n",
    "# representing the correct class labels for four input examples. \n",
    "y_true = np.array([0, 1, 0, 1])\n",
    "\n",
    "# Define the predicted labels y_pred as a numpy array of \n",
    "# four values, representing the class labels predicted by the model.\n",
    "y_pred = np.array([0, 0, 1, 1])\n",
    "\n",
    "# Calculate the 0/1 loss\n",
    "loss = np.mean(y_true != y_pred)\n",
    "\n",
    "# def zero_one_loss(y_true, y_pred):\n",
    "#     loss = 0\n",
    "#     for yt, yp in zip(y_true, y_pred):\n",
    "#         if yt != yp:\n",
    "#             loss += 1\n",
    "#     return loss\n",
    "\n",
    "# Print the loss\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72264972",
   "metadata": {},
   "source": [
    "> **Note:** The output will be `0.5`, indicating that the model made 2 incorrect predictions out of a total of 4, or 50% error."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2866a04",
   "metadata": {},
   "source": [
    "<h3 align=\"center\"><b>Squared Loss Function</b></h3>\n",
    "\n",
    "The squared loss function, also known as the mean squared error (MSE) loss, is a common loss function used in regression tasks. It measures the average squared difference between the predicted values and the true values. The squared loss function is defined as:\n",
    "\n",
    "$$L_{sq}(h) = \\frac{1}{2n} \\sum_{i=1}^n \\left( h(x^{(i)}) - y^{(i)} \\right)^2$$\n",
    "\n",
    "Where $h(x^{(i)})$ is the predicted value, $y^{(i)}$ is the true value, and $n$ is the number of samples. These are defined for a dataset $\\mathcal{D} = \\{(x^{(i)}, y^{(i)}) \\mid i = 1,2,...,n\\}$.\n",
    "\n",
    "\n",
    "- Typically used in regression settings\n",
    "- The loss is always non-negative\n",
    "- The loss grows quadratically with the absolute magnitude of mis-prediction\n",
    "- Encourages no predictions to be really far off\n",
    "- If a prediction is very close to be correct, the square will be tiny and little attention will be given to that example to obtain zero error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "f22d4220",
   "metadata": {},
   "outputs": [],
   "source": [
    "def squared_loss(y_true, y_pred):\n",
    "    loss = 0\n",
    "    for yt, yp in zip(y_true, y_pred):\n",
    "        loss += (yt - yp) ** 2\n",
    "    loss /= len(y_true)\n",
    "    return loss\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c54177eb",
   "metadata": {},
   "source": [
    "**Note:** Function `squared_loss` takes in two arguments: `y_true` and `y_pred`. `y_true` is a list of true values, and `y_pred` is a list of predicted values. The function calculates the squared difference between each pair of true and predicted values and sums them up. It then divides the total loss by the number of samples to get the average loss. The function returns the average loss as a float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "13d8b9a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.65\n"
     ]
    }
   ],
   "source": [
    "y_true = [1, 2, 3, 4, 5]\n",
    "y_pred = [1.5, 2.5, 2.5, 4.5, 6.5]\n",
    "\n",
    "loss = squared_loss(y_true, y_pred)\n",
    "print(loss)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56d0091b",
   "metadata": {},
   "source": [
    "<h3 align=\"center\"><b>Absolute Loss Function</b></h3>\n",
    "\n",
    "The absolute loss function, also known as the mean absolute error (MAE) loss, is another common loss function used in regression tasks. It measures the average absolute difference between the predicted values and the true values. The absolute loss function is defined as:\n",
    "\n",
    "$$L_{abs}(h) = \\frac{1}{n} \\sum_{i=1}^n \\left| h(x^{(i)}) - y^{(i)} \\right|$$\n",
    "\n",
    "Where $h(x^{(i)})$ is the predicted value, $y^{(i)}$ is the true value, and $n$ is the number of samples. These are defined for a dataset $\\mathcal{D} = \\{(x^{(i)}, y^{(i)}) \\mid i = 1,2,...,n\\}$.\n",
    "\n",
    "\n",
    "- The loss is always non-negative\n",
    "- The loss grows linearly with the absolute magnitude of mis-prediction\n",
    "- Better suited for noisy data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "ceb11696",
   "metadata": {},
   "outputs": [],
   "source": [
    "def absolute_loss(y_true, y_pred):\n",
    "    loss = 0\n",
    "    for yt, yp in zip(y_true, y_pred):\n",
    "        loss += abs(yt - yp)\n",
    "    loss /= len(y_true)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aadf20f1",
   "metadata": {},
   "source": [
    "**Note:** The `absolute_loss` function takes in two arguments: `y_true` and `y_pred`. `y_true` is a list of true values, and `y_pred` is a list of predicted values. The function calculates the absolute difference between each pair of true and predicted values and sums them up. It then divides the total loss by the number of samples to get the average loss. The function returns the average loss as a float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "61ef8124",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7\n"
     ]
    }
   ],
   "source": [
    "y_true = [1, 2, 3, 4, 5]\n",
    "y_pred = [1.5, 2.5, 2.5, 4.5, 6.5]\n",
    "\n",
    "loss = absolute_loss(y_true, y_pred)\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9a5e164",
   "metadata": {},
   "source": [
    "<h4 align=\"center\"><b>Comparsion</b></h4>\n",
    "<img src=\"images/p12.png\" align=\"left\">\n",
    "<img src=\"images/p13.png\" align=\"right\" height=400px width=400px>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f22bfa82",
   "metadata": {},
   "source": [
    "\n",
    "<h2 align=\"center\"> Learning: The elusive h</h2>\n",
    "\n",
    "We wish to select hypothesis (machine) $h\\in \\mathbf{H} $ such that \n",
    "$$h^* = avgmin_{h\\in H} L(h)$$\n",
    "\n",
    "> **Note:** The average of the minimum loss achieved by the hypothesis h when it is trained on different datasets. In other words, it is a measure of the performance of the hypothesis h across multiple datasets. To calculate the average minimum loss, we would need to first define a loss function that quantifies the difference between the model's predictions and the true values in the dataset. We would then train the hypothesis h on multiple datasets and record the minimum loss achieved on each dataset. Finally, we would take the average of all the minimum losses to get the average minimum loss of the hypothesis h. It is important to note that the average minimum loss is just one way to measure the performance of a hypothesis. Other metrics, such as accuracy or precision, may also be used to evaluate the performance of the model.\n",
    "\n",
    "**Recall:** We assume that the data points $(\\mathbf{x_i}, y_i)$ are drawn from some unknown distribution P(X,Y). We can come up with a function $h$ after solving this minimization problem that gives low loss on our data.\n",
    "\n",
    "**Question: How can we ensure that hypothesis h will give low loss on the input not in D?**\n",
    "\n",
    "**Answer:** To view this, let us consider a model $h$ trained on every input in D, that is, giving zero loss. Such fucntion is referred to as memorizer and can be formulated as follows\n",
    "<img src=\"images/p64.png\">\n",
    "\n",
    "> **Note:** A memorizer function is a type of machine learning model that simply memorizes the training data and uses it to make predictions, without attempting to learn any patterns or relationships. Memorizer functions are sometimes referred to as \"memorization algorithms\" or \"lookup tables.\" One disadvantage of memorizer functions is that they do not generalize well to new data. Since they have simply memorized the training data and do not understand the underlying patterns or relationships, they are not able to make accurate predictions on unseen data. This means that memorizer functions are likely to perform poorly on test sets or in real-world situations where they are presented with new data.\n",
    "\n",
    "**Interpretation:**\n",
    "We can interpretate from above function that\n",
    "- 0% loss error on the training data (Model is fit to every data point in D).\n",
    "- Large error for some input not in D\n",
    "- First glimpse of `overfitting.`\n",
    "\n",
    "\n",
    "**Revisit:**\n",
    "\n",
    "**Question: How can we ensure that hypothesis h will give low loss on the input not in D?**\n",
    "\n",
    "**Answer:**\n",
    "$$\\mathbf{Train/Test \\;Split} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d8b4ba9",
   "metadata": {},
   "source": [
    "<h3 align=\"center\"> Generalization: The Train-Test Split </h3>\n",
    "\n",
    "To resolve the overfitting issue, we usually, split $D$ into train and test subsets:\n",
    "\n",
    "   - $D_{TR} $ as the training data, (70,80, or 80%)\n",
    "   - $D_{TE} $ as the testing data, (30,20, or 10%)    \n",
    "<img src=\"images/p65.png\">\n",
    "\n",
    "How to carry out splitting?\n",
    "- Split should be capturing the variations in the distributions.\n",
    "- Usually, we carry out sampling using independent and identically distributed sampling and time series with respect to time.\n",
    "\n",
    "> **Note:** In statistics and machine learning, i.i.d. (independent and identically distributed) sampling refers to the process of selecting a sample of data from a population in such a way that each data point is chosen independently of the others, and the probability distribution of the sample is the same as the distribution of the population.\n",
    "\n",
    "**Note:**\n",
    "- **You can only use the test dataset once after deciding on the model using training dataset**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0acd1565",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Learning (Revisit after train-test split</h3>\n",
    "\n",
    "We had the following optimization problem as  $h\\in \\mathbf{H} $ such that \n",
    "$$h^* = avgmin_{h\\in H} L(h)$$\n",
    "we generalize it as\n",
    "<img src=\"images/p66.png\">\n",
    "\n",
    "**Evaluation:**\n",
    "Loss on the testing data is given by\n",
    "<img src=\"images/p67.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4604a81e",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Generalization loss</h3>\n",
    "\n",
    "- We define the generalized loss on the distribution P from which the D is drawn as the expected value(average value, probaility weighted average to be precise) of the loss for a given $h^*s$\n",
    "$$e  = E_{(x,y)\\backsim P}[L(\\mathbf{x},y|h^*)] $$\n",
    "\n",
    "- That the expected loss should be calculated on any data point sampled from the distribution P, not necessarily those present in D.\n",
    "- Under the assumption that data D is i.i.d drawn from $P$, $e_TE$ serves as unbiased estimator of the generalized loss $e$. This simply means $e_TE$ converges to $e$ with the increase in the data size, that is, \n",
    "$$\\lim_{n \\to \\infty} e_{TE} = e$$\n",
    "\n",
    "- How to get a new datapoint $(x, y)$ ∼ P?\n",
    "    - All we have are the $n$ data points!\n",
    "\n",
    "- We estimate $e$ by splitting the D into parts.\n",
    "- We train on $D_{TR}$ and test on $D_{TE}$ only once! Don’t train on test inadvertently! (e.g. repeated testing). We never look at the test data. We train only using the $D_{TR}$ and only use the $D_{TE}$ once. Then the test error approximates the generalization loss\n",
    "\n",
    ">**Note:** Generalization loss refers to the difference between the performance of a ML model on the $D_{TR}$ and its performance on new, unseen data. It is a measure of how well a model is able to generalize from the training data to make accurate predictions on new data. In general, the goal of a ML model is to have low generalization loss, which means that it is able to perform well on both the $D_{TR}$ and new data. A model with high generalization loss may be overfitting, which means that it is too closely tailored to the training data and is not able to generalize to new data. On the other hand, a model with low generalization loss may be underfitting, which means that it is not able to fully capture the patterns and relationships in the $D_{TR}$. There are various techniques that can be used to reduce generalization loss, such as `regularization`, `feature selection`, and `early stopping`. These techniques aim to balance the complexity of the model with its ability to generalize to new data, in order to improve its overall performance.\n",
    "\n",
    "**Question: How to we evaluate the model, if we do not have access to\n",
    "the test data while training?**\n",
    "\n",
    "**Answer**:\n",
    "\n",
    "We usually split $D$ into three subsets datasets that are:\n",
    "\n",
    "   - $D_{TR} $ as training data, (80%)\n",
    "   - $D_{VA} $ as validation data, (10%)    \n",
    "   - $D_{TE} $ as test data, (10%)    \n",
    "   \n",
    "> **Idea:** Validation data is used evaluate the loss for a function h that is determined using the learning on the training data-set. If the loss on validation data is high for a given h, the hypothesis or model\n",
    "needs to be changed.\n",
    "\n",
    "**More explanation* to better understand the difference between validation and test data:**\n",
    "\n",
    "- **Training set:** A set of examples used for learning, that is to fit the parameters of the hypothesis (model).\n",
    "- **Validation set:** A set of examples used to tune the hyper-parameters of the hypothesis function, for example to choose the number of hidden units in a neural network OR the order of polynomial approximating the data.\n",
    "- **Test set:** A set of examples used only to assess the performance of a fully-specified model or hypothesis.\n",
    "\n",
    "### Generalization: The Train-Test Split (Example)\n",
    "<img src=\"images/p68.png\">\n",
    "\n",
    "- Train on DTR, tune parameters or calculate error on DVA, and finally test once on DTE\n",
    "- Cross validation simulates multiple train-test splits on the training data\n",
    "- Finally, train on the whole data once, before shipping out\n",
    "\n",
    "> **Note:** Cross-validation is a useful tool for evaluating the performance of a ML model and selecting the best model based on the performance. It helps to avoid `overfitting`, which is the phenomenon where a model performs well on the training data but poorly on new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14244c10",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "106f08a0",
   "metadata": {},
   "source": [
    "# Linear Regression\n",
    "\n",
    "**Regression:** Quantitative Prediction on a continuous scale.\n",
    "<img src=\"images/p11.png\">\n",
    "\n",
    "> Here, `PROCESS` or `SYSTEM` refers to any underlying physical or logical phenomenon which maps our input data to our observed and noisy output data.\n",
    "\n",
    "- **One Variable Regression:** $\\;\\;y$ is a scalar.\n",
    "- **Multi-Variable Regression:** $\\;\\;y$ is a vector.\n",
    "- **Single feature Regression:** $\\;\\;x$ is a scalar.\n",
    "- **Multiple feature Regression:** $\\;\\;x$ is a vector.\n",
    "\n",
    "<h3 align=\"center\"> Model Formulation and Setup</h3>\n",
    "\n",
    "#### *True Model:*\n",
    "We assume there is an inherent but unknown relationship between\n",
    "input and output.\n",
    "$y = f(x) + n$\n",
    "\n",
    "<img src=\"images/p16.png\" align=\"right\" height=400px width=400px>\n",
    "\n",
    "#### *Goal:* \n",
    "Given noisy observations, we need to estimate the unknown functional\n",
    "relationship as accurately as possible.\n",
    "\n",
    "We have:\n",
    "- For some input $x,\\hat y$ is our model output.\n",
    "- Assume that our model is $\\hat f(x,\\theta)$, characterized by the paramter(s) $\\theta$.\n",
    "- Model $f(x,\\theta)$ has\n",
    "    - A structure (e.g linear, polynomial, inverse).\n",
    "    - Parameters in the vector $\\theta = [\\theta_1,\\theta_2, \\theta_3,....,\\theta_M]$\n",
    "- Our Model error is $\\epsilon = y - \\hat y$.\n",
    "\n",
    "<img src=\"images/p17.png\" align=\"left\" height=400px width=400px>\n",
    "\n",
    "<img src=\"images/p18.png\" align=\"right\" height=500px width=500px>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e115be28",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Model</h3>\n",
    "\n",
    "We have :\n",
    "\n",
    "$$\\mathcal{D} = \\{(x^{(i)}, y^{(i)}) \\mid i = 1,2,...,n\\} \\in X^{R} \\times Y$$\n",
    "\n",
    "Model is a linear function of the features, that is \n",
    "\n",
    "$$\\hat f(X,\\theta) = \\theta_0 + \\sum_{i=1}^d(\\theta_ix_i) = \\theta_0 + \\theta^TX$$\n",
    "- Linear Structure\n",
    "- Model Parameters: $\\theta_0 \\;\\; and \\; \\theta = [\\theta_1,\\theta_2, \\theta_3,....,\\theta_d]$\n",
    "    - $\\theta_0$ is bias or intercept.\n",
    "    - $\\theta = [\\theta_1,\\theta_2, \\theta_3,....,\\theta_d]$ represents the weights or slope.\n",
    "    - $\\theta_i$ quanitfies the contribution of i-th feature $x_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10f3215",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">What is Linear Model?</h3>\n",
    "\n",
    "#### When d = 1:\n",
    "$$\\hat f(X,\\theta) = \\theta_0 + \\theta_1x$$\n",
    "$$\\hat f(X,\\theta) = \\theta_0 + \\theta_1x$$\n",
    "- This represents the equation of `line`.\n",
    "<img src=\"images/p19.png\" align=\"right\">\n",
    "\n",
    "#### When d=2:\n",
    "$$\\hat f(X,\\theta) = \\theta_0 + \\theta_1x_1 + \\theta_2x_2$$\n",
    "- This represents the equation of the `Plane`\n",
    "<img src=\"images/p20.png\" align=\"right\">\n",
    "\n",
    "\n",
    "#### When d=d:\n",
    "$$\\hat f(X,\\theta) = \\theta_0 + \\theta_TX$$\n",
    "- This represents the `Hyper-plane` in $R^{d+1}$.\n",
    "\n",
    "----\n",
    "- For different $\\theta_0$ and , <b>$\\theta$</b>, we have differnt hyper-planes.\n",
    "- How do we find the `best` line?\n",
    "- What do we mean by the `best`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "746dc641",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f1f282d4",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Linear Regression with one Variable </h3>\n",
    "\n",
    "**Notation:**\n",
    "\n",
    "- **m** = Number of training samples\n",
    "- **x** = Feature\n",
    "- **y** = Label\n",
    "- $(x^i , y^i)$: the ith sample in the dataset\n",
    "\n",
    "<img src=\"https://cdn.scribbr.com/wp-content/uploads//2020/02/simple-linear-regression-graph.png\" align=\"right\">\n",
    "<img src=\"images/21.png\" align=\"left\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fdde7a7",
   "metadata": {},
   "source": [
    "$$h_\\theta(x) = \\theta_0+ \\theta_1x$$\n",
    "\n",
    "> **Linear Regression with one variable is also called univarite linear regression, simple linear regression.**\n",
    "\n",
    "**Parameters:**\n",
    "$$\\theta_0, \\theta_1$$\n",
    "\n",
    "**Cost Function**\n",
    "$$J(\\theta_0, \\theta_1) = \\frac{1}{m} \\sum_{i=1}^m (h_\\theta(x^{(i)})-y^{(i)})^2$$\n",
    "\n",
    "**Goal:**\n",
    "$$minimum_{\\theta_0,\\theta_1} J(\\theta_0,\\theta_1)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "92a1b03c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.3333333333333333\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "def cost_function(y_pred, y):\n",
    "    n = len(y)\n",
    "    cost = 1/n * np.sum((y_pred - y)**2)\n",
    "    return cost\n",
    "y_pred = np.array([1, 2, 3])\n",
    "y = np.array([1, 2, 2])\n",
    "cost = cost_function(y_pred, y)\n",
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d52de0e",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">A simplified case</h3>\n",
    "\n",
    "<img src=\"images/21.jpeg\">\n",
    "<img src=\"images/22.jpeg\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca337a78",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Using both of the <i>knobs</i> </h3>\n",
    "\n",
    "**Hypothesis:**\n",
    "$$\\theta_0(x) = \\theta_0 + \\theta_1x$$\n",
    "\n",
    "**Parameters:**\n",
    "$$\\theta_0, \\theta_1$$\n",
    "\n",
    "**Cost Function**\n",
    "$$J(\\theta_0, \\theta_1) = \\frac{1}{m} \\sum_{i=1}^m (h_\\theta(x^{(i)})-y^{(i)})$$\n",
    "\n",
    "**Goal:**\n",
    "$$minimum_{\\theta_0,\\theta_1} J(\\theta_0,\\theta_1)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "064d618c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "099b9204",
   "metadata": {},
   "source": [
    "<h3><i>Question:</i> How do we find the <i>best</i> line? What do we mean by <i>best</i>?</h3>\n",
    "\n",
    "**Answer:**\n",
    "\n",
    "#### Define Loss Function:\n",
    ">Loss function should be a function of model parameters.\n",
    "\n",
    "- For input $x$, our model error is $e = y - \\hat y = y - \\hat f(x,\\theta) = y - \\theta_0 - \\theta^Tx$.\n",
    "- e is also termed as residual error as it is the differnce between observed value and predicted value.\n",
    "- **d=1**\n",
    "\n",
    "<img src=\"images/23.png\" align=\"center\">\n",
    "\n",
    "- For $\\mathcal{D} = \\{(x^{(i)}, y^{(i)}) \\mid i = 1,2,...,n\\} \\in X^{R} \\times Y$, we have\n",
    "$$e_i = y_i-\\theta_0-\\theta^T,\\;\\;\\;\\; i=1,2,3,4,.....,n$$\n",
    "\n",
    "- Using Residual error, we can define different loss functions:\n",
    "   $$L_{LSE}(\\theta_0,\\theta_1) = \\sum_{i=1}^n(y_i-\\theta_0-\\theta^Tx_i)^2$$\n",
    "   $$L_{MSE}(\\theta_0,\\theta_1) = {1/n} \\sum_{i=1}^n(y_i-\\theta_0-\\theta^Tx_i)^2$$\n",
    "   $$L_{MSE}(\\theta_0,\\theta_1) = \\sqrt {{1/n} \\sum_{i=1}^n(y_i-\\theta_0-\\theta^Tx_i)^2}$$\n",
    "   \n",
    "> One minimizer for all loss functions.\n",
    "\n",
    "- We minimize the following loss function:\n",
    "   $$L_{MSE}(\\theta_0,\\theta_1) = {1/n} \\sum_{i=1}^n(y_i-\\theta_0-\\theta^Tx_i)^2$$\n",
    "   \n",
    "- We have an **optimzation problem**: find the parameters which minimize the loss function. We write optimization problem (with no constraints) as \n",
    "   $$minimize_{\\theta_0,\\theta}\\;\\; L_{MSE}(\\theta_0,\\theta_1) = {1/n} \\sum_{i=1}^n(y_i-\\theta_0-\\theta^Tx_i)^2$$\n",
    "   \n",
    "   \n",
    "### How to solve?\n",
    "- **Analytically:** Determine a critical point that makes the derivtive(if it exists) equal to zero.\n",
    "- **Numerically:** Solve optimization using some algorithm that iteratively takes use closer to the critical point minimizing objective function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44c686f9",
   "metadata": {},
   "source": [
    "## Example-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "0144d5df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting gradient descent at b = 0, m = 0, error = 5565.107834483211\n",
      "Running...\n",
      "After 1000 iterations b = 0.08893651993741346, m = 1.4777440851894448, error = 112.61481011613473\n"
     ]
    }
   ],
   "source": [
    "from numpy import *\n",
    "\n",
    "# y = mx + b\n",
    "# m is slope, b is y-intercept\n",
    "def compute_error_for_line_given_points(b, m, points):\n",
    "    totalError = 0\n",
    "    for i in range(0, len(points)):\n",
    "        x = points[i, 0]\n",
    "        y = points[i, 1]\n",
    "        totalError += (y - (m * x + b)) ** 2\n",
    "    return totalError / float(len(points))\n",
    "\n",
    "def step_gradient(b_current, m_current, points, learningRate):\n",
    "    b_gradient = 0\n",
    "    m_gradient = 0\n",
    "    N = float(len(points))\n",
    "    for i in range(0, len(points)):\n",
    "        x = points[i, 0]\n",
    "        y = points[i, 1]\n",
    "        b_gradient += -(2/N) * (y - ((m_current * x) + b_current))\n",
    "        m_gradient += -(2/N) * x * (y - ((m_current * x) + b_current))\n",
    "    new_b = b_current - (learningRate * b_gradient)\n",
    "    new_m = m_current - (learningRate * m_gradient)\n",
    "    return [new_b, new_m]\n",
    "\n",
    "def gradient_descent_runner(points, starting_b, starting_m, learning_rate, num_iterations):\n",
    "    b = starting_b\n",
    "    m = starting_m\n",
    "    for i in range(num_iterations):\n",
    "        b, m = step_gradient(b, m, array(points), learning_rate)\n",
    "    return [b, m]\n",
    "\n",
    "def run():\n",
    "    points = genfromtxt(\"datasets/data.csv\", delimiter=\",\")\n",
    "    learning_rate = 0.0001\n",
    "    initial_b = 0 # initial y-intercept guess\n",
    "    initial_m = 0 # initial slope guess\n",
    "    num_iterations = 1000\n",
    "    print(\"Starting gradient descent at b = {0}, m = {1}, error = {2}\".format(initial_b, initial_m, compute_error_for_line_given_points(initial_b, initial_m, points)))\n",
    "    print(\"Running...\")\n",
    "    [b, m] = gradient_descent_runner(points, initial_b, initial_m, learning_rate, num_iterations)\n",
    "    print(\"After {0} iterations b = {1}, m = {2}, error = {3}\".format(num_iterations, b, m, compute_error_for_line_given_points(b, m, points)))\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    run()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc3a7a23",
   "metadata": {},
   "source": [
    "<img src=\"images/gradient_descent_example.gif\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fd1cd0f",
   "metadata": {},
   "source": [
    "### Example - 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "4c21d5f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0: Cost = 1.5427137296536395\n",
      "Iteration 10: Cost = 0.4055899449885261\n",
      "Iteration 20: Cost = 0.12820367641883942\n",
      "Iteration 30: Cost = 0.06040293217462261\n",
      "Iteration 40: Cost = 0.04382901547954687\n",
      "Iteration 50: Cost = 0.03977749736664156\n",
      "Iteration 60: Cost = 0.038787097644943355\n",
      "Iteration 70: Cost = 0.03854499293476682\n",
      "Iteration 80: Cost = 0.03848581007209732\n",
      "Iteration 90: Cost = 0.03847134273178296\n",
      "Final w: 1.9890302120562438\n",
      "Final b: 0.9697502088675694\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "# Generate some fake data for linear regression\n",
    "N = 100\n",
    "x = np.linspace(-1, 1, N)\n",
    "y = 2 * x + 1 + np.random.randn(N) * 0.2\n",
    "# Initialize weight and bias with random values\n",
    "w = np.random.randn()\n",
    "b = np.random.randn()\n",
    "# Set the learning rate\n",
    "alpha = 0.1\n",
    "# Set the number of iterations\n",
    "num_iterations = 100\n",
    "# Iterate through the gradient descent algorithm\n",
    "for i in range(num_iterations):\n",
    "    # Calculate the predicted values\n",
    "    y_pred = w * x + b\n",
    "    # Calculate the cost function\n",
    "    cost = 1/N * np.sum((y_pred - y) ** 2)\n",
    "    # Calculate the gradients\n",
    "    dw = 2/N * np.sum((y_pred - y) * x)\n",
    "    db = 2/N * np.sum(y_pred - y)\n",
    "    # Update the weights and biases\n",
    "    w = w - alpha * dw\n",
    "    b = b - alpha * db\n",
    "    # Print the cost every 10 iterations\n",
    "    if i % 10 == 0:\n",
    "        print(f'Iteration {i}: Cost = {cost}')\n",
    "# Print the final weights and biases\n",
    "print(f'Final w: {w}')\n",
    "print(f'Final b: {b}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5d6332",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Define Loss Function: Reformulation</h3>\n",
    "\n",
    "   $$L_{MSE}(\\theta_0,\\theta_1) = {1/n} \\sum_{i=1}^n(y_i-\\theta_0-\\theta^Tx_i)^2 = \\frac{1}{2}e^Te$$\n",
    "   \n",
    "**Explanation:**\n",
    "The transpose of a vector `v` is denoted by `v^T` and is defined as the `reflection` of `v` over the main diagonal of a matrix. For example, if `v` is a column vector:\n",
    "\n",
    "`\n",
    "v = [v1]\n",
    "    [v2]\n",
    "    [v3]\n",
    "`\n",
    "\n",
    "then the transpose of `v` is:\n",
    "\n",
    "`v^T = [v1 v2 v3]`\n",
    "\n",
    "The `dot product` of two vectors `u` and `v` is denoted by `u.v` and is defined as the sum of the products of the corresponding elements of the two vectors. For example, if u and v are column vectors:\n",
    "\n",
    "`\n",
    "u = [u1]\n",
    "    [u2]\n",
    "    [u3]\n",
    "v = [v1]\n",
    "    [v2]\n",
    "    [v3]`\n",
    "    \n",
    "then the dot product of u and v is:\n",
    "\n",
    "`u.v = u1*v1 + u2*v2 + u3*v3`\n",
    "\n",
    "Now, to prove that `e^T.e = n`, where `e` is an n-dimensional column vector with all elements equal to `1` and `n` is the number of elements in `e`, we can use the definition of the dot product:\n",
    "\n",
    "`e^T.e = e1*e1 + e2*e2 + ... + en*en\n",
    "       = 1*1 + 1*1 + ... + 1*1\n",
    "       = n`\n",
    "       \n",
    "Therefore, `e^T.e = n.`\n",
    "\n",
    "Note that this result holds for any n-dimensional column vector e with all elements equal to 1.\n",
    "\n",
    "\n",
    "   $$L_{MSE}(\\theta_0,\\theta_1) = {1/n} \\sum_{i=1}^n(y_i-\\theta_0-\\theta^Tx_i)^2 = \\frac{1}{2}e^Te$$\n",
    "  \n",
    "Here $e=[e_1,e_2,....,e_n]^T$ (column vector) where\n",
    "$$e_i = y_i-\\theta_0-\\theta^T,\\;\\;\\;\\; i=1,2,3,4,.....,n$$\n",
    "\n",
    "<img src=\"images/p26.png\" align=\"left\">\n",
    "<img src=\"images/p24.png\" align=\"right\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95951fa1",
   "metadata": {},
   "source": [
    "**Consequently:** \n",
    "\n",
    "\n",
    "Recall that we may fit a linear model by choosing $\\theta$ that minimizes the squared error:\n",
    "$$J(\\theta_0,\\theta)=\\frac{1}{2}\\sum_{i=1}^n(y_i-\\theta_0-\\theta^\\top x_i)^2 = \\frac{1}{2}e^Te$$\n",
    "We can write this sum in matrix-vector form as:\n",
    "$$J(\\theta_0,\\theta)=J(w) = \\frac{1}{2} (y-Xw)^\\top(y-Xw) = \\frac{1}{2} \\|y-Xw\\|^2,$$\n",
    "where $X$ is the design matrix and $\\|\\cdot\\|$ denotes the Euclidean norm."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a1e7e6",
   "metadata": {},
   "source": [
    "### Solve Optimization Problem: (Analytical Solution employing Calculus)\n",
    "- very beautiful, elegant function we have here!\n",
    "\n",
    "We first write the loss function as\n",
    "<img src=\"images/p27.png\" align=\"center\">\n",
    "\n",
    "-  To further solve this, let us quickly talk about the concept of a gradient of a function.\n",
    "\n",
    "#### Gradient of a function: Overview\n",
    "- For a function $f(x)$ that maps $x \\; \\epsilon \\; R^d$ to $R$, we define a gradient (directional derivative) with respect to $x$ as.\n",
    "<img src=\"images/p28.png\" align=\"center\">\n",
    "\n",
    "- Derivative quantifies the rate of change along different directions.\n",
    "\n",
    "**Question:** Calculate $\\nabla$ of following functions.\n",
    "- $f(x) = a^Tx = x^Ta$\n",
    "- $f(x) = x^Tx$\n",
    "- $f(x) = x^TPx$\n",
    "\n",
    "\n",
    "\n",
    "**We have a loss function:**\n",
    "$$L(w) = \\frac{1}{2}(y^Ty - 2w^TX^Ty + w^TX^TXw)$$\n",
    "\n",
    "- Take gradient with respect to $w$ as\n",
    "\\begin{align*}\n",
    "\\nabla_w J(w) \n",
    "& = \\nabla_w \\frac{1}{2} (X w - y)^\\top  (X w - y) \\\\\n",
    "& = \\frac{1}{2} \\nabla_w \\left( (Xw)^\\top  (X w) - (X w)^\\top y - y^\\top (X w) + y^\\top y \\right) \\\\\n",
    "& = \\frac{1}{2} \\nabla_w \\left( w^\\top  (X^\\top X) w - 2(X w)^\\top y \\right) \\\\\n",
    "& = \\frac{1}{2} \\left( 2(X^\\top X) w - 2X^\\top y \\right) \\\\\n",
    "& = (X^\\top X) w - X^\\top y\n",
    "\\end{align*}\n",
    "\n",
    "We used the facts that $a^\\top b = b^\\top a$ (line 3), that $\\nabla_x b^\\top x = b$ (line 4), and that $\\nabla_x x^\\top A x = 2 A x$ for a symmetric matrix $A$ (line 4).\n",
    "\n",
    "> We know from calculus that a function is minimized when its derivative is set to zero. In our case, our objective function is a (multivariate) quadratic; hence it only has one minimum, which is the global minimum.\n",
    "\n",
    "- Setting the above derivative to zero, we obtain the *normal equations*:\n",
    "$$ (X^\\top X) w = X^\\top y.$$\n",
    "\n",
    "Hence, the value $w^*$ that minimizes this objective is given by:\n",
    "$$ w^* = (X^\\top X)^{-1} X^\\top y.$$\n",
    "\n",
    "\n",
    "- We have determined the weights for which LSE,MSE,RMSE or the norm of the residual is minimized.\n",
    "- This solution is referred to as least-squared solution as it minimizes the squared error."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e747c30",
   "metadata": {},
   "source": [
    "<h2 align=\"center\">Gradient Descent Algorihtm</h2>\n",
    "\n",
    "**Goal:**   Minimize cost function $J(\\theta_0, \\theta_1)$.\n",
    "\n",
    "**Definition:** Used all over machine learning for minimization.\n",
    "\n",
    "**Problem:**\n",
    "- We have $J(\\theta_0, \\theta_1)$.\n",
    "- We want to get $min\\;J(\\theta_0, \\theta_1)$.\n",
    "\n",
    "**Solution**:\n",
    "- Start with some $J(\\theta_0,\\theta_1)$. For example $J(0,0)$.\n",
    "\n",
    "\n",
    "#### How does it work?\n",
    "- Start with initial guesse\n",
    "    - Start at 0,0 (or any other value)\n",
    "    - Keeping changing $\\theta_0$ and $\\theta_1$ a little bit to try and reduce $J(\\theta_0,\\theta_1)$.\n",
    "- Each time you change the parameters, you select the gradient which reduces J(θ0,θ1) the most possible \n",
    "- Repeat\n",
    "- Do so until you converge to a local minimum\n",
    "- Has an interesting property\n",
    "    - Where you start can determine which minimum you end up\n",
    "    - Here we can see one initialization point led to one local minimum\n",
    "    - The other led to a different one\n",
    "\n",
    "<img src=\"images/p29.png\" align=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5993f004",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">A simplified version of gradient descent</h3>\n",
    "\n",
    "Assume again that we set $\\theta_0 = 0$ and our hypothesis and cost function practically have only one coefficient, $\\theta_1$.\n",
    "   $$h_\\theta(x) = \\theta_1x$$\n",
    "\n",
    "repeat until convergence{"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "97a964c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The minimum value of the function is 8.3156 at x = 3.8375\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Define the function to be minimized\n",
    "def f(x):\n",
    "    return x**2 + 10*np.sin(x)\n",
    "\n",
    "# Define the gradient of the function\n",
    "def grad_f(x):\n",
    "    return 2*x + 10*np.cos(x)\n",
    "\n",
    "# Choose the step size (learning rate)\n",
    "alpha = 0.1\n",
    "\n",
    "# Set the initial value of x\n",
    "x = 5\n",
    "\n",
    "# Set the tolerance for the convergence criterion\n",
    "tol = 1e-6\n",
    "\n",
    "# Initialize a list to store the values of x at each iteration\n",
    "x_values = [x]\n",
    "\n",
    "# Iterate until convergence\n",
    "while True:\n",
    "    # Compute the gradient at the current value of x\n",
    "    grad = grad_f(x)\n",
    "    \n",
    "    # Update the value of x using gradient descent\n",
    "    x = x - alpha * grad\n",
    "    \n",
    "    # Store the new value of x\n",
    "    x_values.append(x)\n",
    "    \n",
    "    # Check for convergence\n",
    "    if np.abs(grad) < tol:\n",
    "        break\n",
    "\n",
    "# Print the minimum value found\n",
    "print(f\"The minimum value of the function is {f(x):.4f} at x = {x:.4f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5298b7e5",
   "metadata": {},
   "source": [
    "### Batch gradient descent\n",
    "Batch gradient descent is an algorithm used to minimize a loss function in a model with multiple parameters, such as linear regression. In linear regression, we aim to find the values of the weights that minimize the mean squared error (MSE) between the predicted output and the true output.\n",
    "\n",
    "The MSE for a single sample is given by:\n",
    "\n",
    "$$ MSE(\\mathbf{w}) = \\frac{1}{2}(\\mathbf{x}^T \\mathbf{w} - y)^2 $$\n",
    "\n",
    "Where $\\mathbf{w}$ is the vector of weights, $\\mathbf{x}$ is the input feature vector, and $y$ is the true output.\n",
    "\n",
    "To find the weights that minimize the MSE for a dataset with $n$ samples, we can take the mean of the MSE for each sample:\n",
    "\n",
    "$$ J(\\mathbf{w}) = \\frac{1}{n} \\sum_{i=1}^n MSE(\\mathbf{w}) = \\frac{1}{n} \\sum_{i=1}^n \\frac{1}{2}(\\mathbf{x}_i^T \\mathbf{w} - y_i)^2 $$\n",
    "\n",
    "Where $J(\\mathbf{w})$ is the mean squared error loss function.\n",
    "\n",
    "To minimize this loss function, we can use gradient descent. In gradient descent, we iteratively update the weights in the opposite direction of the gradient of the loss function. The gradient of the loss function with respect to the weights is given by:\n",
    "\n",
    "$$ \\nabla_{\\mathbf{w}} J(\\mathbf{w}) = \\frac{1}{n} \\sum_{i=1}^n \\nabla_{\\mathbf{w}} MSE(\\mathbf{w}) = \\frac{1}{n} \\sum_{i=1}^n \\nabla_{\\mathbf{w}} \\frac{1}{2}(\\mathbf{x}_i^T \\mathbf{w} - y_i)^2 $$\n",
    "\n",
    "Using the chain rule, we can expand the gradient as:\n",
    "$$ \\nabla_{\\mathbf{w}} J(\\mathbf{w}) = \\frac{1}{n} \\sum_{i=1}^n (\\mathbf{x}_i^T \\mathbf{w} - y_i) \\mathbf{x}_i $$\n",
    "\n",
    "We can then update the weights in the opposite direction of the gradient using the learning rate $\\eta$:\n",
    "\n",
    "$$ \\mathbf{w} = \\mathbf{w} - \\eta \\nabla_{\\mathbf{w}} J(\\mathbf{w}) = \\mathbf{w} - \\eta \\frac{1}{n} \\sum_{i=1}^n (\\mathbf{x}_i^T \\mathbf{w} - y_i) \\mathbf{x}_i $$\n",
    "\n",
    "This process is repeated until the loss function reaches a minimum or the change in the weights is small enough."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acdd18cb",
   "metadata": {},
   "source": [
    "### Batch gradient descent algorithm pseudocode\n",
    "Here is a pseudocode for the batch gradient descent algorithm for minimizing a loss function $J(\\mathbf{w})$ with respect to the parameters $\\mathbf{w}$:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "000b974d",
   "metadata": {},
   "outputs": [],
   "source": [
    "input: learning rate eta, number of iterations n_iter\n",
    "\n",
    "# Initialize the weights w\n",
    "w = random initialization\n",
    "\n",
    "for iteration in range(n_iter):\n",
    "    # Calculate the gradient of the loss function with respect to the weights\n",
    "    gradient = gradient(J, w)\n",
    "    \n",
    "    # Update the weights in the opposite direction of the gradient\n",
    "    w = w - eta * gradient\n",
    "\n",
    "output: w"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bdfdd20",
   "metadata": {},
   "source": [
    "This pseudocode assumes that the function gradient(J, w) calculates the gradient of the loss function $J$ with respect to the weights $\\mathbf{w}$, and returns it as a vector. The batch gradient descent algorithm iteratively updates the weights using the entire dataset, until the loss function reaches a minimum or the change in the weights is small enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c68a0118",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f8631fd0",
   "metadata": {},
   "source": [
    "### Stochastic gradient descent\n",
    "Stochastic gradient descent (SGD) is an optimization algorithm used to minimize a loss function in a model with multiple parameters, such as linear regression. Like batch gradient descent, SGD is an iterative algorithm that updates the model's parameters in the opposite direction of the gradient of the loss function. However, instead of calculating the gradient using the entire dataset, as in batch gradient descent, SGD calculates the gradient using a single sample at a time.\n",
    "\n",
    "In linear regression, we aim to find the values of the weights that minimize the mean squared error (MSE) between the predicted output and the true output. The MSE for a single sample is given by:\n",
    "\n",
    "$$ MSE(\\mathbf{w}) = \\frac{1}{2}(\\mathbf{x}^T \\mathbf{w} - y)^2 $$\n",
    "\n",
    "Where $\\mathbf{w}$ is the vector of weights, $\\mathbf{x}$ is the input feature vector, and $y$ is the true output.\n",
    "\n",
    "To minimize the MSE for a dataset with $n$ samples, we can use SGD to iteratively update the weights. In each iteration, we randomly select a sample from the dataset and calculate the gradient of the MSE with respect to the weights using only that sample:\n",
    "\n",
    "$$ \\nabla_{\\mathbf{w}} MSE(\\mathbf{w}) = (\\mathbf{x}^T \\mathbf{w} - y) \\mathbf{x} $$\n",
    "\n",
    "We can then update the weights in the opposite direction of the gradient using the learning rate $\\eta$:\n",
    "\n",
    "$$ \\mathbf{w} = \\mathbf{w} - \\eta \\nabla_{\\mathbf{w}} MSE(\\mathbf{w}) = \\mathbf{w} - \\eta (\\mathbf{x}^T \\mathbf{w} - y) \\mathbf{x} $$\n",
    "\n",
    "This process is repeated for a number of iterations, using a different sample in each iteration.\n",
    "One advantage of SGD over batch gradient descent is that it can make faster progress towards a minimum of the loss function, since it processes samples one at a time and updates the weights more frequently. However, the updates made by SGD can also be more erratic and have higher variance, which can make the optimization process more noisy and may lead to suboptimal solutions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "357bf7fc",
   "metadata": {},
   "source": [
    "### Stochastic gradient descent  algorithm pseudocode\n",
    "\n",
    "Here is a pseudocode for the stochastic gradient descent (SGD) algorithm for minimizing a loss function $J(\\mathbf{w})$ with respect to the parameters $\\mathbf{w}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c76e305d",
   "metadata": {},
   "outputs": [],
   "source": [
    "input: learning rate eta, number of iterations n_iter\n",
    "\n",
    "# Initialize the weights w\n",
    "w = random initialization\n",
    "\n",
    "for iteration in range(n_iter):\n",
    "    # Shuffle the data at the beginning of the epoch\n",
    "    shuffled_indices = shuffle(data)\n",
    "    X_shuffled = X[shuffled_indices]\n",
    "    y_shuffled = y[shuffled_indices]\n",
    "    \n",
    "    for i in range(n):\n",
    "        # Select a sample from the data\n",
    "        x_i = X_shuffled[i]\n",
    "        y_i = y_shuffled[i]\n",
    "        \n",
    "        # Calculate the gradient of the loss function with respect to the weights\n",
    "        gradient = gradient(J, w, x_i, y_i)\n",
    "        \n",
    "        # Update the weights in the opposite direction of the gradient\n",
    "        w = w - eta * gradient\n",
    "output: w\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bf4d6bd",
   "metadata": {},
   "source": [
    "This pseudocode assumes that the function gradient(J, w, x, y) calculates the gradient of the loss function $J$ with respect to the weights $\\mathbf{w}$ using the sample (x, y), and returns it as a vector. It also assumes that the function shuffle(data) shuffles the data and returns a list of shuffled indices. The SGD algorithm iteratively updates the weights using a single sample at a time, until the loss function reaches a minimum or the change in the weights is small enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ab3cde3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "id": "3bacfc1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4.1699862 ]\n",
      " [2.96811741]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Generate some synthetic data\n",
    "np.random.seed(42)\n",
    "X = 2 * np.random.rand(100, 1)\n",
    "y = 4 + 3 * X + np.random.randn(100, 1)\n",
    "\n",
    "# Add a column of ones to X to represent the bias term\n",
    "X_b = np.c_[np.ones((100, 1)), X]\n",
    "\n",
    "# Initialize the weights randomly\n",
    "w = np.random.randn(2, 1)\n",
    "\n",
    "# Set the learning rate and the number of iterations\n",
    "eta = 0.1\n",
    "n_iter = 50\n",
    "\n",
    "\n",
    "# Perform SGD for n_iter iterations\n",
    "for iteration in range(n_iter):\n",
    "    for i in range(len(X_b)):\n",
    "        # Select a sample from the data\n",
    "        x_i = X_b[i:i+1]\n",
    "        y_i = y[i:i+1]\n",
    "        \n",
    "        # Calculate the error\n",
    "        y_pred = x_i.dot(w)\n",
    "        error = y_i - y_pred\n",
    "        \n",
    "        # Calculate the gradient of the error with respect to the weights\n",
    "        gradient = -2 * x_i.T.dot(error)\n",
    "        \n",
    "        # Update the weights\n",
    "        w = w - eta * gradient\n",
    "print(w)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efbf6e76",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "25216510",
   "metadata": {},
   "source": [
    "Here's a brief explanation of what's happening in the code:\n",
    "\n",
    "- We generate some synthetic data with a linear relationship and add some noise.\n",
    "- We add a column of ones to the feature matrix X to represent the bias term.\n",
    "- We initialize the weights randomly.\n",
    "- We set the learning rate `eta` and the number of iterations `n_iter`.\n",
    "- We perform SGD for n_iter iterations. In each iteration, we:\n",
    "    - Select a sample from the data.\n",
    "    - Calculate the error between the predicted value and the true value for the sample.\n",
    "    - Calculate the gradient of the error with respect to the weights using the sample.\n",
    "    - Update the weights using the gradient and the learning rate.\n",
    "- Finally, we print the weights, which should be close to the true weights of the synthetic data we generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc16dbc4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "81f2449e",
   "metadata": {},
   "source": [
    "### Mini-batch gradient descent\n",
    "Mini-batch gradient descent is an optimization algorithm used to minimize a loss function in a model with multiple parameters, such as linear regression. Like batch gradient descent, mini-batch gradient descent calculates the gradient of the loss function using the entire dataset. However, instead of updating the model's parameters using the entire dataset, as in batch gradient descent, mini-batch gradient descent divides the dataset into smaller \"mini-batches\" and updates the model's parameters using a mini-batch of samples at a time.\n",
    "\n",
    "In linear regression, we aim to find the values of the weights that minimize the mean squared error (MSE) between the predicted output and the true output. The MSE for a single sample is given by:\n",
    "\n",
    "$$ MSE(\\mathbf{w}) = \\frac{1}{2}(\\mathbf{x}^T \\mathbf{w} - y)^2 $$\n",
    "\n",
    "Where $\\mathbf{w}$ is the vector of weights, $\\mathbf{x}$ is the input feature vector, and $y$ is the true output.\n",
    "\n",
    "To minimize the MSE for a dataset with $n$ samples, we can use mini-batch gradient descent to iteratively update the weights. In each iteration, we divide the dataset into mini-batches of size $m$, and calculate the gradient of the MSE with respect to the weights using the samples in a single mini-batch:\n",
    "$$ \\nabla_{\\mathbf{w}} MSE(\\mathbf{w}) = \\frac{1}{m} \\sum_{i=1}^m (\\mathbf{x}_i^T \\mathbf{w} - y_i) \\mathbf{x}_i $$\n",
    "\n",
    "Where the sum is over the $m$ samples in the mini-batch.\n",
    "\n",
    "We can then update the weights in the opposite direction of the gradient using the learning rate $\\eta$:\n",
    "\n",
    "$$ \\mathbf{w} = \\mathbf{w} - \\eta \\nabla_{\\mathbf{w}} MSE(\\mathbf{w}) = \\mathbf{w} - \\eta \\frac{1}{m} \\sum_{i=1}^m (\\mathbf{x}_i^T \\mathbf{w} - y_i) \\mathbf{x}_i $$\n",
    "\n",
    "This process is repeated for a number of iterations, using a different mini-batch of samples in each iteration.\n",
    "\n",
    "One advantage of mini-batch gradient descent over batch gradient descent is that it can make faster progress towards a minimum of the loss function, since it processes samples in smaller batches and updates the weights more frequently. However, it still requires the storage and processing of the entire dataset, as opposed to stochastic gradient descent which only requires the storage and processing of a single sample at a time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5b81947",
   "metadata": {},
   "source": [
    "### Mini-Batch gradient descent algorithm pseudocode\n",
    "Here is a pseudocode for the mini-batch gradient descent algorithm for minimizing a loss function $J(\\mathbf{w})$ with respect to the parameters $\\mathbf{w}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "944ba601",
   "metadata": {},
   "outputs": [],
   "source": [
    "input: learning rate eta, number of iterations n_iter, mini-batch size m\n",
    "\n",
    "# Initialize the weights w\n",
    "w = random initialization\n",
    "\n",
    "for iteration in range(n_iter):\n",
    "    # Shuffle the data at the beginning of the epoch\n",
    "    shuffled_indices = shuffle(data)\n",
    "    X_shuffled = X[shuffled_indices]\n",
    "    y_shuffled = y[shuffled_indices]\n",
    "    \n",
    "    # Split the data into mini-batches\n",
    "    mini_batches = [(X_shuffled[i:i+m], y_shuffled[i:i+m]) for i in range(0, n, m)]\n",
    "    \n",
    "    # Update the weights for each mini-batch\n",
    "    for mini_batch in mini_batches:\n",
    "        X_mini, y_mini = mini_batch\n",
    "        # Calculate the gradient of the loss function with respect to the weights\n",
    "        gradient = gradient(J, w, X_mini, y_mini)\n",
    "        \n",
    "        # Update the weights in the opposite direction of the gradient\n",
    "        w = w - eta * gradient\n",
    "output: w"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ad34316",
   "metadata": {},
   "source": [
    "This pseudocode assumes that the function gradient(J, w, X, y) calculates the gradient of the loss function $J$ with respect to the weights $\\mathbf{w}$ using the samples in the mini-batch (X, y), and returns it as a vector. It also assumes that the function shuffle(data) shuffles the data and returns a list of shuffled indices. The mini-batch gradient descent algorithm iteratively updates the weights using small batches of samples from the dataset, until the loss function reaches a minimum or the change in the weights is small enough."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab50b424",
   "metadata": {},
   "source": [
    "### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbf5b425",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# Generate some synthetic data\n",
    "np.random.seed(42)\n",
    "X = 2 * np.random.rand(100, 1)\n",
    "y = 4 + 3 * X + np.random.randn(100, 1)\n",
    "\n",
    "# Add a column of ones to X to represent the bias term\n",
    "X_b = np.c_[np.ones((100, 1)), X]\n",
    "\n",
    "# Initialize the weights randomly\n",
    "w = np.random.randn(2, 1)\n",
    "\n",
    "# Set the learning rate and the number of samples per mini-batch\n",
    "eta = 0.1\n",
    "batch_size = 20\n",
    "\n",
    "# Perform mini-batch gradient descent for 50 iterations\n",
    "for iteration in range(50):\n",
    "    # Shuffle the data at the beginning of each epoch\n",
    "    shuffled_indices = np.random.permutation(len(X_b))\n",
    "    X_b_shuffled = X_b[shuffled_indices]\n",
    "    y_shuffled = y[shuffled_indices]\n",
    "    \n",
    "    # Split the data into mini-batches\n",
    "    mini_batches = [(X_b_shuffled[i:i+batch_size], y_shuffled[i:i+batch_size])\n",
    "                    for i in range(0, len(X_b_shuffled), batch_size)]\n",
    "    \n",
    "    # Update the weights for each mini-batch\n",
    "    for mini_batch in mini_batches:\n",
    "        X_mini, y_mini = mini_batch\n",
    "        # Calculate the error\n",
    "        y_pred = X_mini.dot(w)\n",
    "        error = y_mini - y_pred\n",
    "        \n",
    "        # Calculate the gradient of the error with respect to the weights\n",
    "        gradient = -2 * X_mini.T.dot(error) / len(X_mini)\n",
    "        \n",
    "        # Update the weights\n",
    "        w = w - eta * gradient\n",
    "print(w)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "399d323a",
   "metadata": {},
   "source": [
    "Here's a brief explanation of what's happening in the code:\n",
    "\n",
    "- We generate some synthetic data with a linear relationship and add some noise.\n",
    "- We add a column of ones to the feature matrix X to represent the bias term.\n",
    "- We initialize the weights randomly.\n",
    "- We set the learning rate eta and the number of samples per mini-batch batch_size.\n",
    "- We perform mini-batch gradient descent for 50 iterations. In each iteration, we:\n",
    "    - Shuffle the data at the beginning of the epoch.\n",
    "    - Split the data into mini-batches.\n",
    "    - For each mini-batch, we:\n",
    "        - Calculate the error between the predicted values and the true values.\n",
    "        - Calculate the gradient of the error with respect to the weights.\n",
    "        - Update the weights using the gradient and the learning rate.\n",
    "- Finally, we print the weights, which should be close to the true weights of the synthetic data we generated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33b394f3",
   "metadata": {},
   "source": [
    "### Two extension to the algorithm\n",
    "#### 1. Normal equation for numeric solution\n",
    "To solve the minimization problem we can solve it $[ min J(\\theta_0, \\theta_1) ]$ exactly using a numeric method which avoids the iterative approach used by gradient descent. Normal equations method has advantages and disadvantages.\n",
    "\n",
    "**Advantage:**\n",
    "- No longer an alpha term\n",
    "- Can be much faster for some problems\n",
    "\n",
    "**Disadvantage:**\n",
    "- Much more complicated\n",
    "\n",
    "We discuss the normal equation in the linear regression with multiple features section\n",
    "#### 2. We can learn with a larger number of features\n",
    "- There are many parameters which contribute towards a price e.g. with houses (Size, Age, Number bedrooms, Number floors).\n",
    "- With multiple features becomes hard to plot can't really plot in more than 3 dimensions. Notation becomes more complicated too\n",
    "    - Best way to get around with this is the notation of linear algebra\n",
    "    - Gives notation and set of things you can do with matrices and vectors e.g. Matrix\n",
    "- We see here this matrix shows us (Size, Number of bedrooms, Number floors, Age of home). All in one variable Block of numbers, take all data organized into one big block. Vector shown as y, shows us the prices.\n",
    "\n",
    "<img src=\"images/p30.png\" align=\"left\">\n",
    "<img src=\"images/p31.png\" align=\"center\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d28872c",
   "metadata": {},
   "source": [
    "<h2>Linear regression with multiple features(Multivariate)</h2>\n",
    "<img src=\"images/p32.png\">\n",
    "\n",
    "**Hypothesis:**\n",
    "$$h_\\theta (X) = \\theta_0 + \\theta_1 x_1 + \\theta_2x_2 + \\theta_3x_3 + ⋯ + \\theta_nx_n$$\n",
    "\n",
    "**Notation:**\n",
    "\n",
    "- **m** = Number of training samples\n",
    "- **n** = Number of features\n",
    "- **x** = Feature\n",
    "- **y** = Label\n",
    "- $(x_{j}^{(i)}, y_{j}^{(i)})$: the jth feature of the ith sample in the dataset\n",
    "\n",
    "<h4 align=\"center\">Hypothesis</h4>\n",
    "\n",
    "**Previously:**\n",
    "$$h_\\theta(x) = \\theta_0 + \\theta_1x$$\n",
    "\n",
    "**Now:**\n",
    "$$h_\\theta (x) = \\theta_0 + \\theta_1 x_1 + \\theta_2x_2 + \\theta_3x_3 + ⋯ + \\theta_nx_n$$\n",
    "\n",
    "where, $X = [x_1, x_2 ... x_n]$ is the n-dimensional feature vector, and $\\theta = [\\theta_0, \\theta_1 ... \\theta_n]$ is an (n+1)-dimensional vector of weights.\n",
    "\n",
    "To make this more uniform, assume $x_0 = 1$ to get:\n",
    "$$h_\\theta (X) = \\theta_0x_0 + \\theta_1 x_1 + \\theta_2x_2 + \\theta_3x_3 + ⋯ + \\theta_nx_n$$\n",
    "\n",
    "where, $X = [x_0,x_1, x_2 ... x_n]$ is the (n+1)-dimensional feature vector, and $\\theta = [\\theta_0, \\theta_1 ... \\theta_n]$ is an (n+1)-dimensional vector of weights.\n",
    "\n",
    "<h4 align=\"center\">Geometric Interpretation</h4>\n",
    "$$h_\\theta (X) = \\theta_0x_0 + \\theta_1 x_1 + \\theta_2x_2 $$ (a 2-D hyperplane in 3-D)\n",
    "<img src=\"images/p33.gif\" align=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea732cc6",
   "metadata": {},
   "source": [
    "<h4 align=\"center\">Vectorizing the notation</h4>\n",
    "\n",
    "$$h_\\theta (x) = \\theta_0x_0 + \\theta_1 x_1 + \\theta_2x_2 + \\theta_3x_3 + ⋯ + \\theta_nx_n$$.\n",
    "\n",
    "Now we can redefine our hypothesis as:\n",
    "\n",
    "$$\\theta= \\begin{bmatrix} \\theta_0 \\\\ \\theta_1 \\\\ .. \\\\ \\theta_n \\end{bmatrix} \\;\\; X = \\begin{bmatrix} x_0 \\\\ x_1 \\\\ .. \\\\ x_n \\end{bmatrix}$$ and where $\\theta \\; \\epsilon \\; R ^ {n+1}$ and $X \\; \\epsilon \\; R ^ {n+1}$ and ,\n",
    "$$h_\\theta(X) = \\theta^TX$$\n",
    "This is represents **Multivariate Linear Regression**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6794dd14",
   "metadata": {},
   "source": [
    "<h4 align=\"center\">For m-training instances<h4>\n",
    "\n",
    "$$h_\\theta (X) = \\theta_0x_0 + \\theta_1 x_1 + \\theta_2x_2 + \\theta_3x_3 + ⋯ + \\theta_nx_n$$.\n",
    "\n",
    "<img src=\"images/p33.png\" align=\"center\">\n",
    "\n",
    "$h_\\theta(X) = \\theta^TX = ?$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e900a9be",
   "metadata": {},
   "source": [
    "### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "19d21605",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speed</th>\n",
       "      <th>dist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   speed  dist\n",
       "0      4     2\n",
       "1      4    10"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "# To b#uild linear regression we will use the classic cars data from cmdlinetips.com‘s github page.\n",
    "data_url = 'https://raw.githubusercontent.com/cmdlinetips/data/master/cars.tsv'\n",
    "cars = pd.read_csv(data_url, sep=\"\\t\")\n",
    "cars.head(2)\n",
    "# cars dataset contains distance needed for cars at different speeds to stop from 1920 cars. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "3dc74177",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXDElEQVR4nO3df5Bdd3nf8fezWCAsyfKvtazaOEKJJy7yYOHuuJBAxoFOMIRgkWZcKNOQNjOmHVKbmBZMoA2d8gcdiB06SZzaQDEZYnDBGIcQF8dlxs0U3Kxc2cg2BNsRMa4sre1Yls2okbxP/7hnl/VKuzp3955zz4/3a2ZHe8+ee+7z3bP6aPW95znfyEwkSf0xMe4CJEn1MvglqWcMfknqGYNfknrG4Jeknjlh3AWUcfrpp+eWLVvGXYYktcrOnTufyMzJxdtbEfxbtmxhenp63GVIUqtExA+Otd2pHknqGYNfknrG4JeknjH4JalnDH5J6pnKgj8iXhYR34yIByLi/oi4stj+kYh4LCJ2FR9vrqoGSaM1O5s8MvMs33r4CR6ZeZbZWW/y2EZVXs55BHhfZt4TERuAnRFxR/G1azPzExW+tqQRm51Nbr//ca66eReHDs+yds0E11y2nUu2ncnERIy7PA2hst/4M3NvZt5TfH4QeBA4q6rXk1StPU8+Nx/6AIcOz3LVzbvY8+RzY65Mw6pljj8itgCvAu4uNv1GRNwXEZ+JiFOWeM7lETEdEdMzMzN1lClpGfueOTQf+nMOHZ5l/8FDY6pIK1V58EfEeuDLwHsz8xngOuAnge3AXuB3jvW8zLw+M6cyc2py8qiOY0k123TSWtaueWFkrF0zwRkb1o6pIq1UpcEfEWsYhP7nM/MWgMzcl5nPZ+YscANwUZU1SBqNLaet45rLts+H/9wc/5bT1o25Mg2rsjd3IyKATwMPZuY1C7Zvzsy9xcO3AburqkHS6ExMBJdsO5Pzrngd+w8e4owNa9ly2jrf2G2hKq/q+VngnwHfiYhdxbbfAt4REduBBPYA766wBkkjNDERbJ1cz9bJ9eMuRatQWfBn5l8Ax/pV4OtVvaYk6fjs3JWknmnF/filvpmdTfY8+Rz7njnEppO6N5e+cHybN67l+VnYf7CbY12pKn8GDH6pYbreIbtwfKec+GJ+9TU/wSfv/H4nx7pSVf8MONUjNUzXO2QXju+XLzx7PvShe2Ndqap/Bgx+qWG63iG7cHwRdHqsK1X1z4DBLzVM1ztkF4+vy2Ndqap/Bgx+qWG63iG7cHxf3vlDrnzDuZ0d60pV/TMQmc2/n/bU1FROT0+PuwypNnNXdHS1Q3bh+M48aXBVz8yz3RzrSo3iZyAidmbm1FHbDX5J6qalgt+pHknqGa/jV690vTFKKsPgV290vTFKKsupHvVG1xujpLIMfvVG1xujpLIMfvVG1xujpLIMfvVG1xujpLJ8c1e94dKB0oDBr15x6UDJqR5J6h1/45dqYvOYmsLgl2pg85iaxKkeqQY2j6lJDH6pBjaPqUkMfqkGNo+pSQx+qQY2j6lJfHNXqoHNY2oSg1+qic1jagqneiSpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6prJ79UTEy4DPAZuABK7PzE9GxKnAF4EtwB7gssz826rq0HDasDxg2RpHvV8VNUrjUOVN2o4A78vMeyJiA7AzIu4Afg24MzM/FhFXA1cDH6iwDpXUhuUBy9Y46v2qqFEal8qmejJzb2beU3x+EHgQOAu4FLix2O1GYEdVNWg4bVgesGyNo96vihqlcalljj8itgCvAu4GNmXm3uJLjzOYCjrWcy6PiOmImJ6ZmamjzN5rw/KAZWsc9X5V1CiNS+XBHxHrgS8D783MZxZ+LTOTwfz/UTLz+sycysypycnJqssU7VgesGyNo96vihqlcak0+CNiDYPQ/3xm3lJs3hcRm4uvbwb2V1mDymvD8oBlaxz1flXUKI1LDH7pruDAEcFgDv+pzHzvgu0fB55c8ObuqZn5/uWONTU1ldPT05XUqReauxqlycsDlq1x1PtVUaNUpYjYmZlTR22vMPhfC/xP4DvA3ITnbzGY578ZOAf4AYPLOZ9a7lgGvyQNb6ngr+xyzsz8C2CpX3HeUNXrSpKWZ+euJPVMlQ1c6rC6OlO73gHb9fGpmQx+Da2uztSud8B2fXxqLqd6NLS6OlO73gHb9fGpuQx+Da2uztSud8B2fXxqLoNfQ6urM7XrHbBdH5+ay+DX0OrqTO16B2zXx6fmqqyBa5Rs4GqeujpTu94B2/Xxabxqb+BSt01MBFsn17N1cn0nXmdcuj4+NZNTPZLUM/7Gr9o1rWmpafWslktJ6ngMftWqaU1LTatntVxKUmU41aNaNa1pqWn1rJZLSaoMg1+1alrTUtPqWS2XklQZBr9q1bSmpabVs1ouJakyDH7VqmlNS02rZ7VcSlJl2MCl2jWtaalp9ayWS0lqTu1LL46SwS9Jw1sq+J3qkaSeMfglqWds4NILjLpD0yUapeYx+DVv1B2aLtEoNZNTPZo36g5Nl2iUmsng17xRd2i6RKPUTAa/5o26Q9MlGqVmMvg1b9Qdmi7RKDWTDVx6gVF3aLpEozQ+Lr2oUka9FKBLNErN41SPJPWMv/HrBdrawCWpPINf89rawCVpOE71aF5bG7gkDcfg17y2NnBJGo7Br3ltbeCSNJxlgz8iLlzuo64iVY+2NnBJGs6yDVwR8c3i07XAFHAvEMArgenMfE3lFWIDV53a2sAl6WgrauDKzJ8vnnwLcGFmfqd4fD7wkQrq1Ji1tYFLUnll5/h/ei70ATJzN/D3qylJklSlssF/X0R8KiIuLj5uAO5b7gkR8ZmI2B8Ruxds+0hEPBYRu4qPN6+m+K6anU0emXmWbz38BI/MPMvsbPn7Ka3muX3g90cq38D1z4F/BVxZPL4LuO44z/ks8HvA5xZtvzYzP1G2wL5ZTdOTDVPL8/sjDZT6jT8zDwF/CFydmW/LzGuLbcs95y7gqRHU2CuraXqyYWp5fn+kgVLBHxFvBXYBtxePt0fEbSt8zd+IiPuKqaBTlnnNyyNiOiKmZ2ZmVvhS7bOapicbppbn90caKDvH/9vARcDTAJm5C3j5Cl7vOuAnge3AXuB3ltoxM6/PzKnMnJqcnFzBS7XTapqebJhant8faaBs8B/OzAOLtg39rlhm7svM5zNzFriBwT8mWmA1TU82TC3P7480UPbN3fsj4p8CL4qIc4ErgP817ItFxObM3Fs8fBuwe7n9+2hiIrhk25mcd8Xrhm56Ws1z+8DvjzRQaunFiDgR+BDwC8Wm/w58dLk3eCPiJuBi4HRgH4PpoosZTPMksAd494J/CJZk564kDW+pzt2h1tyNiBMz80cjrawEg1+ShrdU8Je9qudnIuIB4LvF4wsi4g9GXKMkqQZl5/ivBd4I3AaQmfdGxM9VVpVWrK6lDvu6pGKXxt2lsWg4pZdezMxHI17wQ/H86MvRatTVmdrXDtgujbtLY9Hwyl7O+WhE/AyQEbEmIv4N8GCFdWkF6upM7WsHbJfG3aWxaHhlg/9fAu8BzgL+L4Mrc95TUU1aobo6U/vaAdulcXdpLBpeqamezHwCeGfFtWiV5jpTF/6FrqIzta7XaZoujbtLY9Hwyl7VszUi/iQiZopbLX81IrZWXZyGU1dnal87YLs07i6NRcMr28D1beD3gZuKTW8H/nVm/sMKa5vndfzl1bXUYV+XVOzSuLs0Fh3bqhq4IuK+zHzlom33ZuYFI6xxSQa/JA1vRWvuLvBnEXE18AUGt1v4J8DXI+JUgMz0vvuS1BJlg/+y4s938+O7cgaDKZ8EnO9viCqaco4cmeX+vQfYe+AQmze+lG2bT+KEE8peELY6NhlJo1c2+D8A3J6Zz0TEvwMuBP5jZt5TXWkaVhVNOUeOzHLrvY/x4Vt3zx/zozvOZ8cFZ1Ue/jYZSdUo+zf3w0XovxZ4PfApjr/mrmpWRVPO/XsPzIf+3DE/fOtu7t+7eHmG0bPJSKpG2eCfuz3DLwI3ZOafAi+upiStVBVNOXsPHPuYjx+ovtHHJiOpGmWD/7GI+C/8+E3dlwzxXNWkiqUFN2986TGPeebG6ht9XCpRqkbZ8L6MweIrb8zMp4FTgX9bVVFamSqacrZtPomP7jj/Bcf86I7z2bZ540hqXo5NRlI1hlqIZVy8jr+8Kppy5q7qefzAIc7cuJZtmzfWflWPTUbS8EayAte4GPySNLxVrcAlSeoOg1+Seqb0Clwqb7XdpmWfP+r9JPWDwT9iq+02Lfv8Ue8nqT+c6hmx1Xabln3+qPeT1B8G/4itttu07PNHvZ+k/jD4R2y13aZlnz/q/ST1h8E/YqvtNi37/FHvJ6k/bOCqwGq7Tcs+f9T7SeoWO3clqWfs3JUkAV7HPxZ1N1TZwCVpIYO/ZnU3VNnAJWkxp3pqVndDlQ1ckhYz+GtWd0OVDVySFjP4a1Z3Q5UNXJIWM/hrVndDlQ1ckhbzOv4xqLuhygYuqZ+Wuo7fq3rGYGIi2Dq5nq2T6zv5epKazakeSeqZyoI/Ij4TEfsjYveCbadGxB0R8f3iz1Oqev02mJ1NHpl5lm89/ASPzDzL7Ozqp93ackxJ41PlVM9ngd8DPrdg29XAnZn5sYi4unj8gQpraKwqGqvackxJ41XZb/yZeRfw1KLNlwI3Fp/fCOyo6vWbrorGqrYcU9J41T3Hvykz9xafPw5sWmrHiLg8IqYjYnpmZqae6mpURWNVW44pabzG9uZuDq4jXXKyODOvz8ypzJyanJyssbJ6VNFY1ZZjShqvuoN/X0RsBij+3F/z6zdGFY1VbTmmpPGqtIErIrYAX8vM84vHHweeXPDm7qmZ+f7jHadrDVxzqmisassxJVWv9hW4IuIm4GLgdGAf8NvArcDNwDnAD4DLMnPxG8BH6WrwS1KVau/czcx3LPGlN1T1mpKk47NzV5J6xnv1DMElDCV1gcFfkh2skrrCqZ6S7GCV1BUGf0l2sErqCoO/JDtYJXWFwV+SHaySusI3d0uamAgu2XYm513xOjtYJbWawT8ElzCU1AVO9UhSz/gbf01s/pLUFAZ/DWz+ktQkTvXUwOYvSU1i8NfA5i9JTWLw18DmL0lNYvDXwOYvSU3im7s1sPlLUpMY/DWx+UtSUzjVI0k9Y/BLUs/0aqqnzd2zba5dUrP0Jvjb3D3b5tolNU9vpnra3D3b5tolNU9vgr/N3bNtrl1S8/Qm+NvcPdvm2iU1T2+Cv83ds22uXVLzRGaOu4bjmpqayunp6VUfZ+7KmDZ2z7a5dknjERE7M3Nq8fbeXNUD7e6ebXPtkpqlN1M9kqQBg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6Ziy3bIiIPcBB4HngyLHuJSFJqsY479Xz85n5xBhfX5J6yakeSeqZcQV/At+IiJ0RcfmYapCkXhrXVM9rM/OxiDgDuCMivpuZdy3cofgH4XKAc845Zxw1SlInjeU3/sx8rPhzP/AV4KJj7HN9Zk5l5tTk5GTdJUpSZ9Ue/BGxLiI2zH0O/AKwu+46JKmvxjHVswn4SkTMvf4fZ+btY6hDknqp9uDPzEeAC+p+XUnSgJdzSlLPdHax9dnZZM+Tz7HvmUNsOmktW05bx8RENO6YklS3Tgb/7Gxy+/2Pc9XNuzh0eJa1aya45rLtXLLtzBUHdRXHlKRx6ORUz54nn5sPaIBDh2e56uZd7HnyuUYdU5LGoZPBv++ZQ/MBPefQ4Vn2HzzUqGNK0jh0Mvg3nbSWtWteOLS1ayY4Y8PaRh1Tksahk8G/5bR1XHPZ9vmgnpuP33LaukYdU5LGITJz3DUc19TUVE5PTw/1nLkrcPYfPMQZG0Z7Vc8ojylJVYmIncda76STV/UATEwEWyfXs3VyfaOPKUl16+RUjyRpaQa/JPWMwS9JPWPwS1LPGPyS1DOtuJwzImaAH5TY9XTgiYrLqVOXxtOlsUC3xtOlsYDjWegnMvOoJQxbEfxlRcT0sa5ZbasujadLY4FujadLYwHHU4ZTPZLUMwa/JPVM14L/+nEXMGJdGk+XxgLdGk+XxgKO57g6NccvSTq+rv3GL0k6DoNfknqmM8EfEZdExPci4qGIuHrc9QwjIl4WEd+MiAci4v6IuLLYfmpE3BER3y/+PGXctZYVES+KiP8TEV8rHr88Iu4uzs8XI+LF466xrIg4OSK+FBHfjYgHI+I1LT83v1n8nO2OiJsiYm2bzk9EfCYi9kfE7gXbjnk+YuA/F+O6LyIuHF/lR1tiLB8vftbui4ivRMTJC772wWIs34uIN670dTsR/BHxIuD3gTcBrwDeERGvGG9VQzkCvC8zXwG8GnhPUf/VwJ2ZeS5wZ/G4La4EHlzw+D8B12bmTwF/C/z6WKpamU8Ct2fmecAFDMbVynMTEWcBVwBTmXk+8CLg7bTr/HwWuGTRtqXOx5uAc4uPy4HraqqxrM9y9FjuAM7PzFcCfwV8EKDIhLcD24rn/EGRfUPrRPADFwEPZeYjmfl3wBeAS8dcU2mZuTcz7yk+P8ggWM5iMIYbi91uBHaMpcAhRcTZwC8CnyoeB/B64EvFLm0ay0bg54BPA2Tm32Xm07T03BROAF4aEScAJwJ7adH5ycy7gKcWbV7qfFwKfC4Hvg2cHBGbaym0hGONJTO/kZlHioffBs4uPr8U+EJm/r/M/GvgIQbZN7SuBP9ZwKMLHv+w2NY6EbEFeBVwN7ApM/cWX3oc2DSuuob0u8D7gbnV6U8Dnl7ww9ym8/NyYAb4r8XU1aciYh0tPTeZ+RjwCeBvGAT+AWAn7T0/c5Y6H23Phn8B/Fnx+cjG0pXg74SIWA98GXhvZj6z8Gs5uO628dfeRsRbgP2ZuXPctYzICcCFwHWZ+SrgORZN67Tl3AAUc9+XMvgH7e8B6zh6qqHV2nQ+lhMRH2IwDfz5UR+7K8H/GPCyBY/PLra1RkSsYRD6n8/MW4rN++b+W1r8uX9c9Q3hZ4G3RsQeBlNur2cwR35yMbUA7To/PwR+mJl3F4+/xOAfgjaeG4B/BPx1Zs5k5mHgFgbnrK3nZ85S56OV2RARvwa8BXhn/rjZamRj6Urw/yVwbnFlwosZvAFy25hrKq2YA/808GBmXrPgS7cB7yo+fxfw1bprG1ZmfjAzz87MLQzOw//IzHcC3wR+pditFWMByMzHgUcj4qeLTW8AHqCF56bwN8CrI+LE4udubjytPD8LLHU+bgN+tbi659XAgQVTQo0UEZcwmCp9a2b+aMGXbgPeHhEviYiXM3jD+n+v6EUysxMfwJsZvAP+MPChcdczZO2vZfBf0/uAXcXHmxnMjd8JfB/4c+DUcdc65LguBr5WfL61+CF9CPhvwEvGXd8Q49gOTBfn51bglDafG+A/AN8FdgN/BLykTecHuInB+xOHGfyP7NeXOh9AMLji72HgOwyuZhr7GI4zlocYzOXPZcEfLtj/Q8VYvge8aaWv6y0bJKlnujLVI0kqyeCXpJ4x+CWpZwx+SeoZg1+SeuaE4+8iaU5EfAR4FjgJuCsz/3yJ/XYAf5WZD9RXnVSOv/FLK5CZ/36p0C/sYHCnWKlxvI5fOo7ininvYnAbgEcZ3NTsfAbNaV+KiI8Bb2VwX5VvMLgNwtcY3ADtAPCPM/PhcdQuHYtTPdIyIuIfMLj1xHYGf1/uYRD8c18/DXgbcF5mZkScnJlPR8RtFP8wjKFsaVlO9UjLex3wlcz8UQ7umLr4HlAHgEPApyPil4EfLT6A1DQGv7QKObiH/UUM7tr5FuD28VYkHZ/BLy3vLmBHRLw0IjYAv7Twi8UaChsz8+vAbzJYmhHgILCh1kqlkpzjl5aRmfdExBeBexm8ufuXi3bZAHw1ItYyuBPkVcX2LwA3RMQVwK/45q6axKt6JKlnnOqRpJ4x+CWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqmf8PAP1YPAh3nzoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# First, we visualize the relationship between speed and dist variables using a scatter plot.\n",
    "sns.scatterplot(x=\"dist\", y=\"speed\", data=cars);\n",
    "# We can see a clear linear relationship between the two variables."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dae91f2d",
   "metadata": {},
   "source": [
    "Here `X` is the predictor variable and `Y` is response variable. Our observed data are pairs of x and y values\n",
    "$$(x_1,y_1),(x_2,y_2),(x_3,y_3).....(x_n, y_n)$$.\n",
    "With linear regression model, we fit our observed data and estimate the parameters of the linear model.\n",
    "$$ y = w_0 + w_1x+ e$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "1eaa85b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a column of ones to X to represent the bias term using np.vstack or np.c_\n",
    "X = cars.dist.values\n",
    "Y = cars.speed.values\n",
    "X_mat = np.vstack((np.ones(len(X)),X)).T\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71c62ce0",
   "metadata": {},
   "source": [
    "Our the goal is to minimize the mean square error of a system of linear equations we can get our parameter estimates in the form of matrix multiplications.\n",
    "$$ \\mathbf{w} \\;\\;or\\; \\beta^\\hat \\;= (\\mathbf{X}^T \\mathbf{X})^{-1} \\mathbf{X}^T \\mathbf{y} $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "9f1cc885",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([8.28390564, 0.16556757])"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "beta_hat = np.linalg.inv(X_mat.T.dot(X_mat)).dot(X_mat.T).dot(Y)\n",
    "beta_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6745eee",
   "metadata": {},
   "source": [
    "It is vector containing y-axis intercept and slope of the linear regression model. Let us use the parameters to estimate the values of Y using X values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "id": "9e4a78e0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 8.61504079,  9.93958139,  8.94617594, 11.92639228, 10.93298684,\n",
       "        9.93958139, 11.26412199, 12.58866258, 13.91320318, 11.09855441,\n",
       "       12.91979773, 10.60185169, 11.59525713, 12.25752743, 12.91979773,\n",
       "       12.58866258, 13.91320318, 13.91320318, 15.90001408, 12.58866258,\n",
       "       14.24433833, 18.21796012, 21.52931161, 11.59525713, 12.58866258,\n",
       "       17.22455467, 13.58206803, 14.90660863, 13.58206803, 14.90660863,\n",
       "       16.56228437, 15.23774378, 17.55568982, 20.86704131, 22.19158191,\n",
       "       14.24433833, 15.90001408, 19.54250072, 13.58206803, 16.23114922,\n",
       "       16.89341952, 17.55568982, 18.88023042, 19.21136557, 17.22455467,\n",
       "       19.87363587, 23.51612251, 23.68169008, 28.1520146 , 22.35714949])"
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# predict using coefficients\n",
    "y_pred = X_mat.dot(beta_hat)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "3b04bd12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdQElEQVR4nO3deZRV5Znv8e8jAqK4RAWJoFiaRE2cwC4HrrbL4eYiDqFCRrWNafGalcRoEuOAmqTttqOGaPQmxsQIUdEo0SDQRhwaTTs1aiEKRMU4AKZEqahEW0DL4rl/7FPWGakz7/3u+n3WclHnPafOfnadyi+73v28e5u7IyIi4dks7gJERKQ6CnARkUApwEVEAqUAFxEJlAJcRCRQmzdzY8OHD/eWlpZmblJEJHiLFi36m7uPyB9vaoC3tLTQ3t7ezE2KiATPzFYWG9cUiohIoBTgIiKBUoCLiARKAS4iEigFuIhIoBTgIiKBUoCLiARKAS4i0khPPgkHHADr1tX9rZu6kEdEpN944QXYY4/ex+3tcNhhdd2EAlxEpJ5Wr4ZddoGurt6xBQvqHt6gKRQRkfr4+9+j4B41qje8Z80CdzjyyIZsUgEuIlKLDRtg/HgYNgxWrYrGrrkmCu4vfamhm1aAi4hUo7sbvvAFGDIEFi6Mxi66KArub36zKSVoDlxEpBLucOaZ8Itf9I6deipcfz2YNbUUBbiISLkuvRQuuKD38cSJMHcuDBwYSzkKcBGRvsyYAVOm9D7ebz949FHYaqv4akIBLiJS2n/8B3z2s72Phw+H55+H7bePr6YsCnARkXyPPgqHHpo7tnIljBkTTz0lKMBFRHr8+c+w9965Y0uXFo4lhNoIRURWrYo6SLKD+uGHo46ThIY3KMBFpD97803YYYdoBWWPuXOj4M6fQkkgBbiI9D/r1sHYsdFJyc7OaOz666Pgzj5pmXAKcBHpP7q64Jhjova/Z56Jxi65JAru7DbBQCjARST93OG002DQIJg/Pxo74wzYuBEuvDDe2mqgLhQRSbcf/hD+7d96H0+eDL//PQwYEF9NdaIAF5F0+uUv4Vvf6n180EHwpz/BFlvEVlK9KcBFJF3uuAO++MXexzvvDEuWRJd7TRkFuIikw4MP5t44YcCAqL971Kj4amowBbiIhO3pp2HcuNyx55/PvR9lSinARSRML78MH/947tjjj8OBB8ZTTwzURigiYVmzBoYOzQ3v+fOjVsF+FN5QRoCb2c5m9qCZPWtmfzazszLj/2JmHWb2dOa/YxpfrojUw5zFHRxy2QPsev4fOeSyB5izuCPukvr27ruw++4wciS89140NnNmFNxHHx1vbTEpZwrlQ+Bsd3/KzLYGFpnZ/ZnnfubuP21ceSJSb3MWdzB19lLWd3UD0LF2PVNnLwWgbdzoOEsr7oMPYMKEqAWwxxVXwPe+F1tJSdHnEbi7r3b3pzJfvws8ByTwUxaRcky7d/lH4d1jfVc30+5dHlNFJWzcCCedBIMH94b3978fjSu8gQrnwM2sBRgHPJ4ZOsPMlpjZDDPbtsT3nG5m7WbW3tlz0RgRic1ra9dXNN507lFQDxgAv/tdNHbiidFd4KdNa/qNg5Os7AA3s6HAH4DvuPs7wLXAx4GxwGrgimLf5+7XuXuru7eOGDGi9opFpCajhg2paLyprrwSNtssmiIBOPxweP99uOWWaFxylPUTMbOBROF9i7vPBnD3N9y92903Ar8B+tfpX5FAnTNhD4YMzL0OyJCBAzhnQox90zffHB1Zn3129Hj33eGdd6LFOYMGxVdXwvV5EtPMDJgOPOfuV2aN7+juqzMPPwcsa0yJIlJPPScqp927nNfWrmfUsCGcM2GPeE5g3nMPTJzY+3joUHjppegmC9KncrpQDgFOBpaa2dOZsQuAE8xsLODACuDrDahPRBqgbdzoeDtOnngiurhUtpdegt12i6eeQPUZ4O7+CFDsrMHd9S9HJF3mLO5IxpFug2Tv3zZDBmIGa9d1ld7X5cthzz1zxxYvju6Ok1KN/B3QUnqRBgmu37pC+fu3dn3XR88V7Otrr8GYMVEnSY8HHoAjjmhqzc3W6N8BndYVaZBg+q2rVGz/sq3v6ubaOYui4B49uje8b789ahVMeXhD438HdAQu0iCJ77eu0ab2Y/CHHzDrd+czdvULvYPXXAPf/GYTKkuORv8O6AhcpEES3W9dB8X2Y7ON3fx69iUsv2Jyb3j/4AfREXc/C29o/O+AAlykQRLZb11HOfvnzsX3X8vL0yYx4S8LAbhj7ATmLHoV/vVfY6wyXo3+HdAUikiDJKrfugF69uON83/I1++b8dH4g7v9Az869cd875i9UrOv1Wr074C5e13eqBytra3e3t7etO2JSANNnw6nndb7eL/94LHHYMst46sppcxskbu35o/rCFyClPb+6kSbNw8mTep9PGIEPPccbL99fDX1UwpwCU7a+6sT65FH4B//MXds5cqoTVBioZOYEpy091cnzrJl0YWmssN72bKos0ThHSsFuAQn7f3VibFqVRTc++zTO/bII1Fw77VXfHXJRxTgEpy091fH7s03o3ntXXbpHZs3LwruQw6Jry4poACX4KS9vzo2770H++4Lw4fD3/4WjU2fHgX38cfHW5sUpQCX4LSNG82lk/dh9LAhGDB62BAunbyPTmBWq6sruqv70KGwNDoZzL//exTcp54ab22ySepCkSDFfj3rNHCHKVPgt7/tHfv2t+Hqq3XfyUAowEX6owcegKOO6n38+c/DrFnRjYQlGApwkQoFvYjon/8Zbrih9/HBB0f3ndxii9hKkuppDlykAj2LiDrWrsfpXUQ0Z3FH3KVt2sUXR9MiPeE9ZQqsXw///d8K74DpCFykAptaRJTIo/AZM6KwzrZ8eXTXdwmeAlykAsEsIpo/H445Jnfsscdg/Ph46pGG0BSKSAUSv4ho0aJoqiQ7vO+8M+o4UXinjgJcpAKJXUS0cGEU3K1ZVxy95poouNvaYitLGktTKCIVSNxNGl55BXbbLXfsvPPgssviqUeaSgEuUqFELCJauxa23bZwvIk3aJH4KcBFQtLVBYMGFY5v3KjVk/2QAlwkBO6wWZFTVl1dsLn+Z9xf6SSmSNKZFYb32rVRqCu8+zUFuEhSjRpVOC2yYkUU3NtsE0tJkiwKcJGkOf74KLhXr+4de+KJKLizb7Ig/Z4CXCQppk6Ngvuuu3rH7rgjCu4DDoivLkksBbhI3GbMiII7u3f78suj4P785+OrSxJPZ0BE4vLgg3DkkbljX/ta7g0WRDahzwA3s52Bm4CRgAPXufvVZrYdMAtoAVYAX3L3txtXqlQihGtWl1tjvV/XiBor8vzz8KlP5Y6NHQuLF9f2vtLvlHME/iFwtrs/ZWZbA4vM7H7ga8ACd7/MzM4HzgfOa1ypUq6ea1b3XPa055rVQGJCvNwa6/26RtRYts5O2GGHwnGtnpQq9TkH7u6r3f2pzNfvAs8Bo4FJwI2Zl90ItDWoRqnQpq5ZnRTl1ljv1zWixj5t2BDNceeHt7vCW2pS0UlMM2sBxgGPAyPdvafP6XWiKZZi33O6mbWbWXtnZ2cttUqZQrhmdbk11vt1laj5Pd2j4B6Sd6nZ7m4Ft9RF2QFuZkOBPwDfcfd3sp9zdyeaHy/g7te5e6u7t44YMaKmYqU8ib9mNeXXWO/XVaKm9yy2evK990oviRepQlm/SWY2kCi8b3H32ZnhN8xsx8zzOwJrGlOiVCqx16zOUm6N9X5dI2rMMWhQ4erJ116LgnvLLauuRaSYcrpQDJgOPOfuV2Y9NQ84Bbgs8+/chlQoFUvcNauLKLfGer+uETUCxa8EuGQJ7LNP1dsX6Yt5H3NxZnYo8DCwFNiYGb6AaB7898AYYCVRG+Fbm3qv1tZWb29vr7VmkeTYd19YujR3bP58OProeOqRVDKzRe7emj/e5xG4uz8ClLrQ8FG1Fibxala/eAh96RU57DB4+OGcodvHf45zD5vCqKcHcc7IjrD3T4KglZj9WLP6xUPoSy/beefBT36SM/TW3uM4pO3H6dg/CYpOh/djzeoXD6EvvU+zZkXz3HnhjTvHn/TT8PdPgqQj8H6sWf3iIfSll7R4Mey/f+F41rmjoPdPgqYj8H6sWf3iIfSlF1izJjrizg/vIqsng9w/SQUFeD/WrH7xEPrSP/LBB1Fwj8xbWLxxY8nVk0Htn6SKplD6sWb1i4fQlw4U7+V+910YOnST3xbM/knq9NkHXk/qA5dEKhbcL78Mu+7a/FpEiqi6D1yklKT1dldcT7HgXrCg8CYLMQnm+uYSGwW4VCVpvd0V1VMsuK++Gs48s9Flli2I65tL7HQSU6qStN7usurZddfC8D7xxOjkZILCGxJ+fXNJDB2BS1WS1vu8yXpOPbXwPpMjR8Lrrzehsuok8vrmkjg6ApeqJK33udh2/2nx3bxy+XGF4e2e6PCGBF7fXBJJAS5VSVrvc3Y9B766jBWXH8cl9/0y90UB3cIsMdc3l0TTFIpUJWm9z23jRjNkdQcTjj2o8MlAQjtb7Nc3lyCoD1zCt24dbLVV4fjGjcU7TkQCoz5wSZ9S95fcsAEGD25+PSJNpgBPqXov2EjcjR+KHVmvXg0f+1jdaxJJKgV4CtV7wUaibvxQLLiffBJaC/66FEk9daGkUL0XbCTixg9mheE9c2Y0jaLwln5KR+ApVO8FG3He+OEv0yYxcGNuqHPWWXDVVXXdtkiIFOApNGrYEDqKhGG1Czbq/X7lbOe3t/+II15elPuCcePgqafquk2RkGkKJYXqvWCjmTd+uPQ/r2XF5ccVhre7wlskj47AU6jeCzaasgBk5kzavvrVguE5T/1VC01EStBCHolXGTcNFunvtJCnn0l8H/ibb8Lw4YXjCm6RsinAUyjRfeDd3bB5kV+7rq7i4yJSkk5iplBi+8DNCkO6szM66lZ4i1RMAZ5CiesDL7YIZ9GiKLiLTaOISFkU4ClU7wv3V/1+xYJ7+vQouIuduBSRiijAUyj2PvBiwX388VFwn3pqVTWISCFNPKZQbH3gpa69rc4SkYZQH7jUbv/9o37ufApukbqoug/czGYAxwFr3H3vzNi/AP8X6My87AJ3v7t+5aZDLb3Tzbr+dk2uugq++93C8SYEdxA/H5EGK2cK5QbgF8BNeeM/c/ef1r2ilKild7pZ19+u2n/9Fxx+eOF4k464E//zEWmSPk9iuvtDwFtNqCVVaumdbtb1tyv26qvRPHd+eDf5bu+J/fmINFktXShnmNkSM5thZtuWepGZnW5m7WbW3tnZWeplqVNL73Szrr9dtvXro+AeMyZ3fOPGWOa5E/fzEYlJtQF+LfBxYCywGrii1Avd/Tp3b3X31hEjRlS5ufDU0otd7z7uqrlHwb3llrnj77zT+1wMEvPzEYlZVQHu7m+4e7e7bwR+AxxY37LCV0svdrOuv71JZoV3fF+6NArurbduXh1FJOLnI5IAVfWBm9mO7r468/BzwLL6lZQOtfRiN+X626UUO6q++WY46aTGb7tMsf58RBKkzz5wM7sVOBwYDrwB/CjzeCzgwArg61mBXpL6wBOsWHCffjr8+tfNr0VEclTdB+7uJxQZnl6XqqSkpvU5FwvukSPh9dfrv60ESVMfeZr2RSqja6EkUE+fc8fa9Ti9fc5zFnfUbyPjxxcN709dNJ858xcV+Yb0aMrPt0nStC9SOQV4AjW0z/mii6LgXrgwZ7jlvLtoOe+uftFPnaY+8jTti1ROF7NKoIb0Oc+cCUVuGtxy3l313U4A0tRHnqZ9kcrpCDyB6trnvGRJdMSdH97uHHLpgvptJyBp6iNP075I5RTgCVSXPuee1ZP77Zc7nrXsvb/2U6dpv9O0L1I5TaEkUE19zu6FC3AgCvQttqjfdgKWpv1O075I5XQ98DQp1hL4yivQ0tL0UkSkfqruA5d4VNTbWyy4n38e9sj9M/qiOUu59fFX6XZngBknHLQzl7Tt04DqC6lXWaT+FOAJVPb1rosF9333wWc+UzB80Zyl3Lxw1UePu90/etzoENf1u0UaQycxE6jP3t6DDioM75//PJr/LhLeALc+/mpF4/WkXmWRxlCAJ1CpHt7bf/zlKLifeKJ3cMqUKLjPOGOT79ld4lxHqfF6Uq+ySGNoCiWBRg0bQkdWuP34np9z4jP35r5or71gWfkXgRxgVjSsBzThmt75+5M9LiLV0xF4AvX09k5etoAVlx9XGN7uFYU3wAkH7VzReD2pV1mkMXQEnkBtwz6g7ZKJhU/UMN3Rc6Iyji4U9SqLNIb6wJPknXdgm20Kx2O476SIJIf6wKtQa+9yud8/p30VbQfsUvgG3d05qyrVSy0i2RTgJdTau1xJL3db3vf+w7mz+cFXDqQtL7zVSy0i2XQSs4Rae5f7/H6zgl7u//WNGbScdxdv2qCC7aiXWkTy6Qi8hFp7l0u97tGpR8HU3LHJ/zSNp0Z/apPfr15qEcmnAC+h1t7l/O9fcflxhS+66SYO6Rhd1nbUSy0i+TSFUkKtvcs93z9/xhmF4X388VFnycknl70d9VKLSD4dgZdQa+9y24OzaLvk7Jyx7sGDGbBhQ1XbUS+1iORTH3i93X03HHts4bh6uUWkSuoDb7Rnn42uT5JPwS0iDaIAr8GcxR1cN/sJ7r5kcuGTDQhuLeQRkWwK8CrNfXIlbQe2FCzCmbPoVdr236nu29NCHhHJpwCvVOamwZPyhnc/+04+2Hwgo+97oSEBvqmFPApwkf5JAV6JItfO3ves23hni6EfPW7Uwhot5BGRfArwcrS0wMqVOUNfPGcmT262bcFLG7WwRgt5RCSfFvJsyrXXRkfd2eH92GPgzkknHNHUhTVayCMi+XQEXszcudDWljt2223w5S9/9LDZC2u0kEdE8mkhT7aFC2H8+NyxP/4RjjkmnnpERKhhIY+ZzQCOA9a4+96Zse2AWUALsAL4kru/Xc+Cm+qFF2CPvKmI3/wGTjutrG9vRH92KO8pIvEpZw78BuDovLHzgQXu/klgQeZxeN54I5rjzg7vH/4wahWsILynzl5Kx9r1OL392XMWd1RdVijvKSLx6jPA3f0h4K284UnAjZmvb4SC9SzJ9j//EwX3xz7WO3byyVFwX3xxRW/ViBsthPKeIhKvak9ijnT31ZmvXwdGlnqhmZ0OnA4wZsyYKjdXJx9+CAMH5o6NHx91llSpEf3ZobyniMSr5jZCj86CljwT6u7XuXuru7eOGDGi1s1Vxx0+8Ync8N5uu+imwTWEN5Tuw66lPzuU9xSReFUb4G+Y2Y4AmX/X1K+kOjv22OjO7i+91Du2YQO8+WbOHd+r1Yj+7FDeU0TiVW2CzQNOyXx9CjC3PuXU0be/Hc1z331379jbb0dH44MH120zbeNGc+nkfRg9bAgGjB42hEsn71NTd0co7yki8eqzD9zMbgUOB4YDbwA/AuYAvwfGACuJ2gjzT3QWaEof+BVXwPe/nzu2ahXsvHNjtysi0iBV94G7+wklnjqq5qrq6bbb4IS8Up95Bvbdt+Cl6ocWkTQIfyn9n/4ERxyRO7ZgARx5ZNGX67raIpIW4V7MqrMzmuPODu9bbonmuEuEN6gfWkTSI7wAX7cOpk2DHXboHfvJT6LgPvHEPr9d/dAikhbhBHhXF/zqV1E/97nnwoQJcN99UXCfc07Zb6N+aBFJizAC/I47YM894RvfgN12g4cegnvugc98puK3Uj+0iKRFGCcxn30Wtt46urTrxIlFb21WLl1XW0TSIozrgb//frQMvg4rJ0VEQlN1H3gi1HHlZK3UQy4iSRFGgCeEeshFJEk0J1EB9ZCLSJIowCugHnIRSRIFeAXUQy4iSaIAr4B6yEUkSXQSswLqIReRJFGAV6ht3GgFtogkQpABHnIvdsi1i0iyBBfgIfdih1y7iCRPcCcxQ+7FDrl2EUme4AI85F7skGsXkeQJLsBD7sUOuXYRSZ7gAjzkXuyQaxeR5AnuJGbIvdgh1y4iyRPG9cBFRPqxUtcDD24KRUREIgpwEZFAKcBFRAKlABcRCZQCXEQkUApwEZFAKcBFRAKlABcRCZQCXEQkUDUtpTezFcC7QDfwYbGVQiIi0hj1uBbKEe7+tzq8j4iIVEBTKCIigao1wB24z8wWmdnpxV5gZqebWbuZtXd2dta4ORER6VFrgB/q7vsDE4Fvmdlh+S9w9+vcvdXdW0eMGFHj5kREpEdNAe7uHZl/1wB3AgfWoygREelb1QFuZluZ2dY9XwP/B1hWr8JERGTTaulCGQncaWY97/M7d7+nLlWJiEifqg5wd38Z2K+OtRQ1Z3FH3W9B1oj3FBFptkTfE3PO4g6mzl7K+q5uADrWrmfq7KUAVQduI95TRCQOie4Dn3bv8o+Ctsf6rm6m3bs8Ue8pIhKHRAf4a2vXVzQe13uKiMQh0QE+atiQisbjek8RkTgkOsDPmbAHQwYOyBkbMnAA50zYI1HvKSISh0SfxOw5qVjPjpFGvKeISBzM3Zu2sdbWVm9vb2/a9kRE0sDMFhW7XHeip1BERKQ0BbiISKAU4CIigVKAi4gESgEuIhKopnahmFknsLKMlw4H0nSfzTTtT5r2BdK1P2naF9D+ZNvF3QvuiNPUAC+XmbWn6Q73adqfNO0LpGt/0rQvoP0ph6ZQREQCpQAXEQlUUgP8urgLqLM07U+a9gXStT9p2hfQ/vQpkXPgIiLSt6QegYuISB8U4CIigUpcgJvZ0Wa23MxeNLPz466nEma2s5k9aGbPmtmfzeyszPh2Zna/mf0l8++2cddaLjMbYGaLzeyuzONdzezxzOczy8wGxV1jucxsmJndYWbPm9lzZjY+8M/mu5nfs2VmdquZbRHS52NmM8xsjZktyxor+nlY5P9l9muJme0fX+WFSuzLtMzv2hIzu9PMhmU9NzWzL8vNbEK1201UgJvZAOAaYCLwaeAEM/t0vFVV5EPgbHf/NHAw8K1M/ecDC9z9k8CCzONQnAU8l/X4cuBn7v4J4G1gSixVVedq4B533xPYj2i/gvxszGw0cCbQ6u57AwOArxDW53MDcHTeWKnPYyLwycx/pwPXNqnGct1A4b7cD+zt7vsCLwBTATKZ8BVgr8z3/DKTfRVLVIADBwIvuvvL7v4BcBswKeaayubuq939qczX7xIFxGiifbgx87IbgbZYCqyQme0EHAtcn3lswJHAHZmXhLQv2wCHAdMB3P0Dd19LoJ9NxubAEDPbHNgSWE1An4+7PwS8lTdc6vOYBNzkkYXAMDPbsSmFlqHYvrj7fe7+YebhQmCnzNeTgNvc/X13fwV4kSj7Kpa0AB8NvJr1+K+ZseCYWQswDngcGOnuqzNPvQ6MjKuuCl0FnAtszDzeHlib9UsZ0uezK9AJ/DYzJXS9mW1FoJ+Nu3cAPwVWEQX334FFhPv59Cj1eYSeDacC8zNf121fkhbgqWBmQ4E/AN9x93eyn/OobzPxvZtmdhywxt0XxV1LnWwO7A9c6+7jgPfImy4J5bMByMwNTyL6P6ZRwFYU/gkftJA+j00xswuJpldvqfd7Jy3AO4Cdsx7vlBkLhpkNJArvW9x9dmb4jZ4/9zL/romrvgocAnzWzFYQTWUdSTSHPCzzJzuE9fn8Ffiruz+eeXwHUaCH+NkA/G/gFXfvdPcuYDbRZxbq59Oj1OcRZDaY2deA44CTvHfRTd32JWkB/iTwycyZ9EFEE/3zYq6pbJk54unAc+5+ZdZT84BTMl+fAsxtdm2Vcvep7r6Tu7cQfQ4PuPtJwIPAFzIvC2JfANz9deBVM9sjM3QU8CwBfjYZq4CDzWzLzO9dz/4E+flkKfV5zAO+mulGORj4e9ZUSyKZ2dFEU5Cfdfd1WU/NA75iZoPNbFeiE7NPVLURd0/Uf8AxRGdsXwIujLueCms/lOhPviXA05n/jiGaO14A/AX4T2C7uGutcL8OB+7KfL1b5pftReB2YHDc9VWwH2OB9sznMwfYNuTPBrgYeB5YBswEBof0+QC3Es3fdxH9hTSl1OcBGFGH2kvAUqLum9j3oY99eZForrsnC36V9foLM/uyHJhY7Xa1lF5EJFBJm0IREZEyKcBFRAKlABcRCZQCXEQkUApwEZFAKcBFRAKlABcRCdT/Bx8u74nZAuL6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot data and predictions\n",
    "plt.scatter(X, Y)\n",
    "plt.plot(X, y_pred, color='red');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae297a02",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4b878b5b",
   "metadata": {},
   "source": [
    "<h4 align=\"center\">Gradient descent for multiple variables</h4>\n",
    "\n",
    "Fitting parameters for the hypothesis with gradient descent \n",
    "- Parameters are $\\theta_0 \\;\\; to \\;\\; \\theta_n$.\n",
    "- Instead of thinking about this as n separate values, think about the parameters as a single vector (θ). Where θ is n+1 dimensional \n",
    "\n",
    "**Hypothesis:**\n",
    "$$h_\\theta (x)= h_\\theta(X) = \\theta^TX = \\theta_0x_0 + \\theta_1 x_1 + \\theta_2x_2 + \\theta_3x_3 + ⋯ + \\theta_nx_n$$.\n",
    "\n",
    "**Parameter vector:** \n",
    "$$\\theta = \\theta_0, \\theta_1 ... \\theta_n$$\n",
    "\n",
    "**Feature vector:** \n",
    "$$x = x_0x_1 ⋯ x_n$$\n",
    "\n",
    "**Cost Function**:\n",
    "\n",
    "$$J(\\theta) = \\frac{1}{2n} \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2$$\n",
    "\n",
    "**Gradient Descent:**\n",
    "\n",
    "repeat{\n",
    "\n",
    "$$ \\theta_j := \\theta_j - \\alpha \\frac{\\partial} {\\partial \\theta_j} (J(\\theta))  \\;\\;\\;\\;\\;\\; // simultaneously \\; update \\; for \\;\\; all \\;\\; j = 0,1,..n $$\n",
    "}\n",
    "\n",
    "**Previously (n=1):**\n",
    "\n",
    "repeat{\n",
    "\n",
    "$$ \\theta_0 := \\theta_0 - \\alpha \\frac{\\partial} {\\partial \\theta_0} (J(\\theta)) $$  \n",
    "\n",
    "$$ \\theta_1 := \\theta_1 - \\alpha \\frac{\\partial} {\\partial \\theta_1} (J(\\theta)) \\;\\;\\;\\;\\; (simultaneously \\; update \\; \\theta_0, theta_1) $$\n",
    "}\n",
    "\n",
    "**New Algorithm (n $\\geq$ 1):**\n",
    "\n",
    "repeat{\n",
    "\n",
    "$$ \\theta_j := \\theta_j - \\alpha \\frac{\\partial} {\\partial \\theta_j} (J(\\theta))  \\;\\;\\;\\;\\;\\; // simultaneously \\; update \\; for \\;\\; all \\;\\; j = 0,1,..n $$\n",
    "}\n",
    "\n",
    "**Question:** Find $\\theta_0, \\theta_1, \\theta_2=?$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7d83087",
   "metadata": {},
   "source": [
    "### Implementing Gradient Descent using `Python` and `scikit-learn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "ccd7763f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generateXvector(X):\n",
    "    \"\"\" Taking the original independent variables matrix and add a row of 1 which corresponds to x_0\n",
    "        Parameters:\n",
    "          X:  independent variables matrix\n",
    "        Return value: the matrix that contains all the values in the dataset, not include the outcomes values \n",
    "    \"\"\"\n",
    "    vectorX = np.c_[np.ones((len(X), 1)), X]\n",
    "    return vectorX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "151aea03",
   "metadata": {},
   "outputs": [],
   "source": [
    "def theta_init(X):\n",
    "    \"\"\" Generate an initial value of vector θ from the original independent variables matrix\n",
    "         Parameters:\n",
    "          X:  independent variables matrix\n",
    "        Return value: a vector of theta filled with initial guess\n",
    "    \"\"\"\n",
    "    theta = np.random.randn(len(X[0])+1, 1)\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "id": "2187f1c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def Multivariable_Linear_Regression(X,y,learningrate, iterations):\n",
    "    \"\"\" Find the multivarite regression model for the data set\n",
    "         Parameters:\n",
    "          X:  independent variables matrix\n",
    "          y: dependent variables matrix\n",
    "          learningrate: learningrate of Gradient Descent\n",
    "          iterations: the number of iterations\n",
    "        Return value: the final theta vector and the plot of cost function\n",
    "    \"\"\"\n",
    "    y_new = np.reshape(y, (len(y), 1))   \n",
    "    cost_lst = []\n",
    "    vectorX = generateXvector(X)\n",
    "    theta = theta_init(X)\n",
    "    m = len(X)\n",
    "    for i in range(iterations):\n",
    "        gradients = 2/m * vectorX.T.dot(vectorX.dot(theta) - y_new)\n",
    "        theta = theta - learningrate * gradients\n",
    "        y_pred = vectorX.dot(theta)\n",
    "        cost_value = 1/(2*len(y))*((y_pred - y)**2) #Calculate the loss for each training instance\n",
    "        total = 0\n",
    "        for i in range(len(y)):\n",
    "            total += cost_value[i][0] #Calculate the cost function for each iteration\n",
    "        cost_lst.append(total)\n",
    "    plt.plot(np.arange(1,iterations),cost_lst[1:], color = 'red')\n",
    "    plt.title('Cost function Graph')\n",
    "    plt.xlabel('Number of iterations')\n",
    "    plt.ylabel('Cost')\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ef98cb9",
   "metadata": {},
   "source": [
    "- This is a function for performing multivariate linear regression using gradient descent. The function takes four input parameters:\n",
    "\n",
    "    - `X` : a matrix of independent variables\n",
    "    - `y` : a matrix of dependent variables\n",
    "    - `learning-rate` : the learning rate for the gradient descent algorithm\n",
    "    - `iterations` : the number of iterations for the gradient descent algorithm\n",
    "\n",
    "- The function first reshapes the y matrix into a column vector, and then generates a feature vector vectorX by adding a column of ones to X. It initializes the model weights (theta) using the theta_init function (not shown).\n",
    "\n",
    "- The function then performs gradient descent by iterating over the number of iterations specified. For each iteration, it calculates the gradients using the feature vector vectorX and the model weights theta, updates the model weights, and calculates the mean squared error between the predicted values and the true values. The cost function is also calculated and appended to a list cost_lst.\n",
    "\n",
    "- Finally, the function plots the cost function using the number of iterations as the x-axis and the cost function as the y-axis. It returns the final model weights (theta)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "733fa62c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "312f3049",
   "metadata": {},
   "source": [
    "### Checking the code with built-in Linear Regression from sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "id": "b44707d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_diabetes\n",
    "diabetes = load_diabetes()\n",
    "X = diabetes.data\n",
    "y = diabetes.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "id": "bd725186",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "sc=StandardScaler()\n",
    "X_transform=sc.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "id": "337c5106",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(152.13348416289594,\n",
       " array([ -0.47623169, -11.40703082,  24.72625713,  15.42967916,\n",
       "        -37.68035801,  22.67648701,   4.80620008,   8.422084  ,\n",
       "         35.73471316,   3.21661161]))"
      ]
     },
     "execution_count": 227,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X_transform, y)\n",
    "lin_reg.intercept_, lin_reg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "id": "3574ea89",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[152.13348416],\n",
       "       [ -0.3756836 ],\n",
       "       [-11.29461849],\n",
       "       [ 24.98117024],\n",
       "       [ 15.331643  ],\n",
       "       [-15.87196745],\n",
       "       [  5.35824149],\n",
       "       [ -4.93151521],\n",
       "       [  5.66509548],\n",
       "       [ 27.60164001],\n",
       "       [  3.29631025]])"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAgEElEQVR4nO3deZxcZZ3v8c+X7iQkIZDV3CxIWOJVcGSLCCPXEeOExSXoBcRBiYpyVXRcxwvqFQfUQRmXwSt6UZCwXJZBvGRExAgBl5ElbGGXBsEkBBLJQthCEn73j+epprqru1N1uqurqvv7fr3qdU4956lzfifV6W8/55w6pYjAzMysiO0aXYCZmbUuh4iZmRXmEDEzs8IcImZmVphDxMzMCnOImJlZYQ4Rs15I+pikJyU9I2nSIG73i5J+MljbqzdJIWmPRtdh9eEQsaYn6R8kLc2/zFdJukbSwf1c56OS3trH8hHAd4B5EbFDRDzVn+31sZ03S1pR3hYR34iID9dpe9Mk/VjS4/nf8xFJ50t6dT22Z0OfQ8SamqTPAt8DvgFMBV4JnA3Mr/OmpwLbA/fWeTuDJo+m/hMYA/w3YBywH3Aj8Pe9vKZ90Aq01hQRfvjRlA9gJ+AZ4Og++owihczj+fE9YFReNhn4BbAeWAv8jvSH04XAS8Dzef1f6LbOVwHPApGXXw/Mys/by/rdAHw4z38A+D3wr8A64M/A4WV9JwI/zTWuA/4fMDbX8FLezjPAdOCrwEVlr30nKczW522+pmzZo8DngWXABuAyYPte/q2+BtwFbNfHv2dpP08A/gL8Nrf/O/BE3sZvgb3KXnM+8CNgMbCRFEq7lC0P4KPAQ3kffgCo0T9ffgzMwyMRa2YHkUYDP++jz5eAA4F9gL2BA4Av52WfA1YAU0gjiy8CERHvJ/2CfEekQ1XfKl9hRPwJ2Cs/HR8Rb6my3jcAD5LC61vAuZKUl11IGgHsBbwC+G5EPAscDjye69ghIh4vX6GkVwGXAJ/O+/FL4D8kjSzrdgxwGLAr8DpSoPXkrcDPI+KlKvbl74DXAIfm59cAs3PttwMXd+t/HHB63vc7e1j+duD1ub5jytZrLc4hYs1sEvDXiNjSR5/jgNMiYnVErAH+GXh/XrYZmEb6q3hzRPwuIup5s7jHIuLHEbEVWJi3PVXSNFJYfDQi1uVabqxyne8Bro6IxRGxmTTSGQ38bVmfsyLi8YhYC/wHKVB7Mpk0mgBA0jslrZe0UdKvu/X9akQ8GxHPA0TEeRGxMSI2kUZKe0vaqaz/1RHx27z8S8BBknYuW35GRKyPiL8AS/qo0VqMQ8Sa2VPA5G0cl58OPFb2/LHcBnAm0AH8Op9APrk+ZXbq/AUdEc/l2R2AnYG1EbGuwDq77F8eRSwHZvS0XeC5vM2ePEUKttK6FkXEeOAzwMhufZeXZiS1STpD0sOSniYdQoMUShX9I+IZ0uHD6WXLq63RWoxDxJrZH4FNwJF99Hkc2KXs+StzG/kv589FxG6k8wqflTQ396t1RPJsno4pa/svVb52OTBR0vgelm2rji77lw+P7QysrHLb5a4DjpRUzf/78rr+gXQhw1tJ56lmlcop69M56pC0A+kcUJdDczY0OUSsaUXEBuArwA8kHSlpjKQRkg6XVDqPcQnwZUlTJE3O/S8CkPR2SXvkX7wbgK2kk9gATwK71VDLGtIv7vflv8w/BOxe5WtXkc4pnC1pQt6HN5XVManboaFylwNvkzQ3X3b8OVKw/me1tZf5DjABuFDS7krGse1DS+PyNp8iheg3euhzhKSD87ma04GbImJ5D/1siHGIWFOLiG8DnyWdLF9D+qv+E6SrmyBdcbSUdHXS3aSTvl/Ly2YDvyFd9fRH4OyIWJKX/QspfNZL+nyV5XwE+CfSL9O9qO0X+ftJ52geAFaTTpQTEQ+QgvCRXEv5ISAi4kHgfcD3gb8C7yBdEPBiDdsureuvpIsQXiBdSbaRdBJ8HPCxPl56AemQ2krgPuCmHvr8X+BU0mGs/XPNNgyovucZzWyok3Q+sCIivrytvjb0eCRiZmaFOUTMzKwwH84yM7PCPBIxM7PCht3N1SZPnhyzZs1qdBlmZi3jtttu+2tETOlp2bALkVmzZrF06dJGl2Fm1jIkPdbbMh/OMjOzwhwiZmZWmEPEzMwKc4iYmVlhDhEzMyvMIWJmZoU5RMzMrDCHSLVOPx2uvbbRVZiZNRWHSLXOOAN+85tGV2Fm1lQcItVqa4MtWxpdhZlZU3GIVKu9HbZubXQVZmZNxSFSLY9EzMwqOESq5ZGImVkFh0i1PBIxM6vgEKlWe7tDxMysG4dItXw4y8ysgkOkWj6cZWZWoW4hIuk8Sasl3VPWNlHSYkkP5emE3C5JZ0nqkLRM0n5lr1mQ+z8kaUFZ+/6S7s6vOUuS6rUvgEciZmY9qOdI5HzgsG5tJwPXRcRs4Lr8HOBwYHZ+nAj8EFLoAKcCbwAOAE4tBU/u85Gy13Xf1sDySMTMrELdQiQifgus7dY8H1iY5xcCR5a1XxDJTcB4SdOAQ4HFEbE2ItYBi4HD8rIdI+KmiAjggrJ11YdHImZmFQb7nMjUiFiV558Apub5GcDysn4rcltf7St6aK8fj0TMzCo07MR6HkHEYGxL0omSlkpaumbNmmIr8UjEzKzCYIfIk/lQFHm6OrevBHYu6zczt/XVPrOH9h5FxDkRMSci5kyZMqVY5R6JmJlVGOwQWQSUrrBaAFxV1n58vkrrQGBDPux1LTBP0oR8Qn0ecG1e9rSkA/NVWceXras+PBIxM6vQXq8VS7oEeDMwWdIK0lVWZwCXSzoBeAw4Jnf/JXAE0AE8B3wQICLWSjoduDX3Oy0iSifrP066Amw0cE1+1E9bG2zaVNdNmJm1mrqFSES8t5dFc3voG8BJvaznPOC8HtqXAq/tT401aW+H554btM2ZmbUCf2K9Wj4nYmZWwSFSLd+A0cysgkOkWj6xbmZWwSFSLR/OMjOr4BCplkciZmYVHCLV8kjEzKyCQ6RaHomYmVVwiFTLIxEzswoOkWp5JGJmVsEhUi2PRMzMKjhEquWRiJlZBYdItTwSMTOr4BCplkciZmYVHCLV8kjEzKyCQ6RaHomYmVVwiFSrrS2FSAzK18KbmbUEh0i12vP3d3k0YmbWySFSLYeImVkFh0i12trS1CfXzcw6OUSq5ZGImVkFh0i1PBIxM6vgEKmWRyJmZhUcItXySMTMrIJDpFoeiZiZVXCIVMsjETOzCg6RankkYmZWwSFSLY9EzMwqOESq5ZGImVkFh0i1PBIxM6vgEKlWaSTiEDEz6+QQqZYPZ5mZVXCIVMuHs8zMKjhEquWRiJlZBYdItTwSMTOr4BCpVmkksnlzY+swM2siDQkRSZ+RdK+keyRdIml7SbtKullSh6TLJI3MfUfl5x15+ayy9ZyS2x+UdGhdix4xIk0dImZmnQY9RCTNAP4RmBMRrwXagGOBbwLfjYg9gHXACfklJwDrcvt3cz8k7ZlftxdwGHC2pLa6FT5yZJo6RMzMOjXqcFY7MFpSOzAGWAW8BbgiL18IHJnn5+fn5OVzJSm3XxoRmyLiz0AHcEDdKvZIxMyswqCHSESsBP4V+AspPDYAtwHrI6J01noFMCPPzwCW59duyf0nlbf38JouJJ0oaamkpWvWrClWuEPEzKxCIw5nTSCNInYFpgNjSYej6iYizomIORExZ8qUKcVW4hAxM6vQiMNZbwX+HBFrImIzcCXwRmB8PrwFMBNYmedXAjsD5OU7AU+Vt/fwmoHnEDEzq9CIEPkLcKCkMfncxlzgPmAJcFTuswC4Ks8vys/Jy6+PiMjtx+art3YFZgO31K1qh4iZWYX2bXcZWBFxs6QrgNuBLcAdwDnA1cClkr6W287NLzkXuFBSB7CWdEUWEXGvpMtJAbQFOCki6vdxcoeImVmFQQ8RgIg4FTi1W/Mj9HB1VUS8ABzdy3q+Dnx9wAvsiUPEzKyCP7FeLYeImVkFh0i1HCJmZhUcItXyvbPMzCo4RKolpSBxiJiZdXKI1GLECIeImVkZh0gtHCJmZl04RGrhEDEz68IhUguHiJlZFw6RWjhEzMy6cIjUwiFiZtaFQ6QWDhEzsy4cIrVwiJiZdeEQqYVDxMysC4dILRwiZmZdOERq4RAxM+vCIVILh4iZWRcOkVo4RMzMunCI1MIhYmbWhUOkFg4RM7MuHCK1cIiYmXXhEKmFQ8TMrAuHSC0cImZmXThEauEQMTPrwiFSC4eImVkXDpFaOETMzLpwiNRixAh48cVGV2Fm1jQcIrXwSMTMrAuHSC1KIRLR6ErMzJqCQ6QWI0ak6datja3DzKxJVBUiki6spm3IK4WID2mZmQHVj0T2Kn8iqQ3Yf+DLaXIOETOzLvoMEUmnSNoIvE7S0/mxEVgNXDUoFTaTkSPT1FdomZkB2wiRiPiXiBgHnBkRO+bHuIiYFBGnDFKNzWPUqDR1iJiZAdUfzvqFpLEAkt4n6TuSdqljXc2pFCKbNjW2DjOzJlFtiPwQeE7S3sDngIeBC4puVNJ4SVdIekDS/ZIOkjRR0mJJD+XphNxXks6S1CFpmaT9ytazIPd/SNKCovVUzSFiZtZFtSGyJSICmA/874j4ATCuH9v9N+BXEfFqYG/gfuBk4LqImA1cl58DHA7Mzo8TSYGGpInAqcAbgAOAU0vBUzcOETOzLqoNkY2STgHeD1wtaTtgRJENStoJeBNwLkBEvBgR60kBtTB3WwgcmefnAxdEchMwXtI04FBgcUSsjYh1wGLgsCI1Vc0hYmbWRbUh8h5gE/ChiHgCmAmcWXCbuwJrgJ9KukPST/L5lqkRsSr3eQKYmudnAMvLXr8it/XWXj8OETOzLqoKkRwcFwM7SXo78EJEFD0n0g7sB/wwIvYFnuXlQ1el7QUwYPcWkXSipKWSlq5Zs6b4ihwiZmZdVPuJ9WOAW4CjgWOAmyUdVXCbK4AVEXFzfn4FKVSezIepyNPVeflKYOey18/Mbb21V4iIcyJiTkTMmTJlSsGycYiYmXVT7eGsLwGvj4gFEXE86UT2/yqywTyqWS7pv+amucB9wCKgdIXVAl7+MOMi4Ph8ldaBwIZ82OtaYJ6kCfmE+rzcVj8OETOzLtqr7LddRKwue/4U/bt54yeBiyWNBB4BPpjXd7mkE4DHSCMegF8CRwAdwHO5LxGxVtLpwK2532kRsbYfNW2bQ8TMrItqQ+RXkq4FLsnP30P65V5IRNwJzOlh0dwe+gZwUi/rOQ84r2gdNXOImJl10WeISNqDdNXUP0l6N3BwXvRH0on24cUhYmbWxbZGIt8DTgGIiCuBKwEk/U1e9o461tZ8HCJmZl1s67zG1Ii4u3tjbptVl4qamUPEzKyLbYXI+D6WjR7AOlqDQ8TMrItthchSSR/p3ijpw8Bt9SmpibXno38OETMzYNvnRD4N/FzScbwcGnOAkcC76lhXc5LSaMQhYmYGbCNEIuJJ4G8lHQK8NjdfHRHX172yZuUQMTPrVNXnRCJiCbCkzrW0BoeImVmn/nzqfHhyiJiZdXKI1MohYmbWySFSK4eImVknh0itRo2CF19sdBVmZk3BIVIrj0TMzDo5RGrlEDEz6+QQqZVDxMysk0OkVg4RM7NODpFaOUTMzDo5RGrlEDEz6+QQqdX228Pzzze6CjOzpuAQqdWYMQ4RM7PMIVKr0aMdImZmmUOkVmPGpHMiW7c2uhIzs4ZziNRqdP5W4BdeaGwdZmZNwCFSqzFj0vS55xpbh5lZE3CI1Ko0EvF5ETMzh0jNPBIxM+vkEKmVQ8TMrJNDpFY+nGVm1skhUiuPRMzMOjlEauWRiJlZJ4dIrTwSMTPr5BCplUciZmadHCK18kjEzKyTQ6RWHomYmXVyiNSqFCIeiZiZNS5EJLVJukPSL/LzXSXdLKlD0mWSRub2Ufl5R14+q2wdp+T2ByUdOiiFt7fDiBEeiZiZ0diRyKeA+8uefxP4bkTsAawDTsjtJwDrcvt3cz8k7QkcC+wFHAacLaltUCofM8YjETMzGhQikmYCbwN+kp8LeAtwRe6yEDgyz8/Pz8nL5+b+84FLI2JTRPwZ6AAOGJQdGD3aIWJmRuNGIt8DvgC8lJ9PAtZHxJb8fAUwI8/PAJYD5OUbcv/O9h5e04WkEyUtlbR0zZo1/a/eX5FrZgY0IEQkvR1YHRG3DdY2I+KciJgTEXOmTJnS/xX6cJaZGQDtDdjmG4F3SjoC2B7YEfg3YLyk9jzamAmszP1XAjsDKyS1AzsBT5W1l5S/pr7GjoVnnhmUTZmZNbNBH4lExCkRMTMiZpFOjF8fEccBS4CjcrcFwFV5flF+Tl5+fUREbj82X721KzAbuGVQdmLcONi4cVA2ZWbWzBoxEunN/wQulfQ14A7g3Nx+LnChpA5gLSl4iIh7JV0O3AdsAU6KiK2DUum4cbBq1aBsysysmTU0RCLiBuCGPP8IPVxdFREvAEf38vqvA1+vX4W98EjEzAzwJ9aLcYiYmQEOkWJ23DGFSESjKzEzayiHSBHjxsGWLbBpU6MrMTNrKIdIEePGpakPaZnZMOcQKcIhYmYGOESKcYiYmQEOkWJKIfL0042tw8yswRwiRXgkYmYGOESKcYiYmQEOkWJ23DFNHSJmNsw5RIrwSMTMDHCIFLPDDmnqE+tmNsw5RIpob0+jkfXrG12JmVlDOUSKmjgR1q5tdBVmZg3lEClq4kR46qlGV2Fm1lAOkaImTfJIxMyGPYdIUT6cZWbmECnMIWJm5hAprBQi/mIqMxvGHCJFTZwIW7f6syJmNqw5RIqaNClNfUjLzIYxh0hREyemqUPEzIYxh0hRpRDxZ0XMbBhziBQ1ZUqarl7d2DrMzBrIIVLUtGlpumpVY+swM2sgh0hR48bB2LEOETMb1hwiRUlpNPL4442uxMysYRwi/TF9ukciZjasOUT6Y/p0j0TMbFhziPTHtGlpJOJbn5jZMOUQ6Y/p0+HZZ/1d62Y2bDlE+mPmzDRdvryxdZiZNYhDpD922y1NH364sXWYmTWIQ6Q/9tgjTTs6GluHmVmDDHqISNpZ0hJJ90m6V9KncvtESYslPZSnE3K7JJ0lqUPSMkn7la1rQe7/kKQFg70vTJwI48d7JGJmw1YjRiJbgM9FxJ7AgcBJkvYETgaui4jZwHX5OcDhwOz8OBH4IaTQAU4F3gAcAJxaCp5BtfvuDhEzG7YGPUQiYlVE3J7nNwL3AzOA+cDC3G0hcGSenw9cEMlNwHhJ04BDgcURsTYi1gGLgcMGb0+y2bPhgQcGfbNmZs2goedEJM0C9gVuBqZGROnj308AU/P8DKD88qcVua239sG1997w2GOwfv2gb9rMrNEaFiKSdgB+Bnw6Irp8x2xEBDBgn+CTdKKkpZKWrlmzZqBWm+yzT5reddfArtfMrAU0JEQkjSAFyMURcWVufjIfpiJPS1/UsRLYuezlM3Nbb+0VIuKciJgTEXOmlL4HZKDsu2+a3nHHwK7XzKwFNOLqLAHnAvdHxHfKFi0CSldYLQCuKms/Pl+ldSCwIR/2uhaYJ2lCPqE+L7cNrqlT04cO//jHQd+0mVmjtTdgm28E3g/cLenO3PZF4AzgckknAI8Bx+RlvwSOADqA54APAkTEWkmnA7fmfqdFRGO+8PyQQ+Caa+Cll2A7f/TGzIaPQQ+RiPg9oF4Wz+2hfwAn9bKu84DzBq66gubOhQsvhGXLXj5HYmY2DPjP5oFw+OHQ1gaXXdboSszMBpVDZCC84hUwbx5cdBFs3tzoaszMBo1DZKB8/OOwYgUsXLjtvmZmQ4RDZKC87W1w0EHwhS/Ao482uhozs0HRiKuzhiYpnVzff384+GD4/vdTsIwc2bXf5s2wYUP6hPv69bBu3cvz5Y9nn4UtW/p+RLz8gK7T/rQNpHp862MrrLMVaqzHOl1j865z8mT4wx8Gdp04RAbW7rvDjTfCMcfAu98No0alr9Ddbjt4/nl4+ukUDn1pa0t3Bh47FkaMgPb2nh9tbS9fTiylR2l+INoG0kCvr1XW2Qo11mOdrrE517nTTgO3rjIOkYG2995w771w9dXw+9/DE0+kvyhGj05v4vjxfT/Gjq3PD6OZWR04ROqhvR3mz08PM7MhzCfWzcysMIeImZkV5hAxM7PCHCJmZlaYQ8TMzApziJiZWWEOETMzK8whYmZmhSnqcc+XJiZpDembE4uYDPx1AMtplKGyH+B9aVZDZV+Gyn5A//Zll4iY0tOCYRci/SFpaUTMaXQd/TVU9gO8L81qqOzLUNkPqN+++HCWmZkV5hAxM7PCHCK1OafRBQyQobIf4H1pVkNlX4bKfkCd9sXnRMzMrDCPRMzMrDCHiJmZFeYQqYKkwyQ9KKlD0smNrqcakh6VdLekOyUtzW0TJS2W9FCeTsjtknRW3r9lkvZrcO3nSVot6Z6ytpprl7Qg939I0oIm2Y+vSlqZ35c7JR1RtuyUvB8PSjq0rL3hP3+Sdpa0RNJ9ku6V9Knc3orvS2/70lLvjaTtJd0i6a68H/+c23eVdHOu6TJJI3P7qPy8Iy+fta39q0pE+NHHA2gDHgZ2A0YCdwF7NrquKup+FJjcre1bwMl5/mTgm3n+COAaQMCBwM0Nrv1NwH7APUVrByYCj+TphDw/oQn246vA53vou2f+2RoF7Jp/5tqa5ecPmAbsl+fHAX/KNbfi+9LbvrTUe5P/bXfI8yOAm/O/9eXAsbn9R8DH8vzHgR/l+WOBy/rav2rr8Ehk2w4AOiLikYh4EbgUaNXvvZ0PLMzzC4Ejy9oviOQmYLykaQ2oD4CI+C2wtltzrbUfCiyOiLURsQ5YDBxW9+LL9LIfvZkPXBoRmyLiz0AH6WevKX7+ImJVRNye5zcC9wMzaM33pbd96U1Tvjf53/aZ/HREfgTwFuCK3N79PSm9V1cAcyWJ3vevKg6RbZsBLC97voK+f+CaRQC/lnSbpBNz29SIWJXnnwCm5vlW2Mdaa2/mffpEPsRzXunwDy20H/kwyL6kv3xb+n3pti/QYu+NpDZJdwKrSYH8MLA+Irb0UFNnvXn5BmAS/dwPh8jQdXBE7AccDpwk6U3lCyONY1vy+u5Wrh34IbA7sA+wCvh2Q6upkaQdgJ8Bn46Ip8uXtdr70sO+tNx7ExFbI2IfYCZp9PDqwa7BIbJtK4Gdy57PzG1NLSJW5ulq4OekH7AnS4ep8nR17t4K+1hr7U25TxHxZP6P/xLwY14+bND0+yFpBOmX7sURcWVubsn3pad9aeX3JiLWA0uAg0iHDtt7qKmz3rx8J+Ap+rkfDpFtuxWYna94GEk6IbWowTX1SdJYSeNK88A84B5S3aWrYRYAV+X5RcDx+YqaA4ENZYcomkWttV8LzJM0IR+WmJfbGqrbuaZ3kd4XSPtxbL6CZldgNnALTfLzl4+dnwvcHxHfKVvUcu9Lb/vSau+NpCmSxuf50cDfk87vLAGOyt26vyel9+oo4Po8euxt/6ozWFcStPKDdKXJn0jHG7/U6HqqqHc30tUWdwH3lmomHf+8DngI+A0wMbcL+EHev7uBOQ2u/xLS4YTNpOOzJxSpHfgQ6SRhB/DBJtmPC3Ody/J/3mll/b+U9+NB4PBm+vkDDiYdqloG3JkfR7To+9LbvrTUewO8Drgj13sP8JXcvhspBDqAfwdG5fbt8/OOvHy3be1fNQ/f9sTMzArz4SwzMyvMIWJmZoU5RMzMrDCHiJmZFeYQMTOzwhwi1tIkhaRvlz3/vKSvDtC6z5d01LZ79ns7R0u6X9KSbu3TJV2R5/cpv6vsAGxzvKSP97Qts1o4RKzVbQLeLWlyowspV/aJ4WqcAHwkIg4pb4yIxyOiFGL7kD6TMFA1jCfd1bWnbZlVzSFirW4L6bujP9N9QfeRhKRn8vTNkm6UdJWkRySdIek4pe9muFvS7mWreaukpZL+JOnt+fVtks6UdGu+Wd//KFvv7yQtAu7roZ735vXfI+mbue0rpA+/nSvpzG79Z+W+I4HTgPcofc/Fe/JdCc7LNd8haX5+zQckLZJ0PXCdpB0kXSfp9rzt0l1mzwB2z+s7s7StvI7tJf00979D0iFl675S0q+UvgvkW2X/HufnWu+WVPFe2NBVy19LZs3qB8Cy0i+1Ku0NvIZ0q/ZHgJ9ExAFKX1D0SeDTud8s0j2UdgeWSNoDOJ50G4/XSxoF/EHSr3P//YDXRrqldidJ04FvAvsD60h3WD4yIk6T9BbS91gs7anQiHgxh82ciPhEXt83SLet+FC+9cUtkn5TVsPrImJtHo28KyKezqO1m3LInZzr3Cevb1bZJk9Km42/kfTqXOur8rJ9SHe93QQ8KOn7wCuAGRHx2ryu8X38u9sQ45GItbxId2C9APjHGl52a6TvldhEut1DKQTuJgVHyeUR8VJEPEQKm1eT7vd0vNItuG8m3fpjdu5/S/cAyV4P3BARayLdhvti0pdWFTUPODnXcAPplhavzMsWR0Tpe0wEfEPSMtJtSWbw8u3ae3MwcBFARDwAPAaUQuS6iNgQES+QRlu7kP5ddpP0fUmHAU/3sE4bojwSsaHie8DtwE/L2raQ/1CStB3p2+dKNpXNv1T2/CW6/r/ofl+gIP1i/mREdLlxoKQ3A88WKb4AAf89Ih7sVsMbutVwHDAF2D8iNkt6lBQ4RZX/u20F2iNinaS9SV849VHgGNL9sWwY8EjEhoT8l/flpJPUJY+SDh8BvJP0zW+1OlrSdvk8yW6kG9RdC3xM6XbiSHqV0t2S+3IL8HeSJktqA94L3FhDHRtJX+Vaci3wSUnKNezby+t2AlbnADmENHLoaX3lfkcKH/JhrFeS9rtH+TDZdhHxM+DLpMNpNkw4RGwo+TZQfpXWj0m/uO8ifc9CkVHCX0gBcA3w0XwY5yekQzm355PR/4dtjOoj3Qb9ZNJtuu8CbouIq/p6TTdLgD1LJ9aB00mhuEzSvfl5Ty4G5ki6m3Qu54Fcz1Okczn3dD+hD5wNbJdfcxnwgXzYrzczgBvyobWLgFNq2C9rcb6Lr5mZFeaRiJmZFeYQMTOzwhwiZmZWmEPEzMwKc4iYmVlhDhEzMyvMIWJmZoX9fxVgbxzXMsL0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "Multivariable_Linear_Regression(X_transform,y, 0.01, 3000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "id": "58dac62b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[151.75490936],\n",
       "       [ -0.29632997],\n",
       "       [-11.22544074],\n",
       "       [ 25.14272388],\n",
       "       [ 15.26133206],\n",
       "       [ -3.19980516],\n",
       "       [ -5.24870086],\n",
       "       [ -9.91778104],\n",
       "       [  5.26898257],\n",
       "       [ 22.26433815],\n",
       "       [  3.62147395]])"
      ]
     },
     "execution_count": 229,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAm70lEQVR4nO3deZxcVZ338c83K4kgCUmMkIUkkIAJD2Bo2YYRFCYEFMF5AHEjyDaiog5uQRzF5QERV1xgkLDKwyLKEDcghoDOyNYgELZAZEtCSEIWVoWE/OaPcxoqnU4v1V11q6q/79frvurWuafu/Z1Up399z7n3XEUEZmZm5ehTdABmZla/nETMzKxsTiJmZlY2JxEzMyubk4iZmZXNScTMzMrmJGK2CZJOkrRM0ouShlXxuF+WdEG1jldpkkLS9kXHYZXhJGI1T9KHJDXnX+ZLJf1B0j7d3OcTkg5oZ3t/4PvAtIjYPCJWdud47RxnP0mLS8si4oyIOL5Cx9ta0s8lPZ3/PR+TdLGkHStxPGt8TiJW0ySdAvwQOAMYCYwFfgYcWuFDjwQ2Ax6o8HGqJp9N/QUYDPwzsAUwFbgF+JdNfKZf1QK0+hQRXrzU5AJsCbwIHNFOnYGkJPN0Xn4IDMzbhgO/BdYAq4A/k/5wugxYD/w97/+LrfY5CXgJiLz9JmBcft+vpN7NwPF5/Rjgv4HvAquBx4GDSupuBVyUY1wN/BfwphzD+nycF4FtgNOBX5R89n2kZLYmH/NtJdueAD4P3Ac8B1wFbLaJf6tvAfcCfdr592xp53HAU8CfcvkvgWfyMf4ETCn5zMXAecAc4AVSUtq2ZHsAHwcezW34KaCif7689MziMxGrZXuRzgaubafOacCewK7ALsDuwFfyts8Bi4ERpDOLLwMRER8l/YI8JFJX1XdKdxgRjwBT8tshEfHuTsa7B7CAlLy+A8ySpLztMtIZwBTgLcAPIuIl4CDg6RzH5hHxdOkOJU0CrgA+m9vxe+A3kgaUVDsSmA6MB3YmJbS2HABcGxHrO9GWfYG3AQfm938AJubY7wYub1X/w8A3c9vvaWP7e4F35PiOLNmv1TknEatlw4BnI2JdO3U+DHwjIpZHxArg68BH87a1wNakv4rXRsSfI6KSk8U9GRE/j4jXgEvysUdK2pqULD4eEatzLLd0cp8fAH4XEXMiYi3pTGcQsHdJnXMi4umIWAX8hpRQ2zKcdDYBgKT3SVoj6QVJN7aqe3pEvBQRfweIiAsj4oWIeIV0prSLpC1L6v8uIv6Ut58G7CVpTMn2b0fEmoh4CpjXToxWZ5xErJatBIZ30C+/DfBkyfsncxnA2cBC4MY8gDyzMmG+7vVf0BHxcl7dHBgDrIqI1WXsc4P25bOIRcCoto4LvJyP2ZaVpMTWsq/ZETEE+HdgQKu6i1pWJPWV9G1Jf5P0PKkLDVJS2qh+RLxI6j7cpmR7Z2O0OuMkYrXsVuAV4LB26jwNbFvyfmwuI//l/LmImEAaVzhF0v65XlfPSF7Kr4NLyt7ayc8uAraSNKSNbR3FsUH7cvfYGGBJJ49dai5wmKTO/L8vjetDpAsZDiCNU41rCaekzutnHZI2J40BbdA1Z43JScRqVkQ8B3wV+KmkwyQNltRf0kGSWsYxrgC+ImmEpOG5/i8AJL1X0vb5F+9zwGukQWyAZcCELsSygvSL+yP5L/Njge06+dmlpDGFn0kamtvwzpI4hrXqGip1NfAeSfvny44/R0qsf+ls7CW+DwwFLpO0nZIt6LhraYt8zJWkJHpGG3UOlrRPHqv5JnBbRCxqo541GCcRq2kR8T3gFNJg+QrSX/WfIl3dBOmKo2bS1UnzSYO+38rbJgJ/JF31dCvws4iYl7edSUo+ayR9vpPhnAB8gfTLdApd+0X+UdIYzcPActJAORHxMCkRPpZjKe0CIiIWAB8Bfgw8CxxCuiDg1S4cu2Vfz5IuQvgH6UqyF0iD4FsAJ7Xz0UtJXWpLgAeB29qo8/+Br5G6sXbLMVsvoMqOM5pZo5N0MbA4Ir7SUV1rPD4TMTOzsjmJmJlZ2SqWRCRdKGm5pPtLys6W9LCk+yRdW3q1iqRTJS2UtEDSgSXl03PZwtJLNCWNl3R7Lr+q1c1XZlYlEXGMu7J6r0qeiVxMuou21Bxgp4jYGXgEOBVA0mTgKNJg5XTSVSx9JfUlTZFwEDAZ+GCuC3AW6a7f7UnTSBxXwbaYmVkbKja5WkT8SdK4VmWld8XeBhye1w8Frsx3uz4uaSFp+gqAhRHxGICkK4FDJT0EvJt0/Tqku4NPB87tKK7hw4fHuHHjOqpmZmYl7rrrrmcjYkTr8iJn6DyWNFkcpLtvSy8bXMwbd+QualW+B2k6jDUl02GU1t+IpBOBEwHGjh1Lc3Nzt4M3M+tNJD3ZVnkhA+uSTgPWsfEkbRUREedHRFNENI0YsVEiNTOzMlX9TETSMaQZPfcvmQxvCSXTJgCjeWNah7bKVwJDJPXLZyOl9c3MrEqqeiYiaTrwReB9JRPUAcwGjpI0UNJ40p3GdwB3AhPzlVgDSIPvs3PymccbYyozgOuq1Q4zM0sqeYnvFaSpJnaQtFjSccBPSFMszJF0j6TzACLiAdIcQQ8C1wOfjIjX8lnGp4AbgIeAq3NdgC+RJtRbSBojmVWptpiZWdt63bQnTU1N4YF1M7OukXRXRDS1Lvcd62ZmVjYnETMzK5uTSGf95Cdw1VUd1zMz60WcRDpr1iy47LKiozAzqylOIp01diw89VTRUZiZ1RQnkc7adlsnETOzVpxEOmvsWHjuubSYmRngJNJ5Y8emV5+NmJm9zkmks5xEzMw24iTSWdtum16dRMzMXuck0lkjR0L//vBkm1Pqm5n1Sk4indWnD4wZ4zMRM7MSTiJd4XtFzMw24CTSFb5XxMxsA04iXTF2LCxZAmvXFh2JmVlNcBLpirFjYf16ePrpoiMxM6sJTiJd4XtFzMw24CTSFb5XxMxsA04iXTFmTHr1vSJmZoCTSNcMHgzDhzuJmJllTiJdNWECPP540VGYmdUEJ5GumjABHnus6CjMzGqCk0hXTZiQurPWrSs6EjOzwjmJdNWECSmBLF5cdCRmZoVzEumqCRPS69/+VmwcZmY1wEmkq1qSiMdFzMycRLps9Gjo189JxMwMJ5Gu69sXxo1zEjEzw0mkPNtt5yRiZoaTSHl8r4iZGVDBJCLpQknLJd1fUraVpDmSHs2vQ3O5JJ0jaaGk+yRNLfnMjFz/UUkzSsp3kzQ/f+YcSapUWzYyYQKsWgVr1lTtkGZmtaiSZyIXA9Nblc0E5kbERGBufg9wEDAxLycC50JKOsDXgD2A3YGvtSSeXOeEks+1PlbltFyh5elPzKyXq1gSiYg/AataFR8KXJLXLwEOKym/NJLbgCGStgYOBOZExKqIWA3MAabnbW+OiNsiIoBLS/ZVeb7M18wMqP6YyMiIWJrXnwFG5vVRwKKSeotzWXvli9sob5OkEyU1S2pesWJF91oAMH58evUNh2bWyxU2sJ7PIKJKxzo/IpoiomnEiBHd3+GWW8KwYU4iZtbrVTuJLMtdUeTX5bl8CTCmpN7oXNZe+eg2yqtn4kRYuLCqhzQzqzXVTiKzgZYrrGYA15WUH52v0toTeC53e90ATJM0NA+oTwNuyNuel7Rnvirr6JJ9VcekSfDII1U9pJlZrankJb5XALcCO0haLOk44NvAv0h6FDggvwf4PfAYsBD4OfAJgIhYBXwTuDMv38hl5DoX5M/8DfhDpdrSpkmT0ky+L71U1cOamdWSfpXacUR8cBOb9m+jbgCf3MR+LgQubKO8GdipOzF2y6RJ6XXhQthll8LCMDMrku9YL1dLEnGXlpn1Yk4i5dp++/S6YEGxcZiZFchJpFxvehOMGeMzETPr1ZxEusNXaJlZL+ck0h2TJqXurKjKPZNmZjXHSaQ7Jk1KM/muXFl0JGZmhXAS6Q5foWVmvZyTSHc4iZhZL+ck0h3jxkH//r7M18x6LSeR7ujXLz1v3UnEzHopJ5Hu2mEHJxEz67WcRLpr8uQ0JvLqq0VHYmZWdU4i3TVlCqxb52eLmFmv5CTSXZMnp9cHHig2DjOzAjiJdNeOO4IEDz5YdCRmZlXnJNJdgwbBhAk+EzGzXslJpCdMmeIzETPrlZxEekLLFVpr1xYdiZlZVTmJ9IQpU1ICefTRoiMxM6sqJ5Ge0HKFlru0zKyXcRLpCS1XaHlw3cx6GSeRnjB4cLpCy2ciZtbLOIn0lMmTfSZiZr2Ok0hPmTLFc2iZWa/jJNJTdt45XaH18MNFR2JmVjVOIj1ll13S6733FhuHmVkVOYn0lEmTYOBAJxEz61WcRHpKv36w005wzz1FR2JmVjVOIj1p113TmUhE0ZGYmVWFk0hP2mUXePZZWLq06EjMzKrCSaQneXDdzHqZQpKIpH+X9ICk+yVdIWkzSeMl3S5poaSrJA3IdQfm9wvz9nEl+zk1ly+QdGARbdnAzjunVycRM+slqp5EJI0CPg00RcROQF/gKOAs4AcRsT2wGjguf+Q4YHUu/0Guh6TJ+XNTgOnAzyT1rWZbNjJkCGy7rQfXzazXKKo7qx8wSFI/YDCwFHg3cE3efglwWF4/NL8nb99fknL5lRHxSkQ8DiwEdq9O+O1oGVw3M+sFqp5EImIJ8F3gKVLyeA64C1gTEetytcXAqLw+CliUP7su1x9WWt7GZzYg6URJzZKaV6xY0bMNam2XXdL0J3//e2WPY2ZWA4rozhpKOosYD2wDvInUHVUxEXF+RDRFRNOIESMqeaiURNavh/nzK3scM7MaUER31gHA4xGxIiLWAr8G/gkYkru3AEYDS/L6EmAMQN6+JbCytLyNzxRnt93S6113FRuHmVkVFJFEngL2lDQ4j23sDzwIzAMOz3VmANfl9dn5PXn7TRERufyofPXWeGAicEeV2rBpY8fC8OHQ3Fx0JGZmFdev4yo9KyJul3QNcDewDvgrcD7wO+BKSd/KZbPyR2YBl0laCKwiXZFFRDwg6WpSAloHfDIiXqtqY9oiQVOTk4iZ9QqKXjZFR1NTUzRX+hf8V78KZ5wBzz+fnnpoZlbnJN0VEU2ty33HeiU0NcFrr/l+ETNreE4ildCUk7W7tMyswTmJVMI228DWWzuJmFnDcxKplHe8w0nEzBqek0ilNDWl562/8ELRkZiZVYyTSKU0NaWHU919d9GRmJlVjJNIpbQMrt95Z7FxmJlVkJNIpYwYAePGwR3F30RvZlYpTiKVtPfe8D//42eum1nDchKppL32gqefhkWLOq5rZlaHnEQqae+90+uttxYbh5lZhTiJVNLOO6e5s/7yl6IjMTOrCCeRSurXD3bf3UnEzBqWk0il7b13mojx5ZeLjsTMrMc5iVTa3nvDunWeAsXMGlKnkoikyzpTZm3Yc8/06i4tM2tAnT0TmVL6RlJfYLeeD6cBDRsGO+zgK7TMrCG1m0QknSrpBWBnSc/n5QVgOW88A906svfe6UzENx2aWYNpN4lExJkRsQVwdkS8OS9bRMSwiDi1SjHWv332gWefTbP6mpk1kM52Z/1W0psAJH1E0vclbVvBuBrLvvum15tvLjQMM7Oe1tkkci7wsqRdgM8BfwMurVhUjWbCBBg92knEzBpOZ5PIuogI4FDgJxHxU2CLyoXVYCTYbz+45RaPi5hZQ+lsEnlB0qnAR4HfSeoD9K9cWA1o331h2TJYsKDoSMzMekxnk8gHgFeAYyPiGWA0cHbFompE++2XXt2lZWYNpFNJJCeOy4EtJb0X+EdEeEykK7bbDrbZJnVpmZk1iM7esX4kcAdwBHAkcLukwysZWMNpGRe5+WaPi5hZw+hsd9ZpwDsiYkZEHA3sDvxH5cJqUPvuC888A488UnQkZmY9orNJpE9ELC95v7ILn7UWLeMi8+YVGoaZWU/pbCK4XtINko6RdAzwO+D3lQurQU2cCGPGwB//WHQkZmY9ol97GyVtD4yMiC9I+ldgn7zpVtJAu3WFBNOmwa9+laaH79fuP7+ZWc3r6Ezkh8DzABHx64g4JSJOAa7N28oiaYikayQ9LOkhSXtJ2krSHEmP5tehua4knSNpoaT7JE0t2c+MXP9RSTPKjaeqpk2DNWv8fBEzawgdJZGRETG/dWEuG9eN4/4IuD4idgR2AR4CZgJzI2IiMDe/BzgImJiXE0lTsCBpK+BrwB6kgf6vtSSemrb//umM5IYbio7EzKzbOkoiQ9rZNqicA0raEngnMAsgIl6NiDWkKVUuydUuAQ7L64cCl0ZyGzBE0tbAgcCciFgVEauBOcD0cmKqqmHDoKkJbryx6EjMzLqtoyTSLOmE1oWSjgfuKvOY44EVwEWS/irpgjxD8MiIWJrrPAOMzOujgEUln1+cyzZVvhFJJ0pqltS8YsWKMsPuQdOmwe23p24tM7M61lES+SzwMUk3S/peXm4BjgM+U+Yx+wFTgXMj4u3AS7zRdQVAnuyxx+7Ii4jzI6IpIppGjBjRU7st37Rp8NprvtTXzOpeRw+lWhYRewNfB57Iy9cjYq88FUo5FgOLI+L2/P4aUlJZlrupyK8t96UsAcaUfH50LttUee3bc0/YfHN3aZlZ3evs3FnzIuLHebmpOwfMyWeRpB1y0f7Ag8BsoOUKqxm88fjd2cDR+SqtPYHncrfXDcA0SUPzgPq0XFb7BgyAd70Lrr/eU6CYWV0r6kaFk4HLJQ0AHgM+RkpoV0s6DniSNEcXpJsaDwYWAi/nukTEKknfBO7M9b4REauq14RuOvhg+M1v4MEHYcqUoqMxMyuLopf9JdzU1BTNtXCPxuLF6e71M8+EmTM7rm9mViBJd0VEU+tyz39VlNGjYerUdDZiZlannESKdMghcOutUAuXHZuZlcFJpEiHHJIG1n/vuSzNrD45iRRp6tT0tEN3aZlZnXISKZIE731vmkfrlVeKjsbMrMucRIp2yCHw4ovpsblmZnXGSaRo+++f7l6/5pqiIzEz6zInkaINGpTORq69FtauLToaM7MucRKpBUccAStXwi23FB2JmVmXOInUgunTU5fWL39ZdCRmZl3iJFILBg1KV2n9+tfp2etmZnXCSaRWHH44PPusu7TMrK44idSKgw6CwYPdpWVmdcVJpFYMHpyu0vrVr3yVlpnVDSeRWvLhD6cureuvLzoSM7NOcRKpJdOnw/DhcNllRUdiZtYpTiK1pH9/+OAHYfZsWL266GjMzDrkJFJrjj46TcboAXYzqwNOIrVmt91gxx3dpWVmdcFJpNZI6Wzkv/8bHnus6GjMzNrlJFKLPvKRlEwuuqjoSMzM2uUkUovGjElXas2a5XtGzKymOYnUqn/7N1i6FH7726IjMTPbJCeRWvWe98CoUfCf/1l0JGZmm+QkUqv69YPjj4cbb4THHy86GjOzNjmJ1LLjj08D7D//edGRmJm1yUmklo0enbq1Zs1KNyCamdUYJ5Fad/LJsHw5XHFF0ZGYmW3ESaTWHXAA7LQTfP/7EFF0NGZmG3ASqXUSnHIKzJ8Pc+cWHY2Z2QYKSyKS+kr6q6Tf5vfjJd0uaaGkqyQNyOUD8/uFefu4kn2cmssXSDqwoKZU3oc+BCNHwg9+UHQkZmYbKPJM5DPAQyXvzwJ+EBHbA6uB43L5ccDqXP6DXA9Jk4GjgCnAdOBnkvpWKfbqGjgQPvEJ+P3v4aGHOq5vZlYlhSQRSaOB9wAX5PcC3g1ck6tcAhyW1w/N78nb98/1DwWujIhXIuJxYCGwe1UaUISTToLNNoPvfKfoSMzMXlfUmcgPgS8C6/P7YcCaiFiX3y8GRuX1UcAigLz9uVz/9fI2PrMBSSdKapbUvGLFih5sRhWNGJGmQrnsMt98aGY1o+pJRNJ7geURcVe1jhkR50dEU0Q0jRgxolqH7Xlf+AL07Qtnnll0JGZmQDFnIv8EvE/SE8CVpG6sHwFDJPXLdUYDS/L6EmAMQN6+JbCytLyNzzSmUaPSXewXXwxPPVV0NGZm1U8iEXFqRIyOiHGkgfGbIuLDwDzg8FxtBnBdXp+d35O33xQRkcuPyldvjQcmAndUqRnF+dKX0qvHRsysBtTSfSJfAk6RtJA05jErl88ChuXyU4CZABHxAHA18CBwPfDJiHit6lFX29ixcMwxaT6tJ58sOhoz6+UUvewu6Kampmhubi46jO5ZtAgmTYIjj4RLLum4vplZN0m6KyKaWpfX0pmIddaYMfDpT6crte69t+hozKwXcxKpVzNnwpAh6dXMrCBOIvVq6FD48pfh+uvhppuKjsbMeiknkXr2qU/BttvCZz8L69Z1WN3MrKc5idSzzTZLkzLOnw8//WnR0ZhZL+QkUu8OOwwOPBC++lV45pmiozGzXsZJpN5J8OMfwz/+AV/8YtHRmFkv4yTSCCZOTPNqXXYZ/PGPRUdjZr2Ik0ijOO002GGHNLfWCy8UHY2Z9RJOIo1i0CC46KI0MaO7tcysSpxEGslee6XnsZ93np/HbmZV4STSaL75zTSv1jHHwMqVRUdjZg3OSaTRDBoEl18Oy5bBscdCL5tg08yqy0mkETU1wdlnw+zZcM45RUdjZg3MSaRRffrT8L73pUt/72j8Z3WZWTGcRBqVlK7WGjUK3v9+ePrpoiMyswbkJNLIttoKrrsOnnsuJZJ//KPoiMyswTiJNLqdd053st9xB5xwggfazaxHOYn0Bu9/f7r09xe/gFNPLToaM2sg/YoOwKrktNNgyRI46ywYPhw+//miIzKzBuAk0ltI8JOfwKpV6YqtrbZK95GYmXWDk0hv0rcvXHoprFmTJmqMgOOOKzoqM6tjHhPpbQYOhP/6r/Qgq+OPh3PPLToiM6tjTiK90aBBKZEccgh84hNpnMRXbZlZGZxEequBA+Gaa+Coo2DmzJRM1q0rOiozqzMeE+nNBgxIkzWOHw9nnglPPglXXglvfnPRkZlZnfCZSG/Xpw+ccQacfz7ceGOavHH+/KKjMrM64SRiyQknwLx58OKLsMce6SouM7MOOInYG/75n+Huu1MSmTEDPvQhP9jKzNrlJGIbeutbYc6cNE3KNdfATjvBb35TdFRmVqOcRGxj/frBV76SJm18y1vSc0mOOAKeeqroyMysxlQ9iUgaI2mepAclPSDpM7l8K0lzJD2aX4fmckk6R9JCSfdJmlqyrxm5/qOSZlS7LQ1v113hzjvTWcnvfgc77gjf+panlDez1xVxJrIO+FxETAb2BD4paTIwE5gbEROBufk9wEHAxLycCJwLKekAXwP2AHYHvtaSeKwHDRiQzkoeeggOPhj+4z9g++3hvPPg1VeLjs7MClb1JBIRSyPi7rz+AvAQMAo4FLgkV7sEOCyvHwpcGsltwBBJWwMHAnMiYlVErAbmANOr15JeZttt0xjJvHkwbhycdBLssANccIHPTMx6sULHRCSNA94O3A6MjIiledMzwMi8PgpYVPKxxblsU+VtHedESc2SmlesWNFzDeiN9tsP/vxn+MMf0pTyJ5wAY8fC6afDsmVFR2dmVVZYEpG0OfAr4LMR8XzptogIoMcmc4qI8yOiKSKaRowY0VO77b0kmD49DbzfdFO6JPjrX0/J5Kij4Prr4bXXio7SzKqgkCQiqT8pgVweEb/OxctyNxX5dXkuXwKMKfn46Fy2qXKrFgne9a50CfCCBfDxj6fLgw86KCWUmTPTwLwndzRrWEVcnSVgFvBQRHy/ZNNsoOUKqxnAdSXlR+ertPYEnsvdXjcA0yQNzQPq03KZFWHSJPjRj+Dpp9PYydSp8N3vwu67p4Ry8skwd67HT8wajKLKfyVK2gf4MzAfWJ+Lv0waF7kaGAs8CRwZEaty0vkJadD8ZeBjEdGc93Vs/izA/4uIizo6flNTUzQ3N/dgi2yTVq6E3/4Wrr0WbrghJZDNNoN99oH990/LrrtC//5FR2pmHZB0V0Q0bVRe7SRSNCeRgrz0Uho/mTs3Lfffn8oHDUpnLbvvnsZWmprSrMJ9fB+sWS1xEsmcRGrEM8/AzTfD7benAfq7736jq2vwYHjb22DKlLRMngwTJqRLiwcPLjJqs17LSSRzEqlRa9emKejvvhseeCCdqTzwACxdumG9kSPTmcr48enelbe+dePlzW9Og/5m1mM2lUT8UCqrDf37p26tqVM3LF+9Ot0t//jj8MQT6fXxx+G22+CXv2z7aYybbQZbbQVDh8KQIRsuQ4fCllumM5rSZdCgjd8PGJDmEevfPy39+kHfvo2RoCLeWNavr63XWoihNJbSf6+O1itVt6c+d9ZZPT4G6TMRq1/r16ck88wzbyzLlqWzl9Wr07JmzcbL+vXt7rZDLYml9LVfv5RcWi/Qdnlb26r5i9XqQ+ufk/bKOrO+bFn6A6msUHwmYo2mTx8YNiwtU6Z07jMR6cFbf/87vPzyxktp+auvpjOdtWs791r6133rvwY7s61Pn/QfvfVrW2U99dqyXqn9dzWOWnrdVMLvaL2n6tYJJxHrXSTYYou0mFm3+TpKMzMrm5OImZmVzUnEzMzK5iRiZmZlcxIxM7OyOYmYmVnZnETMzKxsTiJmZla2XjftiaQVpOeVlGM48GwPhlOkRmlLo7QD3JZa1Sht6W47to2IjZ4v3uuSSHdIam5r7ph61ChtaZR2gNtSqxqlLZVqh7uzzMysbE4iZmZWNieRrjm/6AB6UKO0pVHaAW5LrWqUtlSkHR4TMTOzsvlMxMzMyuYkYmZmZXMS6QRJ0yUtkLRQ0syi4+kMSU9Imi/pHknNuWwrSXMkPZpfh+ZySTont+8+SVPb33vFY79Q0nJJ95eUdTl2STNy/UclzaihtpwuaUn+bu6RdHDJtlNzWxZIOrCkvNCfQUljJM2T9KCkByR9JpfX3ffSTlvq8XvZTNIdku7Nbfl6Lh8v6fYc11WSBuTygfn9wrx9XEdt7FBEeGlnAfoCfwMmAAOAe4HJRcfVibifAIa3KvsOMDOvzwTOyusHA38ABOwJ3F5w7O8EpgL3lxs7sBXwWH4dmteH1khbTgc+30bdyfnnayAwPv/c9a2Fn0Fga2BqXt8CeCTHW3ffSzttqcfvRcDmeb0/cHv+974aOCqXnweclNc/AZyX148CrmqvjZ2JwWciHdsdWBgRj0XEq8CVwKEFx1SuQ4FL8volwGEl5ZdGchswRNLWBcQHQET8CVjVqrirsR8IzImIVRGxGpgDTK948K1soi2bcihwZUS8EhGPAwtJP3+F/wxGxNKIuDuvvwA8BIyiDr+XdtqyKbX8vUREvJjf9s9LAO8Grsnlrb+Xlu/rGmB/SWLTbeyQk0jHRgGLSt4vpv0fuFoRwI2S7pJ0Yi4bGRFL8/ozwMi8Xg9t7Grstd6mT+VungtbuoCok7bkLpC3k/7qrevvpVVboA6/F0l9Jd0DLCcl5b8BayJiXRtxvR5z3v4cMIxutMVJpHHtExFTgYOAT0p6Z+nGSOewdXl9dz3Hnp0LbAfsCiwFvldoNF0gaXPgV8BnI+L50m319r200Za6/F4i4rWI2BUYTTp72LGax3cS6dgSYEzJ+9G5rKZFxJL8uhy4lvTDtaylmyq/Ls/V66GNXY29ZtsUEcvyf/z1wM95o9ugptsiqT/pl+7lEfHrXFyX30tbbanX76VFRKwB5gF7kboP+7UR1+sx5+1bAivpRlucRDp2JzAxX+0wgDQYNbvgmNol6U2StmhZB6YB95PibrkaZgZwXV6fDRydr6jZE3iupIuiVnQ19huAaZKG5m6JabmscK3Gm95P+m4gteWofAXNeGAicAc18DOY+81nAQ9FxPdLNtXd97KpttTp9zJC0pC8Pgj4F9IYzzzg8Fyt9ffS8n0dDtyUzyA31caOVfNKgnpdSFeaPELqazyt6Hg6Ee8E0pUW9wIPtMRM6vucCzwK/BHYKpcL+Glu33ygqeD4ryB1J6wl9c0eV07swLGkAcKFwMdqqC2X5Vjvy/95ty6pf1puywLgoFr5GQT2IXVV3Qfck5eD6/F7aact9fi97Az8Ncd8P/DVXD6BlAQWAr8EBubyzfL7hXn7hI7a2NHiaU/MzKxs7s4yM7OyOYmYmVnZnETMzKxsTiJmZlY2JxEzMyubk4jVNUkh6Xsl7z8v6fQe2vfFkg7vuGa3j3OEpIckzWtVvo2ka/L6rqWzyvbAMYdI+kRbxzLrCicRq3evAP8qaXjRgZQquVu4M44DToiId5UWRsTTEdGSxHYl3ZPQUzEMIc3o2taxzDrNScTq3TrSs6P/vfWG1mcSkl7Mr/tJukXSdZIek/RtSR9Wei7DfEnblezmAEnNkh6R9N78+b6SzpZ0Z56s799K9vtnSbOBB9uI54N5//dLOiuXfZV089ssSWe3qj8u1x0AfAP4gNJzLj6QZyW4MMf8V0mH5s8cI2m2pJuAuZI2lzRX0t352C2zzH4b2C7v7+yWY+V9bCbpolz/r5LeVbLvX0u6XulZIN8p+fe4OMc6X9JG34U1rq78tWRWq34K3NfyS62TdgHeRpqm/THggojYXekBRScDn831xpHmUNoOmCdpe+Bo0jQe75A0EPgfSTfm+lOBnSJNp/06SdsAZwG7AatJMywfFhHfkPRu0nMsmtsKNCJezcmmKSI+lfd3BmnKimPztBd3SPpjSQw7R8SqfDby/oh4Pp+t3ZaT3Mwc5655f+NKDvnJdNj4P5J2zLFOytt2Jc16+wqwQNKPgbcAoyJip7yvIe38u1uD8ZmI1b1IM7BeCny6Cx+7M9JzJV4hTfXQkgTmkxJHi6sjYn1EPEpKNjuS5ns6Wmn67dtJU39MzPXvaJ1AsncAN0fEikhTcF9OemBVuaYBM3MMN5Omsxibt82JiJZnmAg4Q9J9pGlJRvHGdO2bsg/wC4CIeBh4EmhJInMj4rmI+AfpbGtb0r/LBEk/ljQdeL6NfVqD8pmINYofAncDF5WUrSP/oSSpD+npcy1eKVlfX/J+PRv+v2g9L1CQfjGfHBEbTBwoaT/gpXKCL4OA/xsRC1rFsEerGD4MjAB2i4i1kp4gJZxylf67vQb0i4jVknYhPXDq48CRpPmxrBfwmYg1hPyX99WkQeoWT5C6jwDeR3rqW1cdIalPHieZQJqc7gbgJKXpxJE0SWm25PbcAewrabikvsAHgVu6EMcLpEe5trgBOFmScgxv38TntgSW5wTyLtKZQ1v7K/VnUvIhd2ONJbW7TbmbrE9E/Ar4Cqk7zXoJJxFrJN8DSq/S+jnpF/e9pGcslHOW8BQpAfwB+HjuxrmA1JVzdx6M/k86OKuPNA36TNIU3fcCd0XEde19ppV5wOSWgXXgm6SkeJ+kB/L7tlwONEmaTxrLeTjHs5I0lnN/6wF94GdAn/yZq4BjcrffpowCbs5da78ATu1Cu6zOeRZfMzMrm89EzMysbE4iZmZWNicRMzMrm5OImZmVzUnEzMzK5iRiZmZlcxIxM7Oy/S+limS1GITDFQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "Multivariable_Linear_Regression(X_transform,y, 0.001, 3000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "389b2ba0",
   "metadata": {},
   "source": [
    "<h4 align=\"center\">Practical Issues: Feature Scaling</h4>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63cfdb42",
   "metadata": {},
   "source": [
    "Feature scaling is a technique that is used to standardize the range of independent variables or features of a data set. In linear regression, it is important to scale the features because the scale of the variables can affect the model's performance.\n",
    "\n",
    "Feature Scaling:\n",
    "   - If you have a problem with multiple features\n",
    "   - We should make sure those features have a similar scale. It means gradient descent will converge more quickly.\n",
    "**Example:**\n",
    "   - `x1` = size (0 - 2000 feet)\n",
    "   - `x2` = number of bedrooms (1-5)\n",
    "\n",
    "**Note:**\n",
    "This type of data generated `contours` as below if we plot $\\theta_1 \\; vs.\\; \\theta_2$ give a very tall and thin shape due to the huge range difference. Running gradient descent on this kind of cost function can take a long time to find the global minimum.\n",
    "<img src=\"images/p34.png\">\n",
    "\n",
    "Without scaling, the feature x1 will dominate the model because its range is much larger than the range of x2. As a result, the model may be biased towards x1. To avoid this issue, it is recommended to scale the features so that they have a similar range. There are several ways to scale the features, including:\n",
    "\n",
    "   - **Min-Max scaling**: This method scales the features to a specific range (e.g., 0-1). The transformed values are calculated as follows:\n",
    "\n",
    "$$ x_{scaled} = \\frac{x - x_{min}}{x_{max} - x_{min}} $$\n",
    "\n",
    "Where $x_{scaled}$ is the transformed value, $x$ is the original value, $x_{min}$ is the minimum value of the feature, and $x_{max}$ is the maximum value of the feature.\n",
    "\n",
    "   - **Standardization**: This method scales the features to have a mean of 0 and a standard deviation of 1. The transformed values are calculated as follows:\n",
    "\n",
    "$$ x_{scaled} = \\frac{x - \\mu}{\\sigma} $$\n",
    "\n",
    "Where $x_{scaled}$ is the transformed value, $x$ is the original value, $\\mu$ is the mean of the feature, and $\\sigma$ is the standard deviation of the feature.\n",
    "\n",
    "**Goal**:\n",
    "\n",
    "We want to get everything into -1 to +1 range (approximately). We want to avoid large ranges, small ranges or very different ranges from one another, for this purpose we use `mean normalization` or `standardization`. It is because `Standardization` is less sensitive to outliers because it is based on the `mean` and `standard deviation` of the feature, which are not influenced by extreme values as much as the minimum and maximum values. In contrast, `min-max` scaling is based on the minimum and maximum values of the feature, which can be affected by outliers. As a result, standardization is less likely to be affected by outliers than min-max scaling."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f4055b2",
   "metadata": {},
   "source": [
    "<h3 align=\"center\"> Gradient Descent Questions</h3>\n",
    "\n",
    "- **How to make sure gradient descent is working correctly.**\n",
    "- **How to choose learning rate $\\alpha$.**\n",
    "\n",
    "**1. Check the convergence of the cost function:**\n",
    "\n",
    "One way to check the convergence of the gradient descent algorithm is to plot the cost function (e.g., mean squared error) over the number of iterations. If the cost function is decreasing over time and eventually reaches a minimum, then the algorithm is working correctly.\n",
    "<img src=\"images/p35.png\" height=300x width=300x>\n",
    "\n",
    "**2. Compare the model's performance with other algorithms:**\n",
    "\n",
    "You can compare the performance of the gradient descent algorithm with other algorithms (e.g., least squares, normal equation) to see if the results are similar. If the results are significantly different, it may indicate an issue with the gradient descent algorithm.\n",
    "\n",
    "**3. Monitor the learning rate:**\n",
    "\n",
    "If the learning rate is too high, the gradient descent algorithm may diverge and never converge to a minimum. If the learning rate is too low, it may take a long time to converge. Monitoring the learning rate can help ensure that it is set to a reasonable value. If $\\alpha$ is too small,  slow convergence. If $\\alpha$ is too large, $J(\\theta)$ may not decrease on every iteration; may not converge. So to choose $\\alpha$, try different values like 0.0001, 0.001, 0.1, 1.\n",
    "\n",
    "<img src=\"images/p36.png\" height=300px width=300px>\n",
    "\n",
    "**4. Check the initialization of the weights:**\n",
    "\n",
    "The initialization of the weights can also affect the convergence of the gradient descent algorithm. If the weights are initialized to extreme values, the algorithm may converge slowly or not at all. It is generally a good idea to initialize the weights to small random values."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9d707a6",
   "metadata": {},
   "source": [
    "\n",
    "<h3 align=\"center\">Overfitting – Bias and Variance</h3>\n",
    "\n",
    "Overfitting and underfitting are two common issues that can affect the performance of a machine learning model.\n",
    "<img src=\"images/p37.png\">\n",
    "\n",
    "\n",
    "\n",
    "**Overfitting** occurs when the model is too complex and fits the training data too closely, resulting in poor generalization to new data. Overfitting can be caused by high variance and high bias. A model with high variance is sensitive to small fluctuations in the training data and may fit the training data too closely, leading to poor performance on new data. A model with high bias is unable to fit the training data well and may perform poorly on both the training and test sets.\n",
    "\n",
    "\n",
    "**Underfitting** occurs when the model is too simple and cannot capture the underlying pattern in the data, resulting in poor performance on both the training and test sets. Underfitting can be caused by high bias and low variance. A model with high bias is unable to fit the training data well and may perform poorly on both the training and test sets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f965a61f",
   "metadata": {},
   "source": [
    "#### Bias and Variance\n",
    "\n",
    "In regression, bias and variance are two types of error that can affect the performance of a model.\n",
    "\n",
    "**Bias** refers to the difference between the average prediction of a model and the true value of the target. A model with high bias is prone to underfitting, which means that it does not capture the underlying pattern in the data and performs poorly on both the training and test sets.\n",
    "\n",
    "**Variance** refers to the variability of the model's predictions for a given input. A model with high variance is prone to overfitting, which means that it fits the training data too closely and performs poorly on new data.\n",
    "\n",
    "**Underfitting** is characterized by poor performance on both the training and test sets and is caused by high bias. To reduce bias and improve the performance of an underfitted model, you can try the following techniques:\n",
    "- Use more complex models: Adding more parameters or features to the model can increase its flexibility and improve its ability to capture the underlying pattern in the data.\n",
    "- Increase the size of the training set: Increasing the size of the training set can help the model learn more about the data and generalize better.\n",
    "- Use regularization: Regularization is a technique that adds a penalty to the model's complexity to prevent overfitting. However, it can also reduce bias by allowing the model to be more flexible.\n",
    "\n",
    "**Overfitting** is characterized by poor performance on the test set and is caused by high variance. To reduce variance and improve the performance of an overfitted model, you can try the following techniques:\n",
    "- Use simpler models: Removing parameters or features from the model can reduce its complexity and prevent overfitting.\n",
    "- Use regularization: Regularization is a technique that adds a penalty to the model's complexity to prevent overfitting.\n",
    "- Increase the size of the training set: Increasing the size of the training set can help the model generalize better because it will have more data to learn from.\n",
    "\n",
    "<img src=\"images/p38.png\" height=300px width=300px align=\"right\">\n",
    "<img src=\"images/p39.png\" height=300px width=300px align=\"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832a4be5",
   "metadata": {},
   "source": [
    "#### How to overcome Underfitting and overfitting?\n",
    "To address overfitting and underfitting, you can try the following techniques:\n",
    "\n",
    "- **Use regularization:** Regularization is a technique that adds a penalty to the model's complexity to prevent overfitting.\n",
    "    - Reduce the number of features: Removing unnecessary or correlated features can reduce the model's complexity and prevent overfitting.\n",
    "    - Increase the size of the training set: Increasing the size of the training set can help the model generalize better because it will have more data to learn from.\n",
    "\n",
    "- **Use cross-validation:** Cross-validation is a technique that divides the training set into multiple folds and uses each fold as a test set to evaluate the model's performance. This can help identify overfitting and improve the model's generalization.\n",
    "- **Use different models:** Trying different models with different architectures and parameters can help identify the model that performs best for the data.\n",
    "- **Reduce the number of features**\n",
    "    - Manually select features\n",
    "    - Model selection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "931faa55",
   "metadata": {},
   "source": [
    "#### Manual feature selection\n",
    "- Manual feature selection is the process of manually selecting a subset of features from the original dataset to use in a machine learning model. This can be done through domain knowledge or by examining the correlations between the features and the target.\n",
    "\n",
    "- There are several approaches to manual feature selection:\n",
    "    - **Forward selection:** This involves starting with an empty set of features and adding one feature at a time, based on some criterion such as the improvement in model performance. The process is repeated until no further improvement is achieved.\n",
    "\n",
    "    - **Backward selection:** This involves starting with the full set of features and removing one feature at a time, based on some criterion such as the decrease in model performance. The process is repeated until no further improvement is achieved.\n",
    "\n",
    "    - **Recursive feature elimination:** This involves recursively removing features, building the model using the remaining features, and selecting the best performing features. The process is repeated until the desired number of features is reached.\n",
    "\n",
    "    - **Genetic algorithms:** These are optimization algorithms that mimic the process of natural selection and can be used to select a subset of features.\n",
    "\n",
    "**Interpretation:**\n",
    "- Manual feature selection can be time-consuming and may not always yield the best results. It is important to carefully evaluate the impact of the selected features on the performance of the model. In some cases, manually selecting a subset of features may lead to a decrease in performance, while in other cases it may improve the performance of the model.\n",
    "\n",
    "#### Model-based feature selection\n",
    "- Model-based feature selection is a method of selecting a subset of features for a machine learning model using the model itself as a criterion. This is different from manual feature selection, which involves manually selecting features based on domain knowledge or correlations with the target.\n",
    "- There are several approaches to model-based feature selection:\n",
    "\n",
    "    - **Regularization:** Some machine learning models, such as linear and logistic regression, include regularization terms that penalize complex models and promote the selection of a smaller number of features.\n",
    "\n",
    "    - **Decision trees:** Decision tree-based models, such as random forests and gradient boosting, can be used to select features by ranking them based on their importance.\n",
    "\n",
    "    - **Mutual information:** This method selects features based on their mutual information with the target. It is a nonparametric method that does not assume a particular distribution of the data.\n",
    "\n",
    "    - **Embedded methods:** These are methods that incorporate feature selection as part of the training process, such as lasso regression and the elastic net.\n",
    "    \n",
    "**Interpretation:**\n",
    "\n",
    "- Model-based feature selection can be an effective way to select a subset of features that are most relevant to the problem at hand. It is important to carefully evaluate the impact of the selected features on the performance of the model. In some cases, model-based feature selection may lead to a decrease in performance, while in other cases it may improve the performance of the model.\n",
    "\n",
    "<img src=\"images/p107.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa700cbd",
   "metadata": {},
   "source": [
    "<h2 align=\"center\">Polynomial Regression</h2>\n",
    "\n",
    "#### Overview\n",
    "- If the relationship between the inputs and output is not linear, we can use a polynomial to model the relationship.\n",
    "- We will formulate the polynomial regression model for single feature regression problem.\n",
    "- Polynomial Regression is often termed as Non-linear Regression or Linear in Parameter Regression.\n",
    "- We will also revisit the concept of `over-fitting`.\n",
    "\n",
    "<img src=\"images/p108.png\" align=\"center\" height=300px width=300px >\n",
    "\n",
    "**Single Feature Regression: Formulation**\n",
    "- `d=1`, input $x$ is a scalar.\n",
    "- Model is a polynomial function of the input, that is,\n",
    "$$\\hat f(x,\\mathbf{\\theta}) = \\theta_0 + \\theta_1x + \\theta_2x^2 + ....+ \\theta_ix^M = \\sum_{i=0}^M \\theta_ix^i$$\n",
    "$$ \n",
    "where \\; \\mathbf{\\theta} = \\begin{bmatrix} \\theta_0 \\\\ \\theta_1 \\\\ \\theta_2 \\\\ ..... \\\\ \\theta_M \\end{bmatrix} \n",
    "$$\n",
    "\n",
    "- $M$ is the degree of polynomial; characterized by $M+1$ coefficients $\\theta_0, \\theta_1, \\theta_2 ,....., \\theta_M$\n",
    "\n",
    "- $M$ is the Hyper-Parameter of the model and determines the complexity of the model. For $M=1$, we have a linear regression.\n",
    "- We can use linear regression to find these coefficients by formulating the input $x$ and its powers using a `vector-valued` function given by\n",
    "\n",
    "$$\\mathbf{g}(x) = [1,x,x^2,x^3,....,x^M]^T$$\n",
    "\n",
    "- With this notation , we can formulate model as $$\\hat f(x,\\mathbf{\\theta}) = \\mathbf{g}(x)^T \\mathbf{\\theta} $$\n",
    "\n",
    "- Note that the model is linear in terms of parameters due to which ploynomial regression is termed as `Linear in Parameters Regression`.\n",
    "\n",
    "- Note that $\\mathbf{g}(x)$ can be any function of $x$. For example, we can have \n",
    "$$\\mathbf{g}(x) = [\\frac{1}{x}, sin(2\\pi x), x^2, e^x,....]^T$$\n",
    "\n",
    "- For $n$ data points(input, output), we can define residual error in a similar way we computed for linear regression as follows:\n",
    "\n",
    "<img src=\"images/p110.png\" align=\"center\">\n",
    "\n",
    "$$where \\;\\;\\; e = y - X\\mathbf{\\theta}$$\n",
    "$$\\mathbf{\\theta} = (X^TX)^{-1}X^Ty$$\n",
    "\n",
    "> **Note**: We have seen this before. & We are capable to solve this!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "355f381c",
   "metadata": {},
   "source": [
    "<h3 align=\"center\">Polynomial Curve Fitting: A Simple Regression Problem</h3>\n",
    "    \n",
    "- We observe a real-valued input variable `x` and we wish to use this observation to predict the value of a real-valued target variable `t` or $f(x) = sin(2\\pi x)$.\n",
    "- We use synthetically generated data from the function sin(2πx) with random noise included in the target values.\n",
    "    - A small level of random noise having a Gaussian distribution\n",
    "\n",
    "- We have a training set comprising `N` observations of `x`, written $x ≡ (x_1, . . . , x_N)^T$, together with corresponding observations of the values of `f(x)`, denoted $f(x) ≡ (f(x_1), . . . , f(x_N)^T$.\n",
    "\n",
    "- Our goal is to predict the value of t for some new value of x\n",
    "\n",
    "\n",
    "<img src=\"images/p111.png\" align=\"right\" height=400px width=400px>\n",
    "<img src=\"images/p112.png\" align=\"right\" height=400px width=400px>\n",
    "\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "$$\\newline$$\n",
    "\n",
    "\n",
    "**Process:**\n",
    "$$f(x) = sin(2\\pi x)$$\n",
    "\n",
    "**Observations:**\n",
    "$$y = f(x) + n$$\n",
    "\n",
    "**Model:**\n",
    "$$\\hat f(x,\\mathbf{\\theta}) = \\theta_0 + \\theta_1x + \\theta_2x^2 + ....+ \\theta_ix^M $$\n",
    "\n",
    "\n",
    "- A training data set of N = 10 points, (blue circles),\n",
    "- The green curve shows the actual function sin(2πx) used to generate the data.\n",
    "- Our goal is to predict the value of t for some new value of x, without knowledge of the green curve."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b004b044",
   "metadata": {},
   "source": [
    "### Polynomial Curve Fitting\n",
    "\n",
    "- We try to fit the data using a polynomial function/model of the form\n",
    "\n",
    "$$\\hat f(x,\\mathbf{\\theta}) = \\theta_0 + \\theta_1x + \\theta_2x^2 + ....+ \\theta_Mx^M =\\sum_{i=0}^M \\theta_ix^i$$\n",
    "\n",
    "<h4 align=\"center\"> OR</h4>\n",
    "\n",
    "$$y(x,\\mathbf{w}) = w_0 + w_1x + w_2x^2 + ....+ w_Mx^M =\\sum_{i=0}^M w_ix^i$$\n",
    "\n",
    "\n",
    "- The values of the coefficients will be determined by fitting the polynomial to the training data.\n",
    "- This can be done by minimizing an error function that measures the misfit between the function $y(x,\\mathbf{w})$ or $\\hat f(x, \\mathbf{\\theta})$, for any given value of $w$, and the training set data points.\n",
    "\n",
    "$$L(\\mathbf{w}) = \\frac{1}{2} \\sum_{n=1}^N (y(x_n,\\mathbf{w}) - f(x_n))^2$$\n",
    "\n",
    "- Error Function: the sum of the squares of the errors between the predictions $y(x_n ,w)$ for each data point $x_n$ and the corresponding target values $t_n$ or $f(x_n)$.\n",
    "\n",
    "<img src=\"images/p113.png\">\n",
    "\n",
    "\n",
    "- We can solve the curve fitting problem by choosing the value of w for which $L(w)$ or $L(\\mathbf{\\theta})$is as small as possible.\n",
    "- Since the error function is a quadratic function of the coefficients w, its derivatives with respect to the coefficients will be linear in the elements of w, and so the minimization of the error function has a unique solution, denoted by $w^*$ or $\\theta^*$.\n",
    "- The resulting polynomial is given by the function $y(x,w^*)$ or $\\hat f(x,\\theta^*)$.\n",
    "- Choosing the order M of the polynomial is  done by **model selection.**\n",
    "\n",
    "<img src=\"images/p120.png\">\n",
    "\n",
    "\n",
    "- The $0^th$ order `(M=0)` and first order `(M=1)` polynomials give rather poor fits to the data and consequently rather poor representations of the function `sin(2πx)`.\n",
    "- The third order `(M=3)` polynomial seems to give the best fit to the function `sin(2πx)` of the examples.\n",
    "- When we go to a much higher order polynomial `(M=9)`, we obtain an excellent fit to the training data.\n",
    "    - In fact, the polynomial passes exactly through each data point means we have zero residual error , that is , $L(w^*) = 0$ or $L(\\theta^*)=0$.\n",
    "    \n",
    "    \n",
    "**Question: What's happening with the increase in `M`?**\n",
    "\n",
    "**Answer:** `Overfitting` will occur.\n",
    "\n",
    "**Question: Is this a good solution?**\n",
    "\n",
    "**Answer:** No! The model is oscillating wildly and is not close to the the `true function` or $sin(2\\pi x)$.\n",
    "\n",
    "<img src=\"images/p117.png\" height=300px width=300px>\n",
    "\n",
    "\n",
    "- In this toy example, we had informationabout the true function and therefore we can conclude that `M=9`,  is not a good model to fit the data. We can then evaluate the residual value of $L(w*)$ for the training data, and we can also evaluate $L(w*)$ for the test data set.\n",
    "\n",
    "<img src=\"images/p121.png\">\n",
    "\n",
    "\n",
    "**Questions: How to choose model Order $M$ or How do we tell if a model is overfitting  when we do not have knowledge about the true process/function.**\n",
    "\n",
    "**Solution 1:** Train-Validation Split. Overfitting causes poor generalization performance, that is, large error on the testing or validation data.\n",
    "\n",
    "**Solution 2:** Take `MORE` data points to avoid over-fitting.\n",
    "\n",
    "**Solution 3:** Regularization."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "932c4092",
   "metadata": {},
   "source": [
    "#### Polynomial Curve Fitting: Polynomial Coefficients\n",
    "- Magnitude of coefficients increases dramatically as order of polynomial increases.\n",
    "\n",
    "#### Example\n",
    "- Let's pose another question. $M = 3$ degree polynomial is a special case of $ M = 9$ degree polynomial. Why$ M = 9$ gives us poor performance?\n",
    "\n",
    "- $M = 3$ solution cannot be recovered from $ M = 9$ solution by setting the remaining weights equal to zero. 10 coefficients are tuned for 10 data-points when $ M = 9$.\n",
    "\n",
    "<img src=\"images/p122.png\" height=500px width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e6789f2",
   "metadata": {},
   "source": [
    "#### Polynomial Regression : How to Handle Overfitting?\n",
    "\n",
    "- The polynomial degree M is the hyper-parameter of our model, like we had k in kNN, Solution 2: Take more data points to avoid over-fittingand controls the complexity of the model.\n",
    "- If we stick with M=3 model, this is the restriction on the number of parameters.\n",
    "- We encounter overfitting for M=9 because we do not have sufficient data.\n",
    "\n",
    "**Solution 2:** Take more data points to avoid over-fitting.\n",
    "\n",
    "<img src=\"images/p123.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "febba3cf",
   "metadata": {},
   "source": [
    "**Example:** Polynomial regression models with two predictor variables and interaction terms are quadratic forms. Their surfaces can have many different shapes depending on the values of the model parameters with the contour lines being either parallel lines, parabolas or ellipses.\n",
    "    \n",
    "$$y = \\theta_0 + \\theta_1x_1 +\\theta_2x_2 + \\theta_{11}x_1^2 + \\theta_{22}x_2^2 + \\theta_{12}x_1x_2 + e$$\n",
    "\n",
    "<img src=\"images/p124.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "878538c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 9.06297259e-12  1.00000000e+00  7.46069873e-14 -4.44089210e-14]\n",
      "4.5929187419244395e-23\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "def poly_features(X, degree):\n",
    "    \"\"\"\n",
    "    Maps X to an m x d+1 matrix, where each row is a polynomial of X with degree d.\n",
    "    \"\"\"\n",
    "    X_poly = np.ones((X.shape[0], 1))\n",
    "    for i in range(1, degree+1):\n",
    "        X_poly = np.hstack((X_poly, np.power(X, i)))\n",
    "    return X_poly\n",
    "\n",
    "def normal_equation(X, y):\n",
    "    \"\"\"\n",
    "    Solves the linear regression problem using the normal equation.\n",
    "    \"\"\"\n",
    "    theta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
    "    return theta\n",
    "\n",
    "def predict(X, theta):\n",
    "    \"\"\"\n",
    "    Makes predictions using the learned model parameters.\n",
    "    \"\"\"\n",
    "    y_pred = X @ theta\n",
    "    return y_pred\n",
    "\n",
    "def mean_squared_error(y, y_pred):\n",
    "    \"\"\"\n",
    "    Calculates the mean squared error between the true values and the predictions.\n",
    "    \"\"\"\n",
    "    mse = np.mean((y - y_pred) ** 2)\n",
    "    return mse\n",
    "\n",
    "# Test the polynomial regression function\n",
    "X = np.array([[1], [2], [3], [4]])\n",
    "y = np.array([1, 2, 3, 4])\n",
    "\n",
    "# Map the features to a higher-dimensional space\n",
    "X_poly = poly_features(X, 3)\n",
    "\n",
    "# Fit the model using the normal equation\n",
    "theta = normal_equation(X_poly, y)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = predict(X_poly, theta)\n",
    "\n",
    "# Calculate the mean squared error\n",
    "mse = mean_squared_error(y, y_pred)\n",
    "\n",
    "print(theta)  \n",
    "print(mse) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dba41bc5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Position,Level,Salary\r\n",
      "Business Analyst,1,45000\r\n",
      "Junior Consultant,2,50000\r\n",
      "Senior Consultant,3,60000\r\n",
      "Manager,4,80000\r\n",
      "Country Manager,5,110000\r\n",
      "Region Manager,6,150000\r\n",
      "Partner,7,200000\r\n",
      "Senior Partner,8,300000\r\n",
      "C-level,9,500000\r\n",
      "CEO,10,1000000"
     ]
    }
   ],
   "source": [
    "!cat datasets/position_salaries.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c31503cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmdElEQVR4nO3deZwU9ZnH8c8X0CheRHE9OI1i4pFEI1Gj2ajxCBoV1xNF14MNUYFoEjUm7HquV7KaKIiKqCjiEY81aPDIesTEK4AYBUy8udQIHiiiUeTZP3410gwzzMH0VHfX9/16zWu6q6qrnq7p6afq9/vVU4oIzMysuDrkHYCZmeXLicDMrOCcCMzMCs6JwMys4JwIzMwKzonAzKzgnAhshSS9JmmPvOOoIykkbdbMZXeW9KKkhZIOkLSBpEclfSDp4kZe8z1JdzVj3QMlPdDC8K0B2d/nS22wnjsk7d0WMRWNE0GVy/6J6n6WSPqo5PnAFq5rrKT/Lleszdj+rtl7qIt/rqSzV2KV5wAjI2LNiLgLGAzMB9aOiJ828przgAtLYmow8UTE+IjYayViazPZ3+2TbJ+9I+kPkr6Sd1zNlf19XmmDVV0E5Pb5rWZOBFUu+ydaMyLWBGYB+5VMG1+3nKRO+UW5vBXE83rJ+/k2MEjSAa3cTC9ger3nM6KRqyglfRNYJyKebOX2ym4F++2X2T7rBswFrmnHbVeEiPgLsLakvnnHUm2cCGpUdnQ9R9LPJL0JXCfpGEl/rrdcSNpM0mBgIHBadmR5d8li20h6VtICSbdKWq2RbXaQ9J+SZkp6S9INktbJ5vXOtjVI0izgoabeQ0S8CjwObNnI9h6R9B8lzz9/f5JeBr4E3J29n5uBo0veX0PNXXsDf2wqrvrbyp6HpOOzpqj3JF0uSSXzj5P0vKR3Jd0vqVfJvEslzZb0vqQpkv61ZN5Zkm6XdKOk94FjVhRXRHwE/BbYpmQdG2fNJvMkvSrpRyXzVpd0fRbX85JOkzSnZP5r2WfoWeBDSZ0k7Sjp8ex9/lXSrvX2yytZ89urdWel2Wfsj9lnaL6kW+vtu82yx+tkn5t52efoPyV1KN3nkv4ni/dVLd8U9Ajw/RXtI1ueE0Ft2xBYl3QkPHhFC0bEaGA82ZFlROxXMvtQoB+wCfA1Gv8yOib72Y30JbwmMLLeMrsAWwDfayp4SX2AnYEWH6FHxKYse4Z0OMu+v/9r4GVfBf7e0m2V2Bf4JmkfHUr2HiX1B34BHAisD/wJuLnkdZNIX9zrAjcBt9VLtv2B24Eu2XtolKQ1gMOBl7LnHYC7gb+SzhZ2B06WVLf/zwR6k/5eewJHNrDaw0lfrl2ADYDfk5pg1gVOAe6QtH627cuAvSNiLWAn4JlsHecCDwBfBLoDIxp5CyOAdbJ4dgH+HTi2ZP4OpL9RV+CXwDWlCRd4Hvh6I+u2RlRlIpB0bXbEOa2Zyx8qaYak6ZJuKnd8FWQJcGZE/DM7UmytyyLi9Yh4h/Slsk0jyw0ELomIVyJiIfBzYEC9JoWzIuLDFcSzcXak+T7wAvAU8OdGlm1rXYAPVuL1F0bEexExC3iYpfvpeOCCiHg+IhYD55POsnoBRMSNEfF2RCyOiIuBLwBfLlnvExFxV0QsWcF+O0XSe1n83waOyqZ/E1g/Is6JiE+ytvirgQHZ/EOB8yPi3YiYQ/oir++yiJidbftIYGJETMzi+QMwGdgnW3YJsLWk1SPijYioa5r7lHRAsnFEfBwRy/1NJXXM4vp5RHwQEa8BF5e8F4CZEXF1RHwGXA9sREpOdT4g/R2tBaoyEQBjSUeoTcqOKn8O7BwRWwEnly+sijMvIj5ug/W8WfJ4EelIvyEbAzNLns8EOrHsP+rsJrb1ekR0iYi1Sf/QH5H+4dvDu8BaK/H6xvZTL+DSLMG9B7wDiHSEjqRTsmaZBdn8dUhHvHWa2mcA/xMRXUhH9x+xNJH0Ymlyrdv+L1j6N9m43vob2lbptF7AIfXW921go4j4EDiMlPjekPR7Le20Pi17z3/JDsiOa2A7XYFVWP4z1K3k+ef7OCIWZQ9LP49rAe81sG5bgapMBBHxKOmf6XOSNpV0X9bG+qeSD+APgMsj4t3stW+1c7h5qt8p+iHQue6JpA2bWL6lXid9UdTpCSwG/tGabUTEAlJTyX6NLLLM+yE1ha2MZ4HNV3IdDZkN/DBLcHU/q0fE41l/wGmkI/MvZl/mC0hfmnVass9mASeREs/q2bZfrbfttSKi7gj+DVJTTZ0eDa223nsZV299a0TEhdn274+IPUlH6n8jnX0QEW9GxA8iYmPgh8AoLT8aaz5Lzxzq9CR1fjfXFqRmMGuBqkwEjRgNDIuI7UjtlqOy6ZsDm0t6TNKTkpp1JlGj/gpsJWmbrA36rHrz/0Fqm22tm4EfS9pE0pqkJpBbs+aQFsvWMYBlR/6UegY4UFLn7EtlUGu2U2IiqV26vlUlrVby07GF670S+LmkreDzDtFDsnlrkZLlPKCTpDOAtVsZPwBZc83rpH6hvwAfZB2+q0vqKGlrpRFSkDqWfy7pi5K6AUObWP2NwH5K11t0zPbHrpK6K12n0T/rK/gnsJDUVISkQyTVJZx3ScllSb24P8viOU/SWlnT2U+ybTbXLsC9LVjeqJFEkH1h7ETqZHsGuIp0RAKpaaIPsCup0+tqSV3aP8r8RcQLpLH1/we8yPJt79cAW2an/He1YhPXAuOAR4FXgY+BYS1cx8bKriMgNQusS+p7aMivgU9ICex6muhIbUpEPA0skLRDvVnTSc0tdT/H1n9tE+v9X9IY91uyvo9ppBFKAPcD95H6Q2aS9llzmoKa8ivSmUYnUif2NqS/yXxgDKn5CdLnYU427/9IndL/XMF7mU3qvP4FKXnNBk4lfZd0IH1xv046Y98FOCF76TeBp7K/6wTgpEauHRhGOtN7hfT5vIn0uWpSltwWZsNIrQVUrTemkdQbuCcitpa0NvD3iNiogeWuBJ6KiOuy5w8Cp0fEpHYN2KqCpL2AEyPigLxjyYOkE4ABEdHQmVFFk3QHcE1ETMw7lmpTE2cEEfE+8Grd6baSuiFkd5HOBpDUldRU1BZXMVoNiogHipQEJG2kVIqjg6QvAz8F/jfvuFojIg5yEmidqkwEShcHPQF8WemiqUGk5oNBkv5KOpXvny1+P/C2pBmkIX2nRsTbecRtVoFWJTWlfkC6yO93LO1fs4Ko2qYhMzNrG1V5RmBmZm2nootINaRr167Ru3fvvMMwM6sqU6ZMmR8R6zc0r+oSQe/evZk8eXLeYZiZVRVJMxub56YhM7OCcyIwMys4JwIzs4JzIjAzKzgnAjOzgitbImjq5jFZGYjLJL2kdBvEb5QrFjOzqjZ+PPTuDR06pN/jV6q+4nLKeUYwlhXfPGZvUlXQPqRyuVeUMRYzs+o0fjwMHgwzZ0JE+j14cJsmg7IlgoZuHlNPf+CGSJ4EukharnqomVmhDR8OixYtO23RojS9jeTZR9CNZeuuz2HZW9J9TtJgSZMlTZ43b167BGdmVhFmzWrZ9Faois7iiBgdEX0jou/66zd4hbSZWW3q2bNl01shz0Qwl2Xvj9qdlt2b1Mys9p13HnTuvOy0zp3T9DaSZyKYAPx7NnpoR2BBRLyRYzxmZpVn4EAYPRp69QIp/R49Ok1vI2UrOpfdPGZXoKukOcCZwCoAEXEl6Ubh+wAvAYto4X1gzcwKY+DANv3ir69siSAiDm9ifgBDyrV9MzNrnqroLDYzs/JxIjAzKzgnAjOzgnMiMDMrOCcCM7OCcyIwMys4JwIzs4JzIjAzKzgnAjOzgnMiMDMrOCcCM7OCcyIwMys4JwIzs4JzIjAzKzgnAjOzgnMiMDMrOCcCM7OCcyIwMys4JwIzs4JzIjAzKzgnAjOzgnMiMDMrOCcCM7OCcyIwMys4JwIzs4JzIjAzKzgnAjOzgitrIpDUT9LfJb0k6fQG5veU9LCkqZKelbRPOeMxM7PllS0RSOoIXA7sDWwJHC5py3qL/Sfw24jYFhgAjCpXPGZm1rBynhFsD7wUEa9ExCfALUD/essEsHb2eB3g9TLGY2ZmDShnIugGzC55PiebVuos4EhJc4CJwLCGViRpsKTJkibPmzevHLGamRVW3p3FhwNjI6I7sA8wTtJyMUXE6IjoGxF9119//XYP0syslpUzEcwFepQ8755NKzUI+C1ARDwBrAZ0LWNMZmZWTzkTwSSgj6RNJK1K6gyeUG+ZWcDuAJK2ICUCt/2YmbWjsiWCiFgMDAXuB54njQ6aLukcSftni/0U+IGkvwI3A8dERJQrJjMzW16ncq48IiaSOoFLp51R8ngGsHM5YzAzsxXLu7PYzMxy5kRgZlZwTgRmZgXnRGBmVnBOBGZmBedEYGZWcE4EZmYF50RgZlZwTgRmZgXnRGBmVnBOBGZmBedEYGZWcE4EZmYF50RgZlZwTgRmZgXnRGBmVnBOBGZmBedEYGZWcE4EZmYF50RgZlZwTgRmZgXnRGBmVnBOBGZmBedEYGZWcE4EZmYF50RgZlYFPv4YFi8uz7rLmggk9ZP0d0kvSTq9kWUOlTRD0nRJN5UzHjOzajN7NgwfDj16wB13lGcbncqzWpDUEbgc2BOYA0ySNCEiZpQs0wf4ObBzRLwr6V/KFY+ZWbWIgEcfhREj4K670vP99oNNNinP9sqWCIDtgZci4hUASbcA/YEZJcv8ALg8It4FiIi3yhiPmVlF+/BDGD8eRo6E556DddeFn/4UTjgBevcu33bLmQi6AbNLns8Bdqi3zOYAkh4DOgJnRcR99VckaTAwGKBnz55lCdbMLC8vvwyjRsG118J778E228A118Dhh8Pqq5d/++VMBM3dfh9gV6A78Kikr0bEe6ULRcRoYDRA3759o51jNDNrc0uWwB/+kJp/Jk6Ejh3hoINg2DDYaSeQ2i+WciaCuUCPkufds2ml5gBPRcSnwKuSXiAlhklljMvMLDcLFsD118Pll8MLL8AGG8B//Rf88Iew8cb5xFTORDAJ6CNpE1ICGAAcUW+Zu4DDgeskdSU1Fb1SxpjMzHLx/POp7f+GG2DhQthxx9QfcPDBsOqq+cZWtkQQEYslDQXuJ7X/XxsR0yWdA0yOiAnZvL0kzQA+A06NiLfLFZOZWXv67DO4557U/PPgg/CFL8CAATB0KPTtm3d0Symiuprc+/btG5MnT847DDOzRr39dursHTUKZs6E7t3hxBPhP/4D1l8/n5gkTYmIBtNP3p3FZmY145ln0tH/TTelK4F33RUuvhj694dOFfxtW8GhmZlVvk8/hTvvTAngscegc2c4+mgYMgS++tW8o2seJwIzs1Z4800YPRquvBLeeAO+9KV09H/ssfDFL+YdXcs4EZiZNVMEPPVUOvq/7bZ0NtCvH1x9Ney9N3So0jKeTgRmZk34+GO49daUAKZMgbXWSmUfhgyBzTfPO7qV50RgZtaI2bPhiivSEf/8+bDFFulCsKOOSsmgVjgRmJmViIA//nFp5U9IlT+HDYPvfrd9Sz+0FycCMzNS5c8bb0xX/06blip/nnJK+St/VgInAjMrtJdfTs09116b6gC1d+XPStCsPu7sJjNmZjVhyRK47z7Yd1/o0yc1A/XrB3/+Mzz9NBx3XJYExo9PpwMdOqTf48fnHHl5NPeM4EVJdwDXld5hzMysmixYAGPHpjOAF19sovLn+PEweDAsWpSez5yZngMMHNieYZddc0e9fh14ARgj6UlJgyWtXca4zMzazIwZaahnt25w8smw3nrpe37WLDj77EbKPw8fvjQJ1Fm0KE2vMc1KBBHxQURcHRE7AT8DzgTekHS9pM3KGqGZWSt89lka9bPHHrDVVqnd/+CDYdIkeOIJOOKIJso/z5rVsulVrFlNQ1kfwfeBY4HewMXAeOBfgYlkt5w0M8tb/cqfPXrA+ee3ovJnz55pBQ1NrzHN7iMAHgZ+FRGPl0y/XdJ32j4sM7OWmTo1Df2sq/y5225wySWw//6trPx53nnL9hFAqih33nltFnOlaHL3ZGcDYyPinIbmR8SP2jwqM7NmaKzy59ChsPXWK7nyug7h4cNTc1DPnikJ1FhHMTTzxjSS/hIR27dDPE3yjWnMrH7lz003TZ3Bxx4LXbrkHV1laosb0zwmaSRwK/Bh3cSIeLoN4jMza1JjlT/HjEm/q7XyZyVobiLYJvtd2jwUwHfbNBozs3rqV/5ce+1028chQ9LFYLbympUIImK3cgdiZlaqfuXPLbdMI4GOOgrWXDPv6GpLs/vSJX0f2ApYrW5aYx3IZmat0VDlz/33T5U/d9utNit/VoLmXkdwJdAZ2A0YAxwM/KWMcZlZgTRU+fPUU1Plz1698o6u9jX3jGCniPiapGcj4mxJFwP3ljMwM6t99St/brttejxgQHEqf1aC5iaCj7LfiyRtDLwNbFSekMysli1ZAg88kJp/7r0XOnZMpR+GDYNvfcvNP3lobiK4R1IX4FfA06QRQ2PKFZSZ1Z76lT833BDOOCNV/tzIh5W5au6ooXOzh3dIugdYLSIWlC8sM6sVM2aktv8bbkh9Ad/6Vqr4edBBTRR9s3azwkQg6cAVzCMi7mz7kMys2n32Gdx9d2r+eegh+MIX0h2/hg6F7bbLOzqrr6kzgv1WMC+AFSYCSf2AS4GOwJiIuLCR5Q4Cbge+GRGuH2FWpd5+O13pO2pUKs/TowdccEGq/Nm1a97RWWNWmAgi4tjWrjgrVnc5sCcwB5gkaUL9O5xJWgs4CXiqtdsys3xNnZqO/m++eWnlz9/8Bvbbr5WVP61dlfOCsu2BlyLilez1twD9gfq3ujwXuAg4tbmxmFn+Pv0U7rgjJYDHH0+VP485JjX/bLVV3tFZSzT35vVXAocBwwABhwBNXebRDZhd8nxONq10vd8AekTE75vY/mBJkyVNnjdvXnNCNrMyefPN1Nnbq1dq93/rLfj1r2Hu3FQSwkmg+uR2QZmkDsAlwDFNLRsRo4HRkMpQr8x2zazlIuDJJ9PR/+23p7OBvfdOdwL73vdc+bPatfaCsndo+oKyuUCPkufds2l11gK2Bh5RuoJkQ2CCpP3dYWxWGT7+GG65JQ3/rKv8OWRIqv7pyp+1o6UXlP0SmJJNa+qCsklAH0mbkBLAAOCIupnZdQifjyOQ9AhwipOAWf5mzVpa+fPtt1PlzyuugCOPdOXPWtTUdQTfBGbXXVAmaU3gOeBvwK9X9NqIWCxpKHA/afjotRExXdI5wOSImNAWb8DM2kYEPPJIOvqvq/zZv38q/bDrri79UMtWeKtKSU8De0TEO9lN6m8hdRhvA2wREQe3S5QlfKtKs7a1cOHSyp/Tp8N666Vx/678WVtW5laVHSPinezxYcDoiLiDVGrimTaM0cza2Usvpbo/112X6gB94xvp8WGHufJn0TSZCCR1iojFwO7A4Ba81swqzJIlcP/96ei/rvLnIYeksf+u/FlcTX2Z3wz8UdJ80sihPwFI2gxw0TmzKrFgQTrav/zydCaw4YZw5pkweLArf1rTJSbOk/QgaajoA7G0Q6EDqa/AzCrY9Onp6H/cuFT5c6ed4JxzXPnTltVk805EPNnAtBfKE46ZrazFi1Plz5Ejl1b+POKI1PzzjW/kHZ1VIrfzm9WI+fNT5c8rrnDlT2sZJwKzKvf00+no/6ab4J//dOVPazl/TMyq0CefpMqfI0curfx57LGu/Gmt41JRZlXkjTfgrLPShV5HHFHDlT/Hj4fevVM1u96903MrG58RmFW4CHjiiXT0f9ttqTN4771T6YearPw5fnwa17poUXo+c2Z6DjBwYH5x1bAVlpioRC4xYUXx0UdLK38+/XSq/Hnccan652ab5R1dGfXunb786+vVC157rb2jqRkrU2LCzNrZzJmpmWfMmFT5c6utClb5c9aslk23leZEYFYBIuDhh9PR/+9+l6YVtvJnz54NnxH07Nn+sRRErbUumlWVhQvT0f7WW8Puu8Ojj8Jpp8Err8Cdd6ahoIVKAgDnnZeGQZXq3DlNt7LwGYFZDl58EUaNcuXPBtV1CA8fnpqDevZMScAdxWXjRGDWTpYsgfvuW1r5s1OnVPlz2DDYcccCHvmvyMCB/uJvR04EZmX23nswduyylT/POsuVP61yOBGYlUlDlT/PPRcOPNCVP62yOBGYtaG6yp8jRqRRQK78adXAicCsDdSv/NmzJ1x4IQwa5MqfVvmcCMxWwtNPp6P/m29OlT+/+11X/rTq44+qWQvVVf4cMSLVAFpjjaWlH2qm6JsVihOBWTO98QZcdVX6efPNVO/nN7+Bo4+GLl3yjs6s9ZwIzFagocqf++yTOn9rsvKnFZITgVkD6lf+XGeddOHXiSfWeOVPKyQnArMSs2alkT9XX7208ueVV6aLXAtR+dMKqawntpL6Sfq7pJcknd7A/J9ImiHpWUkPSupVznjMGlJX+fPAA2GTTeCXv4RddoGHHoLnnoMf/rBAScB3Biuksp0RSOoIXA7sCcwBJkmaEBEzShabCvSNiEWSTgB+CRxWrpjMSi1cCDfemJp/pk+H9daDn/0Mjj++oBWPfWewwirnGcH2wEsR8UpEfALcAvQvXSAiHo6I7FPHk0D3MsZjBqTKnz/+MXTvDieckK7+ve46mDMHzj+/oEkAUrXPuiRQZ9GiNN1qWjn7CLoBs0uezwF2WMHyg4B7G5ohaTAwGKBnYf9LbWXUr/y5yiqp8ufQoa78+TnfGaywKqKzWNKRQF9gl4bmR8RoYDSkexa3Y2hW5Vz5swV8Z7DCKmfT0FygR8nz7tm0ZUjaAxgO7B8R/yxjPFYg06alZp/u3VMz0AYbpDIQM2fCmWc6CTTIdwYrrHKeEUwC+kjahJQABgBHlC4gaVvgKqBfRLxVxlisABYvhgkTUvOPK3+2gu8MVlhlSwQRsVjSUOB+oCNwbURMl3QOMDkiJgC/AtYEblNqpJ0VEfuXKyarTa782YZ8Z7BCKmsfQURMBCbWm3ZGyeM9yrl9q21TpqSj/9LKn5demip/duyYd3Rm1cOVUqyqfPIJ3HRTuttX376p/s9xx6XrAB58EA44oEqTgC/kshxVxKghs6a8/jqMHr185c9jjkl1gKqaL+SynCmiukZj9u3bNyZPnpx3GNYOIuDxx1Pzz+23w2efwd57p+Jve+1VQ5U/e/dueNhmr17w2mvtHY3VKElTIqJvQ/N8RmAVp67y54gRMHVqASp/+kIuy5kTgVWMmTPTyJ8xYwpW+dMXclnOauXk2qpUROrk/bd/gy99CX71qwJW/vSFXJYznxFYLhYuhHHjUvv/jBkFr/zpC7ksZz4jsHb14otw8snQrVtq819ttQqo/FkJQzcHDkwdw0uWpN9OAtaOfEZgZVdX+XPEiPS7oip/euimmYePWvm891462r/8cnj55VTo7fjj4Qc/qKCibx66aQXh4aPWrqZNS23/48alA+2dd4b//u90K8hVV807uno8dNPMicDaRl3lzxEj4JFHUtv/EUfAkCEVXvnTQzfN3FlsK2f+fLjggjT086CD4NVX4aKLUufvNdc0kQQqoZPWQzfNfEZgrTNlSjr6v+WWVPlz993hsstaUPmzUjppPXTTzJ3F1nyffJJq/owcCU88AWusAUcfnZp/ttyyhStzJ61Zu3Jnsa2U119PVT+vugr+8Q/o06cNKn+6k9asYriPoIia0TYfAY89Bocfng7Szz031f+/917429/gpJNWsvxzY52x7qQ1a3dOBEVT1zY/c2b6tq9rm8+SwUcfpbH/220H3/52+uIfNgxeeAHuuQf69Wuj8s/upDWrGE4E7akSRskMH760g7bOokXM/NkoTj8devRId/z69NNU+XPuXLjkkjKUfx44MN1pplevdGlxr17puTtpzdpfRFTVz3bbbRctduONEb16RUjp9403tnwdK+vGGyM6d45Ix+Hpp3Pn9o9F+nz7SyAeZLc4gDujA4ujQ4eIAw+MePjhiCVL2jcsMysvYHI08r1a+6OG6g9ThNQE0d5Hn5UySqZ3bxbOnM84jmIkQ5nBVnRlHj9Y+7cc/9wQN9Gb1agVjRqq/aahRppCGD68feOogFEyL74IJ3/lProxlxO5gtX5iLEczezVv8z5o7o4CZgVVO0nggr4AgZyGyWzZAlMnJju9bv55jDqoa+w707v8MSG/8YktufoXn9ktatHuG3erMBq/zqCSqklc955DTdRlWmUTEOVP88+O4Ww4YabAP9blu2aWfWp/TOCShmm2E6jZKZNS6Weu3WDn/wENtwwlYF47TU444z03MysVO2fEVRSLZmBA8uy3cYqfw4dCttu2+abM7MaU9ZEIKkfcCnQERgTERfWm/8F4AZgO+Bt4LCIeK3NAynTF3De5s+Hq6+GK66A2bPTScZFF8GgQekewGZmzVG2RCCpI3A5sCcwB5gkaUJEzChZbBDwbkRsJmkAcBFwWLliqhUNVf4cMQL23beZlT/NzEqU84xge+CliHgFQNItQH+gNBH0B87KHt8OjJSkqLaLG9pBXeXPESPgySdT5c9Bg1pZ+dPMrEQ5E0E3YHbJ8znADo0tExGLJS0A1gPmlzGuqtJQ5c9LL03ln1eq6JuZWaYqOoslDQYGA/QswFVPEfD44+no/4474LPPYJ99UufvXnu1UdE3M7NMORPBXKBHyfPu2bSGlpkjqROwDqnTeBkRMRoYDanERFmirQAffQQ335xu/DJ1ajri/9GP4IQTylD0zcwsU85EMAnoI2kT0hf+AOCIestMAI4GngAOBh4qYv/AzJkwahSMGQPvvANbb50qfx55ZOoLMDMrp7IlgqzNfyhwP2n46LURMV3SOaQqeBOAa4Bxkl4C3iEli0KIgIceSs0/d9+drjE74IDU/LPLLum5mVl7KGsfQURMBCbWm3ZGyeOPgUPKGUOlWbgQbrghNf88/zx07Qo/+1lq/unRo+nXm5m1taroLK4FL7yQ6v6MHQvvv5/uADZ2LBx2WLoS2MwsL04EZbRkSbrV48iRcN99sMoqcMgh6daPO+zg5h8zqwxOBGWw4sqfeUdnZrYsJ4I2NG1aOvofNy5Vm95551Tf7sAD09mAmVklciJYSYsXw+9+lxKAK3+aWTVyImilefPSuH9X/jSzaudE0EKTJ6ejf1f+NLNa4UTQDI1V/hw6FLbYIu/ozMxWjhPBCrjyp5kVgRNBPY1V/hw2DPbc05U/zaz2OBFk6ip/jhgBzzwDXbqkyp8nngibbpp3dGZm5VP4RNBQ5c+rrkq3OHblTzMrgkImgsYqfw4bBt/5jks/mFmxFCoRNFT58/TT4fjjXfnTzIqrMIngmmvgJz9JlT/79oXrr4dDD3XlTzOzwiSCXr1gv/1S88/227v5x8ysTmESwR57pB8zM1uWR8WbmRWcE4GZWcE5EZiZFZwTgZlZwTkRmJkVnBOBmVnBORGYmRWcE4GZWcEpIvKOoUUkzQNm5h3HSuoKzM87iAri/bGU98WyvD+WtTL7o1dErN/QjKpLBLVA0uSI6Jt3HJXC+2Mp74tleX8sq1z7w01DZmYF50RgZlZwTgT5GJ13ABXG+2Mp74tleX8sqyz7w30EZmYF5zMCM7OCcyIwMys4J4J2JKmHpIclzZA0XdJJeceUN0kdJU2VdE/eseRNUhdJt0v6m6TnJX0r75jyJOnH2f/JNEk3SyrMjWUlXSvpLUnTSqatK+kPkl7Mfn+xrbbnRNC+FgM/jYgtgR2BIZK2zDmmvJ0EPJ93EBXiUuC+iPgK8HUKvF8kdQN+BPSNiK2BjsCAfKNqV2OBfvWmnQ48GBF9gAez523CiaAdRcQbEfF09vgD0j96t3yjyo+k7sD3gTF5x5I3SesA3wGuAYiITyLivVyDyl8nYHVJnYDOwOs5x9NuIuJR4J16k/sD12ePrwcOaKvtORHkRFJvYFvgqZxDydNvgNOAJTnHUQk2AeYB12VNZWMkrZF3UHmJiLnA/wCzgDeABRHxQL5R5W6DiHgje/wmsEFbrdiJIAeS1gTuAE6OiPfzjicPkvYF3oqIKXnHUiE6Ad8AroiIbYEPacNT/2qTtX/3JyXIjYE1JB2Zb1SVI9K4/zYb++9E0M4krUJKAuMj4s6848nRzsD+kl4DbgG+K+nGfEPK1RxgTkTUnSHeTkoMRbUH8GpEzIuIT4E7gZ1yjilv/5C0EUD2+622WrETQTuSJFIb8PMRcUne8eQpIn4eEd0jojepE/ChiCjsEV9EvAnMlvTlbNLuwIwcQ8rbLGBHSZ2z/5vdKXDneWYCcHT2+Gjgd221YieC9rUzcBTp6PeZ7GefvIOyijEMGC/pWWAb4Px8w8lPdmZ0O/A08Bzpu6ow5SYk3Qw8AXxZ0hxJg4ALgT0lvUg6Y7qwzbbnEhNmZsXmMwIzs4JzIjAzKzgnAjOzgnMiMDMrOCcCM7OCcyKwmiLps2xY7jRJt0nq3MLXbyzp9uzxNqXDeyXtL6lNrvaVtLAt1lPudVoxePio1RRJCyNizezxeGBKay/ek3QMqfrl0DYMsW7dn8dZyeu0YvAZgdWyPwGbZXXc75L0rKQnJX0NQNIuJRf2TZW0lqTe2dnEqsA5wGHZ/MMkHSNpZPba3pIeytb5oKSe2fSxki6T9LikVyQd3FSQkk6VNClb19nZtAslDSlZ5ixJpzS2vNnKcCKwmpSVLt6bdFXq2cDUiPga8AvghmyxU4AhEbEN8K/AR3Wvj4hPgDOAWyNim4i4td4mRgDXZ+scD1xWMm8j4NvAvjRx9aekvYA+wPakq4m3k/Qd4Fbg0JJFDwVuXcHyZq3mRGC1ZnVJzwCTSfVqriF9KY8DiIiHgPUkrQ08Blwi6UdAl4hY3ILtfAu4KXs8LttGnbsiYklEzKDpUsF7ZT9TSeUUvgL0iYipwL9kfRZfB96NiNmNLd+CuM2W0ynvAMza2EfZEf7nUs2y5UXEhZJ+D+wDPCbpe8DHbRDDP0s338SyAi6IiKsamHcbcDCwIekMoanlzVrFZwRWBH8CBgJI2hWYHxHvS9o0Ip6LiIuASaSj61IfAGs1ss7HWXrrxIHZNlrjfuC47B4VSOom6V+yebdm2ziYlBSaWt6sVXxGYEVwFnBtVtVzEUtL+Z4saTfSHdKmA/eS2vfrPAycnjU1XVBvncNIdxM7lXRnsWNbE1hEPCBpC+CJ7MxlIXAk6aY90yWtBcytuzPVipZvzfbNwMNHzcwKz01DZmYF50RgZlZwTgRmZgXnRGBmVnBOBGZmBedEYGZWcE4EZmYF9//5dwkcpdO7NgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Implementation of polynomial regression\n",
    "## importing libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "### read data\n",
    "dataset = pd.read_csv(\"datasets/position_salaries.csv\")\n",
    "X = dataset.iloc[:, 1:2].values\n",
    "y = dataset.iloc[:, 2].values\n",
    "\n",
    "# Splitting the dataset into the Training set and Test set\n",
    "from sklearn.model_selection import train_test_split \n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
    "\n",
    "# Scaling\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "sc_X = StandardScaler()\n",
    "X_train = sc_X.fit_transform(X_train)\n",
    "X_test = sc_X.transform(X_test)\n",
    "\n",
    "# Fitting Linear Regression to the dataset\n",
    "from sklearn.linear_model import LinearRegression\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X, y)\n",
    "\n",
    "# Visualizing the Linear Regression results\n",
    "def viz_linear():\n",
    "    plt.scatter(X, y, color='red')\n",
    "    plt.plot(X, lin_reg.predict(X), color='blue')\n",
    "    plt.title('Truth or Bluff (Linear Regression)')\n",
    "    plt.xlabel('Position level')\n",
    "    plt.ylabel('Salary')\n",
    "    plt.show()\n",
    "    return\n",
    "viz_linear()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09db15cc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "e8b456bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAoNElEQVR4nO3debxVdb3/8dcbcEYlE01RDpo4z4F6rdSyTI3EuoIDZQ5F+dPKrkOWZWoXs8m6ld2kNL15UjkOpKaSOaA5oDhkOabIJKgggwMiIJ/fH9+1ZXM4E7DXWXt4Px+P/Th7r7X2Wp+9zznrs9Znfdf3q4jAzMwaV4+iAzAzs2I5EZiZNTgnAjOzBudEYGbW4JwIzMwanBOBmVmDcyKwDkmaLOkTRcdRIikkbdPFZT8s6d+S3pR0uKRNJd0j6Q1JP2vnPZ+SNLYL6x4h6a8rGb61Ifv9bF2B9Vwn6ZBKxNRonAhqXPZPVHoslfR22esRK7muyyX9d16xdmH7B2SfoRT/S5LOW41Vng/8OiJ6R8RYYCQwG9ggIk5r5z2jgAvLYmoz8UREc0QctBqxVUz2e1uUfWdzJN0uafui4+qq7PczqQKr+hFQ2N9vLXMiqHHZP1HviOgNTAU+UzatubScpF7FRbmiDuKZUfZ5PgKcKOnwVdxME/Bkq9dPRTt3UUoaDGwYEQ+u4vZy18H39uPsO+sHvARc2o3brgoR8RCwgaRBRcdSa5wI6lR2dD1d0rckvQz8QdJxkv7earmQtI2kkcAI4MzsyPKmssV2l/SEpPmSrpG0djvb7CHpu5KmSHpV0v9J2jCbNyDb1omSpgJ3dvYZIuJF4H5gx3a2d7ekL5W9fu/zSXoB2Bq4Kfs8VwFfLPt8bZW7DgHGdxZX621lr0PSV7NS1DxJF0tS2fwTJD0taa6kcZKayub9j6Rpkl6X9Iikj5bNO1fStZKulPQ6cFxHcUXE28AYYPeydWyelU1mSXpR0tfL5q0j6YosrqclnSlpetn8ydnf0BPAW5J6SdpH0v3Z5/yHpANafS+TsvLbi6Wz0uxvbHz2NzRb0jWtvrttsucbZn83s7K/o+9K6lH+nUv6aRbvi1qxFHQ38OmOviNbkRNBffsAsBHpSHhkRwtGxGigmezIMiI+UzZ7OHAwsBWwK+3vjI7LHh8j7YR7A79utcz+wA7ApzoLXtJA4MPASh+hR8QHWf4M6WiW/3x/a+NtuwDPruy2ygwBBpO+o+Fkn1HSUOA7wOeAvsC9wFVl73uYtOPeCPgT0NIq2Q4FrgX6ZJ+hXZLWA44Gns9e9wBuAv5BOls4EDhVUun7/z4wgPT7+iTw+TZWezRp59oH2BT4C6kEsxFwOnCdpL7Ztn8JHBIR6wP7Ao9n6/gB8FfgfcAWwK/a+Qi/AjbM4tkfOBY4vmz+3qTf0cbAj4FLyxMu8DSwWzvrtnbUZCKQdFl2xPmvLi4/XNJTkp6U9Ke846siS4HvR8Q72ZHiqvplRMyIiDmkncru7Sw3ArgoIiZFxJvAt4GjWpUUzo2ItzqIZ/PsSPN14DlgAvD3dpattD7AG6vx/gsjYl5ETAXuYtn39FXghxHxdEQsAS4gnWU1AUTElRHxWkQsiYifAWsB25Wt94GIGBsRSzv43k6XNC+L/yPAF7Lpg4G+EXF+RCzKavG/A47K5g8HLoiIuRExnbQjb+2XETEt2/bngVsi4pYsntuBicCh2bJLgZ0lrRMRMyOiVJpbTDog2TwiFkbECr9TST2zuL4dEW9ExGTgZ2WfBWBKRPwuIt4FrgA2IyWnkjdIv0dbCTWZCIDLSUeoncqOKr8NfDgidgJOzS+sqjMrIhZWYD0vlz1fQDrSb8vmwJSy11OAXiz/jzqtk23NiIg+EbEB6R/6bdI/fHeYC6y/Gu9v73tqAv4nS3DzgDmASEfoSDo9K8vMz+ZvSDriLensOwP4aUT0IR3dv82yRNLEsuRa2v53WPY72bzV+tvaVvm0JmBYq/V9BNgsIt4CjiQlvpmS/qJlF63PzD7zQ9kB2QltbGdjYA1W/BvqV/b6ve84IhZkT8v/HtcH5rWxbutATSaCiLiH9M/0HkkflHRbVmO9t+wP8MvAxRExN3vvq90cbpFaXxR9C1i39ELSBzpZfmXNIO0oSvoDS4BXVmUbETGfVCr5TDuLLPd5SKWw1fEEsO1qrqMt04CvZAmu9FgnIu7PrgecSToyf1+2M59P2mmWrMx3NhX4BinxrJNt+8VW214/IkpH8DNJpZqSLdtabavP8sdW61svIi7Mtj8uIj5JOlJ/hnT2QUS8HBFfjojNga8Av9GKrbFms+zMoaQ/6eJ3V+1AKoPZSqjJRNCO0cDXIuJDpLrlb7Lp2wLbSrpP0oOSunQmUaf+AewkafesBn1uq/mvkGqzq+oq4JuStpLUm1QCuSYrh6y0bB1HsXzLn3KPA5+TtG62UzlxVbZT5hZSXbq1NSWtXfbouZLr/S3wbUk7wXsXRIdl89YnJctZQC9J5wAbrGL8AGTlmhmk60IPAW9kF3zXkdRT0s5KLaQgXVj+tqT3SeoHnNLJ6q8EPqN0v0XP7Ps4QNIWSvdpDM2uFbwDvEkqFSFpmKRSwplLSi5LW8X9bhbPKEnrZ6Wz/8q22VX7A7euxPJGnSSCbIexL+ki2+PAJaQjEkiliYHAAaSLXr+T1Kf7oyxeRDxHalv/N+DfrFh7vxTYMTvlH7sKm7gM+CNwD/AisBD42kquY3Nl9xGQygIbka49tOXnwCJSAruCTi6kdiYiHgXmS9q71awnSeWW0uP41u/tZL03kNq4X51d+/gXqYUSwDjgNtL1kCmk76wrpaDO/IR0ptGLdBF7d9LvZDbwe1L5CdLfw/Rs3t9IF6Xf6eCzTCNdvP4OKXlNA84g7Ut6kHbcM0hn7PsDJ2VvHQxMyH6vNwLfaOfega+RzvQmkf4+/0T6u+pUltzezJqR2kpQrQ5MI2kAcHNE7CxpA+DZiNisjeV+C0yIiD9kr+8AzoqIh7s1YKsJkg4C/l9EHF50LEWQdBJwVES0dWZU1SRdB1waEbcUHUutqYszgoh4HXixdLqtpNSEbCzpbABJG5NKRZW4i9HqUET8tZGSgKTNlLri6CFpO+A04Iai41oVEfGfTgKrpiYTgdLNQQ8A2yndNHUiqXxwoqR/kE7lh2aLjwNek/QUqUnfGRHxWhFxm1WhNUml1DdIN/n9mWXX16xB1GxpyMzMKqMmzwjMzKxyqroTqbZsvPHGMWDAgKLDMDOrKY888sjsiOjb1ryaSwQDBgxg4sSJRYdhZlZTJE1pb55LQ2ZmDc6JwMyswTkRmJk1OCcCM7MG50RgZtbgcksEnQ0ek3UD8UtJzysNg7hnXrGYmdW05mYYMAB69Eg/m1erf8UV5HlGcDkdDx5zCKlX0IGk7nL/N8dYzMxqU3MzjBwJU6ZARPo5cmRFk0FuiaCtwWNaGQr8XyQPAn0krdB7qJlZQzv7bFiwgAnsxRKyoTAWLEjTK6TIawT9WL7f9eksPyTdeySNlDRR0sRZs2Z1S3BmZlVh6lSmsQX7MIGf883lpldKTVwsjojRETEoIgb17dvmHdJmZvWpf3+u5QgAPlveQ3j//hXbRJGJ4CWWHx91C1ZubFIzs/o3ahQtPY5kdx5jG15I09ZdF0aNqtgmikwENwLHZq2H9gHmR8TMAuMxM6s60/YbwQNL92FYn9tBgqYmGD0aRrQ3guvKy63TuWzwmAOAjSVNB74PrAEQEb8lDRR+KPA8sICVHAfWzKwRXHtt+jnsoTNh4Jm5bCO3RBARR3cyP4CT89q+mVk9aGmB3XaDgQPz20ZNXCw2M2tE06bBAw/A8OH5bseJwMysSl13Xfo5bFi+23EiMDOrUt1RFgInAjOzqjR9Otx/f/5nA+BEYGZWld5rLeREYGbWmFpaYNddYdtt89+WE4GZWZUplYXybi1U4kRgZlZluqu1UIkTgZlZlenOshA4EZiZVZWXXoL77uu+swFwIjAzqyrd2VqoxInAzKyKtLTALrvAdtt13zadCMzMqkQRZSFwIjAzqxrd3VqoxInAzKxKlMpC22/fvdt1IjAzqwIzZhRTFgInAjOzqnDddRDhRGBm1rDGjIGdd+7+shA4EZiZFa7IshA4EZiZFa7IshA4EZiZFa6lJZWFdtihmO07EZiZFWjmTPj734s7GwAnAjOzQhVdFgInAjOzQrW0wE47FVcWAicCM7PCzJwJ995b7NkAOBGYmRWmGspC4ERgZlaYUlloxx2LjcOJwMysANVSFgInAjOzQlx/fXWUhcCJwMysEC0tqSRUdFkInAjMzLrdzJlwzz3VcTYATgRmZt2umspCkHMikHSwpGclPS/prDbm95d0l6THJD0h6dA84zEzqwalstBOOxUdSZJbIpDUE7gYOATYEThaUutq2HeBMRGxB3AU8Ju84jEzqwYvv1xdZSHI94xgL+D5iJgUEYuAq4GhrZYJYIPs+YbAjBzjMTMrXLWVhQB65bjufsC0stfTgb1bLXMu8FdJXwPWAz6RYzxmZoVraUn9ClVLWQiKv1h8NHB5RGwBHAr8UdIKMUkaKWmipImzZs3q9iDNzCqhGstCkG8ieAnYsuz1Ftm0cicCYwAi4gFgbWDj1iuKiNERMSgiBvXt2zencM3M8nX99bB0aWMlgoeBgZK2krQm6WLwja2WmQocCCBpB1Ii8CG/mdWllpY0OH01lYUgx0QQEUuAU4BxwNOk1kFPSjpf0mHZYqcBX5b0D+Aq4LiIiLxiMjMryiuvpLLQ8OEgFR3N8vK8WExE3ALc0mraOWXPnwI+nGcMZmbVoFrLQlD8xWIzs4ZQrWUhcCIwM8vdq6/C+PHpbKDaykLgRGBmlrtqLguBE4GZWe7GjIHttoOddy46krY5EZiZ5ahUFqrG1kIlTgRmZjmq9rIQOBGYmeWqpaW6y0LgRGBmlptXX4W7767e1kIlTgRmZjm54YbqLwuBE4GZWW7GjIFtt4Vddik6ko45EZiZ5aBWykLgRGBmlotSWWj48KIj6ZwTgZlZDlpaaqMsBE4EZmYVN2sW3HVXbZSFwInAzKziaqW1UIkTgZlZhbW0wMCBsOuuRUfSNU4EZmYVNGsW3Hln7ZSFwInAzKyiaqm1UIkTgZlZBdVaWQicCMzMKmb27NpqLVTiRGBmViE33ADvvls7rYVKnAjMzCqkpQW22QZ2263oSFaOE4GZWQXMnl17rYVKnAjMzCqgVstC4ERgZlYRpbLQ7rsXHcnKcyIwM1tNtVwWAicCM7PVNnZs7ZaFwInAzGy1tbTABz9Ym2UhcCIwM1str70Gd9xRu2UhcCIwM1sttdxaqMSJwMxsNZTKQnvsUXQkq86JwMxsFdVDWQicCMzMVlmttxYqyTURSDpY0rOSnpd0VjvLDJf0lKQnJf0pz3jMzCqppQW23rq2y0IAvfJasaSewMXAJ4HpwMOSboyIp8qWGQh8G/hwRMyVtEle8ZiZVdKcOaksdNpptV0WgnzPCPYCno+ISRGxCLgaGNpqmS8DF0fEXICIeDXHeMzMKmbsWFiypPbLQpBvIugHTCt7PT2bVm5bYFtJ90l6UNLBba1I0khJEyVNnDVrVk7hmpl13ZgxsNVWsOeeRUey+oq+WNwLGAgcABwN/E5Sn9YLRcToiBgUEYP69u3bvRGambVSKgsNH177ZSHINxG8BGxZ9nqLbFq56cCNEbE4Il4EniMlBjOzqlVPZSHoYiLILvyurIeBgZK2krQmcBRwY6tlxpLOBpC0MalUNGkVtmVm1m1aWuqnLARdPyP4t6SfSNqxqyuOiCXAKcA44GlgTEQ8Kel8SYdli40DXpP0FHAXcEZEvLYS8ZuZdas5c+Bvf6v9m8jKdbX56G6kI/rfS+oBXAZcHRGvd/SmiLgFuKXVtHPKngfwX9nDzKzq/fnP9VUWgi6eEUTEGxHxu4jYF/gW8H1gpqQrJG2Ta4RmZlVkzBgYMAA+9KGiI6mcLl8jkHSYpBuAXwA/A7YGbqLVEb+ZWb0qlYXqpbVQSVdLQ/8m1fB/EhH3l02/VtJ+lQ/LzKz61GNZCLqQCLIWQ5dHxPltzY+Ir1c8KjOzKtTSUn9lIehCaSgi3gWGdEMsZmZVa+7c+mstVNLV0tB9kn4NXAO8VZoYEY/mEpWZWZX5859h8eL6KwtB1xPB7tnP8vJQAB+vaDRmZlWqVBYaNKjoSCqvS4kgIj6WdyBmZtVq7ly4/Xb4xjfqrywEKzEegaRPAzsBa5emtXcB2cysnpTKQsOHFx1JPrp6H8FvgSOBrwEChgFNOcZlZlY16rksBF3va2jfiDgWmBsR5wH/Qeogzsysrs2bl8pCRxxRn2Uh6HoieDv7uUDS5sBiYLN8QjIzqx713FqopKuJ4OZswJifAI8Ck4GrcorJzKw6NDcz5qQ7aWIyg4cNgObmoiPKRVdbDf0ge3qdpJuBtSNifn5hmZkVrLmZG08Yyy2LWvguP0BTp8DIkWneiBHFxlZhSj1BtzNT+lxHb46I6yseUScGDRoUEydO7O7NmlmDebHfR9hzxk1szSTu48OszTtpRlMTTJ5caGyrQtIjEdHm5e7Ozgg+08G8ALo9EZiZ5e2dd2D4jJ8TiBaGLUsCAFOnFhdYTjpMBBFxfHcFYmZWLU47DSYymBs4nK15cfmZ/fsXE1SOfEOZmVmZa66Biy+G0w59isPvvh0WlM1cd10YNaqw2PLiG8rMzDLPPgtf+hLsuy/8cOyOMHp0uiYgpZ+jR9fdhWLo5GLxewtJT0TErmU/ewO3RsRH8w9xeb5YbGZ5WLAA9t4bXn4ZHnsMttii6Igqa3UuFpe0vqFsDr6hzMzqyMknw5NPwm231V8S6ExXE0HphrIfA49k036fS0RmZt3sssvg8svhe9+Dgw4qOpru12EikDQYmFa6oSwrCf0TeAb4ef7hmZnl64kn0tnAgQfC979fdDTF6Oxi8SXAIoBskPoLs2nzgdH5hmZmlq/XX0+dyb3vfan3iJ49i46oGJ2VhnpGxJzs+ZHA6Ii4jtTVxOO5RmZmlqOI1EJo0iS4807YdNOiIypOZ2cEPSWVksWBwJ1l87p8D4KZWbX59a/TOAOjRsF++xUdTbE625lfBYyXNJvUcuheAEnbkMpDZmY156GH0t3DQ4bAGWcUHU3xOutiYpSkO0hNRf8ay2466EG6uczMrKbMmZOGnNx8c7jiCujR1c7461in5Z2IeLCNac/lE46ZWX6WLoVjj4UZM+C++2CjjYqOqDq4zm9mDePHP4a//AV+9SsYPLjoaKqHT4rMrCGMHw9nn53KQiefXHQ01cWJwMzq3iuvwNFHwzbbwO9/X7+D0K+qXBOBpIMlPSvpeUlndbDcf0oKSW12iGRmtqrefReOOQbmzoVrr4X11y86ouqTWyKQ1BO4GDgE2BE4WtKObSy3PvANYEJesZhZ4zrvvHTD2G9+A7vsUnQ01SnPM4K9gOcjYlJELAKuBoa2sdwPgB8BC3OMxcwa0Lhx8N//Dccfnx7WtjwTQT9gWtnr6dm090jaE9gyIv7S0YokjZQ0UdLEWbNmVT5SM6s706alMWR23jndRWztK+xisaQewEXAaZ0tGxGjI2JQRAzq27dv/sGZWU1bvBiOPDINQt/SkkaYtPbleR/BS8CWZa+3yKaVrA/sDNytdAn/A8CNkg6LCA9BZmar7Kyz4IEH0vjD221XdDTVL88zgoeBgZK2krQmcBRwY2lmRMyPiI0jYkBEDAAeBJwEzGy13HADXHQRnHJKumfAOpdbIoiIJcApwDjgaWBMRDwp6XxJh+W1XTNrXC+8AMcdl+4a/ulPi46mduTaxURE3ALc0mraOe0se0CesZhZfVu4EIYNS4PLjBkDa61VdES1w30NmVldOPVUeOwxuOkmGDCg6Ghqi7uYMLOa19wMl1wC3/pWGmPAVo4TgZnVtKeegpEj4aMfTTeP2cpzIjCzmvXmm2nw+d694eqroZeL3avEX5uZ1aQI+OpX4Zln4Pbb04hjtmqcCMysJv3ud+nawPnnw4EHFh1NbXNpyMxqzqOPwte/DgcdlAabsdXjRGBmNWXevHS/wMYbw5VXevD5SnBpyMxqRgSccAJMnZqGnnQflJXhRGBmNeMXv0h9Cf3sZ7DvvkVHUz98UmVmNeH+++HMM+Hww+Gb3yw6mvriRGBmVW/27DS+QP/+8Ic/ePD5SnMiMLPq09ycOgzq0YOlTVvx+Y/PYNasNMhMnz5FB1d/fI3AzKpLc3PqM2LBAgAumDqCcWzOb4+fwJ577l1wcPXJZwRmVl3OPvu9JHAnH+P7nMcxNDPyjiMLDqx++YzAzKrL1KkAzOQDHM1VbMezXMJX0LQFBQdWv3xGYGZVJbbsz1iGsh/38Ca9uZYj6M1b6Uqx5cKJwMyqxoQJsN/aD/FZxtKLJdzMEHbkaVh3XRg1qujw6pYTgZkVbtKk1Dx0n33guXmb8L/HP8Q/+w/hYxoPTU0wejSMGFF0mHXL1wjMrDBz5qTBZH796zSWwPe+B2ecAeuvvxfwQtHhNQwnAjPrdgsXpp3/qFHw+utw/PGpO2mPKVAMJwIz6zZLl8I118B3vgOTJ8PBB8OPfwy77FJ0ZI3N1wjMrFuMHw977w3HHJPuDr79drj1VieBauBEYGa5euYZGDoUDjgAXn4ZrrgCHnkEPvGJoiOzEicCM8vFK6/ASSfBzjvDXXfBBRfAc8/Bscd6MJlq42sEZlZRb70FF12Uav8LF6ZkcM45HkSmmjkRmFlFvPsuXH552unPmAGf/SxceCFsu23RkVlnfIJmZqslAm67DfbYA770JdhyS7j3Xrj+eieBWuFEYGar7PHH4aCD4JBDUklozBh44AH4yEeKjsxWhhOBma206dPhuONgzz3h0UfTWMJPPw3Dhnn0sFrkawRm1mWvv57q/j//eSoJnX56ujnMo4bVNicCM+vU4sWp37dzz03jB48YkbqHaGoqOjKrhFxLQ5IOlvSspOclndXG/P+S9JSkJyTdIcl/VmZVJAJuuCHdC3DKKennww/DlVc6CdST3BKBpJ7AxcAhwI7A0ZJ2bLXYY8CgiNgVuBb4cV7xmFkXlA0aP2Gzw9lv+1f53OegZ0+46Sa4804YNKjoIK3S8jwj2At4PiImRcQi4GpgaPkCEXFXRJTGn3sQ2CLHeMysI83NLP3yV5gwZVOOjKvY5+Wx/Pu54LfHT+CJJ2DIEF8Irld5XiPoB0wrez0d2LuD5U8Ebs0xHjNrwxtvpA7gbv7KGvzl7Rd4lU1Zl7c4h/M4nZ+y/p3vh16Tiw7TclQVF4slfR4YBOzfzvyRwEiA/h631Gy1TZoEN9+cHnffnS4G9+GTHMKtDOFmDuY2NmJuWnjqW4XGavnLMxG8BGxZ9nqLbNpyJH0COBvYPyLeaWtFETEaGA0waNCgqHyoZvVtyRK4//5lO/+nn07Td9gBTj01lX32/cIgek2dtOKbffBV9/JMBA8DAyVtRUoARwHHlC8gaQ/gEuDgiHg1x1jMGs6cOTBuXNrx33orzJ0La6wB++8PX/0qfPrT8MEPlr3hgvNh5EhYsGDZNA8a3xBySwQRsUTSKcA4oCdwWUQ8Kel8YGJE3Aj8BOgNtChdhZoaEYflFZNZPYtIff+Xjvrvuy91BNe3bxoPYMgQ+OQnYYMN2llBaXD4s8+GqVPTmcCoUR40vgEoorYqLYMGDYqJEycWHYZZVXjnHbjnnmU7/0lZZWf33dOOf8gQGDzY/f8bSHokItps/FsVF4vNrOtefRVuuSXt+MeNgzffhLXXTiN+nXkmHHpo6gHUrKucCMyqXAT84x/LjvofeihN69cvVW2GDIGPfzyV881WhROBWTVobl6uNr/gnAu5c5Oj3tv5v/RSuplrr73g/PPTzn+33XyDl1WGE4FZ0ZqbiS+P5IW3N+NvjOTmKUO448QDWQj07g2f+lTa8R9yCGy6adHBWj1yIjArQEQayH38eLj7m70Z//ZzzKAfAFvzAl/hEoZs8jAfndrMWmsVHKzVPScCs24QkW7iGj9+2ePll9O8zRjM/oznAO7mAO5mW55DALMEazUXGbY1CCcCsxwsXQpPPZW6bxg/PjXxfDW7ZbJfPzjwwHRj1wEHwDaf2BdNnbLiSnxHr3UTJwKzCli6FP75z6zUc3fa8b/2WprXvz8cfHDa8e+/P2y9dauLvBeM8h29VignArNV8O67qUlnqcxzzz2pCweArbaCz3wmHe3vv3/q3r9DvqPXCuZEYNYFS5bA448vK/Xcey/Mn5/mbbMNfO5zy474V6miM2KEd/xWGCcCs1Zt+Bk1isXDR/Doo8tKPX//e+q3H2DbbeHII5ft+Pv1KzR6s9XmRGCNrbkZRo5k0YLFTGQfxk/Zn/HHbsLfT1zMW++sAaSumkeMSKWe/faDzTYrNmSzSnMisIYTkQ7+H3oIJvy/t5iw4DYe4UO8TeqjYael/+K4Nf/E/n/8Ivvt55u4rP45EVjdmz8fHn442/FPSI9XXknz1uJY9uRRRjKa/biHj3IvfZkNbwqGfbHYwM26iROB1ZXFi1MzzgkTlu34n3kmnQUAbLdd6rJhr71g771h1//chTWnPr/iityG3xqIE4HVrAiYMmXZUf5DD8Ejj8DChWl+375pZ3/MMWnHP3gwvO99rVZywbluw28Nz4nAitNGa52OmlDOm5dKPOU7/tLdumuvDXvuCSedlHb+e+2V2u932jun2/CbeYQyK0jWWmeFI/HRo2HECBYtWlbiKT2efXbZottvn3b4pccuu6TxeM2sbR6hzKrP2We/lwQCeJGteGjBXkw4aSETLoZHH03DMAJsskna2X/hC+nn4MGw4YbFhW5Wb5wIrFssXZpa6kyZApMnw5QpRzGZJl5kKx5lT2axCQDrvLGAD/WEk09edrTfv78HYDHLkxOBVcTSpTBzZtrJpx398s+nTFl2hJ9cyPuZTRNTGMLN7MVD7M0Edu7/Bmvc20YrHjPLjRNBI1rJi7SQ+tqZMWPFHX3p59SpqelmuU02gaamNKTi0KHp+YAB6dH04DX0/voJK14juGB0RT+qmXXOiaDRtL5IO2UKjBzJ4iVi+n7HtHk0P3kyTJuWetwst9lmaec+eDAMG7b8jr5//04GU9/pSFh7iVvrmFUBtxrqTh0ciUekHe2iRenIenUeHa7jol+yeP4CFrMGs+jLZAYwmQG8RD+W0vO9UKXUmdpyR/FNy37275+abJpZbeio1VBjJIJVKIV05t13U2+U8+alx/z5Hf+c98xM5j89g3lLN+At1mMxa6THWr1ZvLTXCmWVvPTg3dKW2Yg5WRqYTBNTGXDp997b0W+5Jay5ZvfEZGb5a+zmo+2UQhYu6sG8Q47ufAfezrzXX+980+utB336pMeGz7/EJktfYSDP0ps339sZr7HWOqxx8ldYYw3afay5ZvvzuvIof3+PrT+YvoPWmprghO9V5Cs3s9pS/2cEAwbAlClcxvH8iG8xjz7MZ0PeoeO6Rs+eqa16nz7LfpY/72zehhtCr/I026PHsg5vykmpyU136eRGLjOrT419RjB1KgAbM5s9eIw+zKMP89iQ1+nzmwva3aGvt16F267379/2kXh3d27mLhXMrJWGOSNYQVNTag7TXXwkbmYF6uiMoEd3B9PtRo1asR1jEb1LjhiRdvpNTelUo6nJScDMqkL9l4aqqRTiAcrNrArVfyIA74DNzDpQ/6UhMzPrUK6JQNLBkp6V9Lyks9qYv5aka7L5EyQNyDMeMzNbUW6JQFJP4GLgEGBH4GhJO7Za7ERgbkRsA/wc+FFe8ZiZWdvyPCPYC3g+IiZFxCLgamBoq2WGAldkz68FDpTc87yZWXfKMxH0A6aVvZ6eTWtzmYhYAswH3t96RZJGSpooaeKsWbNyCtfMrDHVRKuhiBgNjAaQNEtSG3eI1ZSNgdlFB1FF/H0s4+9ief4+lrc630dTezPyTAQvAVuWvd4im9bWMtMl9QI2BF7raKUR0beSQRZB0sT27vBrRP4+lvF3sTx/H8vL6/vIszT0MDBQ0laS1gSOAm5stcyNwBez50cAd0at9XlhZlbjcjsjiIglkk4BxgE9gcsi4klJ5wMTI+JG4FLgj5KeB+aQkoWZmXWjXK8RRMQtwC2tpp1T9nwhMCzPGKqUB+Zdnr+PZfxdLM/fx/Jy+T5qrvdRMzOrLHcxYWbW4JwIzMwanBNBN5K0paS7JD0l6UlJ3yg6pqJJ6inpMUk3Fx1L0ST1kXStpGckPS3pP4qOqUiSvpn9n/xL0lWSOh5fto5IukzSq5L+VTZtI0m3S/p39vN9ldqeE0H3WgKcFhE7AvsAJ7fR/1Kj+QbwdNFBVIn/AW6LiO2B3Wjg70VSP+DrwKCI2JnU8rCRWhVeDhzcatpZwB0RMRC4I3tdEU4E3SgiZkbEo9nzN0j/6K273WgYkrYAPg38vuhYiiZpQ2A/UpNqImJRRMwrNKji9QLWyW42XReYUXA83SYi7iE1qS9X3jfbFcDhldqeE0FBsi639wAmFBxKkX4BnAksLTiOarAVMAv4Q1Yq+72k9YoOqigR8RLwU2AqMBOYHxF/LTaqwm0aETOz5y8Dm1ZqxU4EBZDUG7gOODUiXi86niJIGgK8GhGPFB1LlegF7An8b0TsAbxFBU/9a01W/x5KSpCbA+tJ+nyxUVWPrAeGirX9dyLoZpLWICWB5oi4vuh4CvRh4DBJk0ldlH9c0pXFhlSo6cD0iCidIV5LSgyN6hPAixExKyIWA9cD+xYcU9FekbQZQPbz1Uqt2ImgG2VjLVwKPB0RFxUdT5Ei4tsRsUVEDCBdBLwzIhr2iC8iXgamSdoum3Qg8FSBIRVtKrCPpHWz/5sDaeCL55nyvtm+CPy5Uit2IuheHwa+QDr6fTx7HFp0UFY1vgY0S3oC2B24oNhwipOdGV0LPAr8k7SvapjuJiRdBTwAbCdpuqQTgQuBT0r6N+mM6cKKbc9dTJiZNTafEZiZNTgnAjOzBudEYGbW4JwIzMwanBOBmVmDcyKwuiLp3axZ7r8ktUhadyXfv7mka7Pnu5c375V0mKSK3O0r6c1KrCfvdVpjcPNRqyuS3oyI3tnzZuCRVb15T9JxpN4vT6lgiKV1vxdnNa/TGoPPCKye3Qtsk/XjPlbSE5IelLQrgKT9y27se0zS+pIGZGcTawLnA0dm84+UdJykX2fvHSDpzmydd0jqn02/XNIvJd0vaZKkIzoLUtIZkh7O1nVeNu1CSSeXLXOupNPbW95sdTgRWF3Kui4+hHRX6nnAYxGxK/Ad4P+yxU4HTo6I3YGPAm+X3h8Ri4BzgGsiYveIuKbVJn4FXJGtsxn4Zdm8zYCPAEPo5O5PSQcBA4G9SHcTf0jSfsA1wPCyRYcD13SwvNkqcyKwerOOpMeBiaT+ai4l7ZT/CBARdwLvl7QBcB9wkaSvA30iYslKbOc/gD9lz/+YbaNkbEQsjYin6Lyr4IOyx2Ok7hS2BwZGxGPAJtk1i92AuRExrb3lVyJusxX0KjoAswp7OzvCf0/qs2xFEXGhpL8AhwL3SfoUsLACMbxTvvlOlhXww4i4pI15LcARwAdIZwidLW+2SnxGYI3gXmAEgKQDgNkR8bqkD0bEPyPiR8DDpKPrcm8A67ezzvtZNnTiiGwbq2IccEI2RgWS+knaJJt3TbaNI0hJobPlzVaJzwisEZwLXJb16rmAZV35nirpY6QR0p4EbiXV90vuAs7KSk0/bLXOr5FGEzuDNLLY8asSWET8VdIOwAPZmcubwOdJg/Y8KWl94KXSyFQdLb8q2zcDNx81M2t4Lg2ZmTU4JwIzswbnRGBm1uCcCMzMGpwTgZlZg3MiMDNrcE4EZmYN7v8Dtv20vC/hVYUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fitting Polynomial Regression to the dataset\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "poly_reg = PolynomialFeatures(degree=5)\n",
    "X_poly = poly_reg.fit_transform(X)\n",
    "pol_reg = LinearRegression()\n",
    "pol_reg.fit(X_poly, y)\n",
    "\n",
    "# Visualizing the Polymonial Regression results\n",
    "def viz_polymonial():\n",
    "    plt.scatter(X, y, color='red')\n",
    "    plt.plot(X, pol_reg.predict(poly_reg.fit_transform(X)), color='blue')\n",
    "    plt.title('Truth or Bluff (Linear Regression)')\n",
    "    plt.xlabel('Position level')\n",
    "    plt.ylabel('Salary')\n",
    "    plt.show()\n",
    "    return\n",
    "viz_polymonial()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "c141851b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fa344ab2ac0>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhUklEQVR4nO3de3RU1fn/8fdDCBBAiQiKRDBQkRalEo0WiTcohaIWEbVVf1rtqlL9WmulaEG/XmtrKtr6VeuFVS9t8YIXjFTQeMEbKCg0IKLFWgUl0IoK3ojKZf/+2AkJYZJMJnPmnJnzea01K8nMycw+mcwze5797L3NOYeIiERXu7AbICIizVOgFhGJOAVqEZGIU6AWEYk4BWoRkYhrH8Sd9ujRwxUXFwdx1yIiOWnx4sUfOud6JrotkEBdXFzMokWLgrhrEZGcZGarmrpNqQ8RkYhToBYRiTgFahGRiFOgFhGJOAVqEZGIC6TqIxtVVFUztXIFazbU0LuwgAtHD2RcSVHYzRIRUaAGH6SnzFxGzaYtAFRvqGHKzGUACtYiEjqlPoCplSu2Bek6NZu2MLVyRUgtEhGpp0ANrNlQ06rrRUQySYEa6F1Y0KrrRUQySYEauHD0QAry87a7riA/jwtHDwypRSIi9TSYSP2Aoao+RCSKFKhrjSspUmAWkUhKKlCb2UrgM2ALsNk5Vxpko0REpF5retTDnXMfBtYSERFJSIOJIiIRl2ygdsCTZrbYzCYkOsDMJpjZIjNbtG7duvS1UEQk5pIN1Ic65w4AxgDnmtnhjQ9wzk1zzpU650p79ky4m4yIiKQgqUDtnKuu/foB8AhwcJCNEhGRei0OJppZF6Cdc+6z2u9HAVcF3rKAaJU8Eck2yVR97A48YmZ1x9/rnHsi0FYFRKvkiUg2ajFQO+feAfbPQFsC19wqeQrUIhJVsSrP0yp5IpKNYhWotUqeiGSjWAVqrZInItkoVosyaZU8EclGsQrUoFXyRCT7xCr1ISKSjRSoRUQiToFaRCTiYpWj1vRxEclGsQnUYU4f1xuEiLRFbFIfzU0fD1LdG0T1hhoc9W8QFVXVgT6uiOSO2ATqsKaPh/UGISK5IzaBuqlp4u3MAu3dan0REWmr2ATqRNPHAbY4F2gqQuuLiEhbxSZQjysp4prxg8nz62pvJ8hUhNYXEZG2ik2gBh+stzqX8LagUhF1bxBFhQUYUFRYwDXjB6vqQ0SSFpvyvDq9CwuoThCU63LVQQRQrS8iIm0Rqx41BJerrqiqpqx8Lv0mz6asfK7K70QkbWIXqIPIVatWWkSCFLtADenPVatWWkSCFMtADektm1OttIgEKVqBeuvWjD1UOsvmVCstIkGKTqD+4gsYORLuuCMjD5fOsjnVSotIkKJTnpeXBx07wpln+p71WWcF/pDpKpvTXowiEqToBOpOneCRR+D442HCBHDOf80SqpUWkaBEJ/UBPljPnAlHHw0/+xncdlvYLRIRCV3SgdrM8sysysweC7JBdOwIDz8MxxwD55wDt9wS6MOJiERda3rU5wNvBtWQ7XTsCA89BGPHwrnnws03Z+RhRUSiKKlAbWZ7AkcDfw62OQ107AgPPgjHHgvnnQc33pixhxYRiZJke9Q3ABcBTRY6m9kEM1tkZovWrVuXjrZBhw7wwANw3HFw/vlwww3puV8RkSzSYqA2s2OAD5xzi5s7zjk3zTlX6pwr7dmzZ9oaSIcOMGOGrwa54AL44x/Td98iIlkgmR51GTDWzFYC9wMjzGx6oK1qLD8f7rsPTjgBJk6EqVMz+vAiImFqsY7aOTcFmAJgZkcCk5xzpwbbrATy8+Hee/3EmIsugo0b4bLLIMEqeG1VUVWtySsiEhnRmfCSjPx8uOceKCiAK66Azz+Ha69Na7CuW7K0bjW8uiVLAQVrEQlFqwK1c+454LlAWpKsvDy/HkiXLnDddX6NkJtvhnbpmbvT3JKlCtQiEobs6lHXadcObrrJB+trr/XB+o47oH3bT0dLlopI1GRnoAaf7igvh65dfa66pgamT/dVIm3Q1J6KWrJURMISrbU+WssMLr0Urr/eT44ZPx6+/LJNd6klS0UkarK3R93QxIm+Z3322X5Bp0cf9T+nQEuWikjU5EagBr8kaufOcMYZfgOC2bNh111TuistWSoiUZLdqY/GTj3Vr7y3ZAkcdhisXh12i0RE2iy3AjX4RZwqK32QLiuDFfU7gVdUVVNWPpd+k2dTVj6XiqrqEBsqIpKc3AvUAEccAc8/7wcWDz0UFi/eNpGlekMNjvqJLArWIhJ1uRmoAUpKYN48P6h45JE8c8v9TU5kERGJstwN1AADBsD8+VBczHV3TWb0ipd2OEQTWUQk6nI7UAP07g3PP89bRftwy6Pl/Ghp5fY3ayKLiERc7pTnNad7d1beV8H6H5/C75+4id0//5gbh52EmVG9oYZvTJnDFucoUs20iERQ7veoa/1g2ADW3/sgf99/JBPn3cPvKm+m3Vafs97iHKABRhGJptgEagCXn89Fx0zkpkN+xClLK5k282oKvt5+yrkGGEUkamIVqKdWrqBm81auP/w0Lh59Lke+s5j775vCrl9s2O44DTCKSJTEKlA3DMD3DhnDhPGXsM+H7zFz+iSKP65Pd2iAUUSiJFaBunEAfmbv73Dyyb+j61cbeXj6hQxZs0Ir5YlI5MQqUCdawnRJ74GccOpUPuvYhfvvv5i7uq9R1YeIREqsAvW4kiKuGT+YosICDCgqLOCGHw3h2WkTKF6xhE4l+zP0V2fCH/8ItZUgIiJhMxdAQCotLXWLFi1K+/0GbuNGqseeSNEzc5g+ZAzTTriAiUcNUg9bRAJnZoudc6WJbovHhJcW/G/FMu5b+D5bnMMOPJsLv+jM/yx4iL4b/sOvNlwCDFWwFpHQxCr1kcj/Vixj+oL3tk16cdaOa484gwvH/IJD3nuNe+6ayN/ufz7kVopInMWqR11RVb3DFlv3LXw/4bEPfnsUq7v14rZHfsvtN58Lx/WHoUMz3GIRkRj1qJtaj3pLMzn6l/f6NuNPu44vO3WGI4+EGTMy1l4RkTqxCdRTK1ckXI+6JWt6FfPag09AaSmcdBJceSVs3RpUM0VEdtBi6sPMOgEvAB1rj3/IOXd50A1Lt1SmhdetpndUSRGUPeM30L3iCli2DO6+O+FO54nSKxqIFJG2SCZH/RUwwjn3uZnlA/PM7HHn3IKA25ZWvQsLqE4QrIsKCxj+zZ7bqj7yzDj5O324etzg7Q/s2NEH5yFDYNIk+Ne/4NFHobh42yF16ZW6nnpdegVQsBaRlLWqjtrMOgPzgHOccwubOi6KddSNgyhAQX4e14wf3PogWlnp0yDt28NDD1Gx895MrVyR8I0A/JvB/MkjUmqzeuci8dBcHXVSOWozyzOzJcAHwFPNBemoSjQrMaUgDTB6NCxcCLvuytaRI1l6yTVNBmlILe2izXhFpE5re9SFwCPAec651xvdNgGYANC3b98DV61alcZmRtQnn/DSd0YzbMVCpg8Zw5UjJ7ApL3+Hw1LpUZeVz20yVZNK71xEoq3NPeo6zrkNwLPA9xPcNs05V+qcK+3Zs2dKDc063bpx2tiLufU7J3DqkseZPuNSenyxfrtDUl2Nr6leuNbKFomfFgO1mfWs7UljZgXA94B/BtyurNGre1d+f+QZ/OIHk/j22n/x2N3nc0D1m0Db0itNrYmttbJF4ieZHvUewLNm9hrwKj5H/ViwzcoOFVXVfPHVZgBmDTqS8adN5av2Hbj/3ik8akuZ/+vhKQ/+JVqSVWtli8STVs9LUaIqEoC+9iUPzLuVXvOegdNOg9tug86dU34MVX2IxINWzwtAopmOAFu67UKv55+Eq6/2k2Neew1mzoT+/Vv9GONKihSYRSQ+U8jTralBveoNNdCuHVx2GcyeDatWwYEHwpw5GW6hiOQKBeoUNTWoZ1Bf6zxmDCxe7GcvHnMMXH45bGl5fRERkYYUqFN04eiBWILrHT4tsk3//jB/Ppx+Olx1FXzve7B2baaaKSI5QIE6ReNKimhqGHaHtEjnznDXXf6yYIFfL+Tpp4NuoojkiFgG6oqqasrK59Jv8mzKyuemPC27qLW1zmecAa++Cj16wKhRPo+tVIiItCBWgbqiqpohVz7JL2csScsaGinVOu+7L7zyig/av/kNjBwZmVRIut7ARCS9YhOo6+qeN9Rs2uG2mk1bts8rJynlhZ66dIE77/TLpr7yik+FPPVUqx8/nVpaBEpBXCQ8samjbqruuU6icrtkJpy0qdb59NPhoIPgxBP9inwXXeQHHDt0SO3+2qCpHXDq3sC0zrZIeGLTo25pMaPGeeWMLTM6aJDPW595Jvz+9zBsGLz1VnofIwnNLQLVUhAXkWDFJlA3t5hRorxyRoNT584wbRo8/DC8+y6UlMAdd0AA0/ub0twiUFrJTyRcsQnUiQb+AHbpnJ8wrxxKcBo/3k85HzrU97BPPBE+/ji4x2uguYFRreQnEq7Y5KjrAnGyixw1tcdi4MGpqMgPLF53HVxyCRvnvcRFYycxu/vAQBdmaunvk2gbM63kJ5IZWj2vCWndYzFFz97zOMW/OIu9Pl7D7d85nj8e+v/IK+iU0TbU0Up+IsFqbvU8BepmhB2cysrn8vEH67nsmWmc/NqTvNmzmInHTOTTffbVdlwiOUaBOkv1mzx72zT1EW+/QvkTN1FY8xk3lp3MpLl3+l3QRSQnKFBnicY9+I1fb2b9xvoJOoU1n/KbJ2/lB/98EQ4+GP7yF/jmNwNvh9IcIsFToM4CiXLi+e0MDDZtqX+OCvLz+GuXdzlo6v/Cxo1wzTXwi1/4NbCbuN/WBN0o5OZF4ihtu5BLcBLVbW/a6ujSof0OU9QPmnwOLF/u1wm54AIYMcLXXzeSyqQdTW4RiR4lOSOiqfrsT2o2seTyUTve0KsXzJrl1ws5/3zYbz/47W/hvPMgz9dDNxd0m+oda3KLSPSoRx0RKU0qMYOf/MT3rocP973rsjL/M6kFXU1uEYkeBeqISGnJ1Dp9+sDf/w733AP//refgn7llfTtmvgDU3NBt03tEJFAKPUREa2dOZlwkPCUU/xWX7/8JVxxBX//xkDOOvxsFu42YNvvtRR0W9sOEQmeqj6yUFKVGbNnw9ln46qrefCQ47jq4JPotvuuCroiEdVc1Yd61CkIu844qUHCo4+G5cuxKVP44S238MNVC+H//g+G9M5YO0XSIezXWxQoR91KGVunuhlJDxLuvDP86U/w0kt+n8YTTvAB/J13MtBKkbaLwustCloM1GbWx8yeNbM3zGy5mZ2fiYZFVRTqjFtdmXHIIbBoEfzhD/Dii37fxquvhq++CrCVIm0XhddbFCTTo94M/Mo5NwgYCpxrZoOCbVZ0RaHOOKXKjPbtffneP/8JP/gBXHop7L8/PPNMwK0VSV0UXm9R0GKgds6tdc79o/b7z4A3gXgliBqIQp1xypvqgl/v+oEH4PHHYfNmP7vx5JNh9erA251p2pA3+0Xh9RYFrar6MLNi4AVgP+fcp41umwBMAOjbt++Bq1atSmMzoyOTa2EEPohSUwPl5XDttX6tkMmTYdIkKIj2iyCZv0u6nycNaIUjTmvPpGVRJjPrCjwP/NY5N7O5Y3O9PC8TL9qM/oOuXOl3QH/wQejb1+8uc8IJfuZjxCT7dykrn5twh56iwoJWr+Udp2ARRXF5k2xzoDazfOAxoNI594eWjs/1QJ0J6Qw0SXv+eb9uyNKlcNhhvpyvpCSYx0pRsn+Xhmt5N2TAu+VHB/KYIm3RptXzzMyAO4A3kwnSkh6hDKIccQQsXgy33w5vvgkHHggTJsB//xvcY7ZSsn+XdOY2ozSgpbx7PCVT9VEGnAaMMLMltZejAm5X7IU2iJKX54Pzv/7lp6LfdRfsvTdcdRV8/nmwj52EZP8u6VyzJCoDWqopjq9kqj7mOefMOfdt59yQ2sucTDQuSC31TMLuuYS+OFJhoa+7Xr4cRo+Gyy/3Afu222DTphZ/PSjJ/l3aVBmT4mMGTTXF8RXLtT5aGhyKyuBRpAZRXn7ZDzjOmwcDB/qdZcaNC2XAMYy/SxSei3Tm3SV6tBVXIy0NDmnwqAnO+eVUf/1rP3Fm2DBf2ldWFnbLYkH/l7lNW3E10tLgUJQGjyLFDMaOhWXLYNo0v2bIoYfCUUf5KeoSqKikYCTzYhmoWxocisrgUWS1bw9nnQVvv+1TIAsXwkEH+VTI0qVhty5npTPvLtklloG6pZ6Jei5J6tLFz2Z8911fFfLcczBkCJx44rbtwCS9xpUUMX/yCN4tP5r5k0coSMdELAN1Sz0T9Vxaaeed/SJPK1fCZZdBZSUMHgynnOJz2RkWdsWOSLrFcjAxm0Sh2qDVPvoIrr8ebrwRNm6E44+HKVPggAMCf+ioVOyItJYGE7NU1k5w2HVX+N3vfErk4ovhqaf8LMcxY/x62AFSrbHkIgXqCMv6oNOzp9+gYNUqP+i4eDEcfrivFJkzx5f7pZkqdiQXKVBHWM4EnW7d/KDjypVw003w/vt+S7CSEpg+Hb7+OuW7bpyP7laQn/C4MCt2lDOXtlKgjrCcKxPs3Bl+/nNf1nf33X4rsNNOg379fI/7o49adXeJUkNffL2Z/Hbbz5YMs2Ina9NXEikK1BGWs2WC+flw+um+hG/OHL+H48UXQ58+cPbZSVeKJEoNbdri6NqpfWQqdqKavlIvP7u0D7sB0rS64JJ1VR/JatfODzCOGQOvvw433OB72rff7q+74AL47nf9cQk0lQLasHETVZeNCq7drRDF9FXjypi6Xj6QO/9bOUaBOuLGlRTF48Wz337w5z/7apHbboNbboFRo2DAAPjZz+CMM3w1SQO9CwsSrn0RpdRQFNvYXC8/Fv9rWUipD4mW3Xbzk2ZWrYK//c3/PGmS35T3xz+Gl17aVi2SDamhKLYxir18aZ4CtURTx45w6ql+WdXXXoMzz4SKCr9S35AhcOutjPvGTpGfQRrFWa45N0gdA5qZKNnj88/hvvvg1luhqsrvlj5+vB+YHDHC704jLdLszWjSzETJDV27+lX7Fi/2K/adfjrMnu1z2cXFvnJkRZZMBgpRFHv50jz1qCVlkViH5MsvYdYs+Mtf4IknYOtWGDrUB/ETT9xhAFIkqtSjlrSLzESOTp3ghz/0PevVq2HqVPjsMzjnHOjVy5f53X03bNiQ2XZJfDgH//iHn317yimBPIQCtaQkkhM59tjDV4gsW+bTIxMn+skzP/kJ7L67353mnnvg00/Da6PkBuf8IPcll8A++/hFx66/3ncIAtj8WXXUkpJIl3iZ+SVVDzgAysvh1Vdhxgx44AG/52PHjr6nfeyxfs2Rnj3DbrFkgy1bYP58X3306KN+K7q8PD+Q/etfw3HHBZZqU6CWlERxIkdCZnDwwf4ydSosWOCD9sMP+xecmd+kd+xYfxk4MJSd1SWiNm70y/RWVMBjj8GHH0KHDjByZH1wzsAbvQYTJSVZX+LlnC/xmzXLX6qq/PUDBviAfdRRvma7Y8dw2ymZ5Rz8+99+l6LKSnj6aaipgcJC/+lr3DgYPRp22intD93cYKICtaQsElUf6fL++z4tMmsWzJ3r84wFBXDEEfC97/nLfvupt52LPv3UP+d1wfndd/31/fv7N+xx4/w66vmJl9BNFwVqkdb47DN49ln/kfepp+prs3v1qg/aRxwBffuG205JzcaNPgX2wgs+QL/8Mmze7Ov0R4zwPeZRo2DvvTParDYFajO7EzgG+MA5t18yD6hALWEIrIf/3nv1Qfvpp+vXze7TBw47zF8OPRQGDWpypT8J0Sef+EHAF17wl1df9YG5XTu/ecWoUT44H3KIzz+HpK2B+nDgc+CvCtQSVYly5gCFBflcMXbfNgXshm8ARTt35Df9tjB83Qq/DsmLL8J//uMP7N7d57WHDYODDvJVJ7vs0pbTip02v9lu3gxvvOGD8Suv+MvSpT73nJ/vn5fDD/eXYcP87kMR0ebUh5kVA48pUEtUlVz1JOs3Jq5fbcsgZ4uDps75Mq0XX/SBe9687aexf+MbUFrq62xLS33wjlBwiJJWD1Bv2eJ3C/rHP+oDc1WVT22A/zuXlvpPO4cf7mesdu6cobNpvYwEajObAEwA6Nu374GrVq1KrbWStcIaXKyoquaXM5Y0e0xRYQHzJ49o9X2Xlc9NWIbY7P19/LGfcLN4MSxa5C8NXw/9+vldbRpevvUtP3gZY03+rbt1Yv6Pv+knMr3+uv+6bBm8+aZfQgD8DNWSEt9jPvhg/3XvvbMqFdVcoE5bHbVzbhowDXyPOl33K9khzF1DkpkNmepEnJQm9nTvXj/oWOfDD+sD97Jlfhuyysr6WWxmvspg3319gOnfv/5SXJz7ZYLO8eWa/3DA+jXstX4te21Yy17r11K8fi391lfDxZ/XH7vHHjB4MAwf7r/uv7+vyAm4KiNMmvAiaRHmriHJBOFUJ+KkbWJPjx5+wGr06PrrNm3yH91ff90H7uXLfS/xqad87W4dM9hzz/qgvcce0Lv39l/32MP3KqPIOT+1uro68WX1anjnHRY3mNq/FaO6226sKuzFs/sP57jTRvugvO++sVxoS4Fa0iLMKeVNBdM6bdlR5cLRAxPmTdOyQ0t+vk95fOtbfqW/Os75Acp33tnxMneuvy3RehK77OLfEAoL/fe77LL99926+Rxtp06JL+3b+8d2jmfe+A93zV/Jh599yW5dO3DGsGJG9Ovm8781Ndt/3bgRvvjCp3waXj76qP77RO3t0cPv3FNUBIceymudenDL+8ZbO+3O6m6783X7/G05arK1Pj9NWgzUZnYfcCTQw8xWA5c75+4IumGSXcKcUp4omBrg8LnktuTKQ9lg2Ky+l1xWtuPtW7f6ILhmDaxdu/3Xjz7yvdf16/3EjbrvN29uVRO+W3vZ5sYkfqlzZ5/26d7d93oHDar/ebfd6oNyUZH/JNAonfNt4PtV1SyrXMGmDTVtfu5yiSa8SFqEPaU8p2ZJpptzvsf7ySd+8K2py6ZNYMaljy7no42bqIsMzgyHsVO3Llx3+iF+0LNz5+2/dumS+3n0gGVkMFHiLZSeZ6PHV2Bugpmfdde1a1KHT6+aTaLumwHXDR+e1qZJchSoJW0ULHND1qyMGCPZU2QoIhlx4eiBFORvv1Fw2gZQJSXqUYvIdsJOY8mOFKhFZAdKY0WLUh8iIhGnHrVITOVaSWOunU9DCtQiMRTm2ixByLXzaUypD5EYam5tlmyUa+fTmHrUaZbLH78kd4S5NksQcu18GlOPOo3qPn5Vb6jBUf/xq6KqOuymiWynqckr2TqpJdfOpzEF6jTK9Y9fkjtybVJLrp1PY0p9pFGuf/yS3JFrk1py7XwaU6BOI62RINkk1ya15Nr5NKTURxrl+sevXFNRVU1Z+Vz6TZ5NWflcjSVIZKlHnUa5/vErl4Rdd6vqIGkNBeo0y+WPX7kkzD0ew36TkOyj1IfEUpgDv6oOktZSj1piKcyB32yvDlLaJvPUo5ZYCnPgN5snZ2hSVzgUqCWWxpUUcc34wRQVFmD43coztRFvNlcHKW0TDqU+JLbCGvjN5uqgbE/bZCsFapEQZGt1kCZ1hUOpDxFJWjanbbKZetQikrRsTttkMwVqEWmVbE3bZLOkUh9m9n0zW2Fmb5vZ5KAbJSIi9VoM1GaWB/wJGAMMAk42s0FBN0xERLxketQHA287595xzn0N3A8cG2yzRESkTjKBugh4v8HPq2uv246ZTTCzRWa2aN26delqn4hI7KWtPM85N805V+qcK+3Zs2e67lZEJPaSCdTVQJ8GP+9Ze52IiGRAMoH6VWCAmfUzsw7AScCsYJslIiJ1Wqyjds5tNrOfA5VAHnCnc2554C0TEREgyQkvzrk5wJyA2yIiIglorQ8RkYhToBYRiTgFahGRiFOgFhGJOAVqEZGIU6AWEYk4BWoRkYjTxgEiMVVRVa2dWrKEArVIDFVUVTNl5jJqNm0BoHpDDVNmLgNQsI4gpT5EYmhq5YptQbpOzaYtTK1cEVKLpDkK1CIxtGZDTauul3ApUIvEUO/CglZdL+FSoBaJoQtHD6QgP2+76wry87hw9MCQWiTN0WCiSAzVDRiq6iM7KFCLxNS4kiIF5iyh1IeISMQpUIuIRJwCtYhIxClQi4hEnAK1iEjEmXMu/Xdqtg5YlfY7Tp8ewIdhNyJNdC7RkyvnATqXTNrLOdcz0Q2BBOqoM7NFzrnSsNuRDjqX6MmV8wCdS1Qo9SEiEnEK1CIiERfXQD0t7Aakkc4lenLlPEDnEgmxzFGLiGSTuPaoRUSyhgK1iEjE5XSgNrPvm9kKM3vbzCYnuL2jmc2ovX2hmRWH0MykJHEuE83sDTN7zcyeMbO9wmhnS1o6jwbHHW9mzswiW06VzLmY2Q9rn5flZnZvptuYrCT+v/qa2bNmVlX7P3ZUGO1siZndaWYfmNnrTdxuZnZj7Xm+ZmYHZLqNKXHO5eQFyAP+DfQHOgBLgUGNjvkf4Lba708CZoTd7jacy3Cgc+3350TxXJI5j9rjdgJeABYApWG3uw3PyQCgCtil9ufdwm53G85lGnBO7feDgJVht7uJczkcOAB4vYnbjwIeBwwYCiwMu83JXHK5R30w8LZz7h3n3NfA/cCxjY45FvhL7fcPAd81M8tgG5PV4rk45551zm2s/XEBsGeG25iMZJ4TgN8Avwe+zGTjWimZczkL+JNzbj2Ac+6DDLcxWcmciwN2rv2+G7Amg+1LmnPuBeDjZg45Fvir8xYAhWa2R2Zal7pcDtRFwPsNfl5de13CY5xzm4FPgF0z0rrWSeZcGvopvtcQNS2eR+1H0T7OudmZbFgKknlO9gH2MbP5ZrbAzL6fsda1TjLncgVwqpmtBuYA52WmaWnX2tdSJGiHlxxjZqcCpcARYbeltcysHfAH4IyQm5Iu7fHpjyPxn3BeMLPBzrkNYTYqRScDdzvnrjezQ4C/mdl+zrmtYTcsDnK5R10N9Gnw85611yU8xsza4z/SfZSR1rVOMueCmY0ELgHGOue+ylDbWqOl89gJ2A94zsxW4nOIsyI6oJjMc7IamOWc2+Scexd4Cx+4oyaZc/kp8ACAc+5loBN+kaNsk9RrKWpyOVC/Cgwws35m1gE/WDir0TGzgNNrvz8BmOtqRxwipsVzMbMS4HZ8kI5qLrTZ83DOfeKc6+GcK3bOFeNz7WOdc4vCaW6zkvn/qsD3pjGzHvhUyDsZbGOykjmX94DvApjZt/CBel1GW5kes4Af11Z/DAU+cc6tDbtRLQp7NDPIC36E9y38iPYltdddhX/xg/9nexB4G3gF6B92m9twLk8D/wWW1F5mhd3mVM6j0bHPEdGqjySfE8Onct4AlgEnhd3mNpzLIGA+viJkCTAq7DY3cR73AWuBTfhPND8FzgbObvCc/Kn2PJdF+f+r4UVTyEVEIi6XUx8iIjlBgVpEJOIUqEVEIk6BWkQk4hSoRUQiToFaRCTiFKhFRCLu/wOJ9ifa8XCLzwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.pipeline import make_pipeline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "def make_data(N, err=1.0, rseed=1):\n",
    "\n",
    "    rng = np.random.RandomState(rseed)\n",
    "    X = rng.rand(N, 1) ** 2\n",
    "    y = 1. / (X.ravel() + 0.3)\n",
    "    if err > 0:\n",
    "        y += err * rng.randn(N)\n",
    "    return X, y\n",
    "\n",
    "def PolynomialRegression(degree=2, **kwargs):\n",
    "    return make_pipeline(PolynomialFeatures(degree), LinearRegression(**kwargs))\n",
    "\n",
    "\n",
    "X, y = make_data(60)\n",
    "X_test = np.linspace(-0.1, 1.1, 200)[:, None]\n",
    "\n",
    "param_grid = {'polynomialfeatures__degree': np.arange(20),\n",
    "'linearregression__fit_intercept': [True, False],\n",
    "'linearregression__normalize': [True, False]}\n",
    "grid = GridSearchCV(PolynomialRegression(), param_grid, cv=7)\n",
    "grid.fit(X, y)\n",
    "\n",
    "model = grid.best_estimator_\n",
    "\n",
    "y_test = model.fit(X, y).predict(X_test)\n",
    "\n",
    "plt.scatter(X, y)\n",
    "plt.plot(X_test.ravel(), y_test, 'r')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76ef4e97",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "decc5d6e",
   "metadata": {},
   "source": [
    "<h2 align=\"center\">Linear Regression: Regularization</h2>\n",
    "\n",
    "**Main Idea:**\n",
    "- Regularization works on assumption that smaller weights generate simpler model and thus helps avoid overfitting.\n",
    "\n",
    "$$f(x_i) = \\theta_0 + \\theta_1x_1 + \\theta_2x_2^2 + \\theta_3x_3^3 + \\theta_4x_4^4$$\n",
    "$$f(x_i) = \\theta_0 + \\theta_1x_1 + \\theta_2x_2^2 $$\n",
    "\n",
    "<img src=\"images/p125.png\">\n",
    "\n",
    "**Model:** \n",
    "$$h_\\theta (X) = \\theta^TX = \\theta_0 + \\theta_1x1$$\n",
    "\n",
    "**Loss Function:**\n",
    "\n",
    "\n",
    "$$J(\\theta) = \\frac{1}{2n} \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2$$\n",
    "\n",
    "**Goal:**\n",
    "$$ min_\\theta \\;\\; J(\\theta) = \\frac{1}{2n} \\sum_{i=1}^n \\left( (\\theta_0 + \\theta_1x1) - y^{(i)} \\right)^2$$\n",
    "\n",
    "- The concept is broad but we will see in the context of linear regression or polynomial regression which we formulated as linear regression. Encourages the model coefficients to be small by adding a penalty term to the error.\n",
    "- The over-fitting phenomenon can be controlled with `regularization`, which involves adding a penalty term to the error function. \n",
    "\n",
    "**We add a `penalty term`, known as `regularizer`, in the loss function**\n",
    "\n",
    "\n",
    "$$h_\\theta(X) = \\theta_0+\\theta_1x_1+\\theta_2x_2+....+\\theta_nx_n$$\n",
    "\n",
    "$$min_\\theta \\; J(\\theta) = min_\\theta \\; \\frac{1}{2n} \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2$$\n",
    "\n",
    "$$min_\\theta \\; J(\\theta) = min_\\theta \\; \\frac{1}{2n} [ \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2 + \\lambda \\sum_{i=1}^n \\theta_j^2 ]$$\n",
    "\n",
    "$$min_\\theta \\; J(\\theta) = min_\\theta \\; \\frac{1}{2n} [ \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2 + \\lambda \\sum_{i=1}^n |\\theta_j| ]$$\n",
    "\n",
    "- $\\lambda \\geq 0$ maintains the trade-off between regularizer and the original loss function as it controls the relative importance of the regulrization term.\n",
    "\n",
    "- `λ` is the penalty term or regularization parameter which determines how much to penalizes the weights\n",
    "- When `λ` is zero then the regularization term becomes zero. We are back to the original Loss function\n",
    "- When `λ` is large, we penalizes the weights and they become close to zero. This results is a very simple model having a high bias or is underfitting.\n",
    "    -  $h_\\theta(X) = \\theta_0$\n",
    "\n",
    "- Use cross-validation to find the optimal value of `λ`.\n",
    "\n",
    "### L2 Regularization or Ridge Regression\n",
    "\n",
    "- We require to discourage the model coefficients from reaching large values. \n",
    "$$min_\\theta \\; J(\\theta) = min_\\theta \\; \\frac{1}{2n} [ \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2 + \\lambda \\sum_{i=1}^n |\\theta_j| ]$$\n",
    "\n",
    "- This regularization term maintains a trade-off between `fit of the model to the data` and `square of norm of the coefficients`.\n",
    "    - If model is fitted poorly, the first term is large.\n",
    "    - If coefficients have high values, the second term (penalty term) is large.\n",
    "- Large $\\lambda$ penalizes coefficient values more.\n",
    "\n",
    "\n",
    "**Intuitive Interpretation:**  We want to minimize the error while\n",
    "keeping the norm of the coefficients bounded.\n",
    "\n",
    "- Regularized loss function is still quadratic, and we can find closed form solution by taking gradient with respect to $\\mathbf{\\theta}$, after gradient we have a solution of the ridge regression.\n",
    "\n",
    "$$\\mathbf\\theta(\\lambda) = (X^TX+\\lambda \\mathbf{I})^{-1} X^Ty$$\n",
    "\n",
    "- $\\lambda = 0$, we have non-regularized solution.\n",
    "- $\\lambda = \\infty$, the solution is a zero vector.\n",
    "- Too small $\\lambda$: no regularization.\n",
    "- Too large $\\lambda$: no weightage to the data.\n",
    "- In practice, we use very small value of $\\lambda$ and therefore it is convenient to work the **$\\ln \\lambda$** and compute it as $\\lambda = e^{\\ln \\lambda}$.\n",
    "\n",
    "\n",
    "<img src=\"images/p126.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e934a0bd",
   "metadata": {},
   "source": [
    "#### Example:\n",
    "- $\\lambda$ restricts the coefficients from exploding as we have included the sqaure of the norm of the coefficients in the loss function being minimized. \n",
    "\n",
    "<img src=\"images/p127.png\">\n",
    "\n",
    "- $\\lambda$ is a hypermeter of the model and we learn it in practice using the validate data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d89434f3",
   "metadata": {},
   "source": [
    "#### Ridge Regression: Graphical Visualization\n",
    "- $\\mathbf{\\theta} = [\\theta_1, \\theta_2]$, we assume we have two coefficients Our loss function is :\n",
    "\n",
    "$$min_\\theta \\; J(\\theta) = min_\\theta \\; \\frac{1}{2n} [ \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2 + \\lambda \\sum_{i=1}^n |\\theta_j| ]$$\n",
    "\n",
    "- Good Value of $\\lambda$ hepls us in avoiding overfitting.\n",
    "- Irrelevant features get small but non-zero value in the regularized solution.\n",
    "\n",
    "- **Ideally:** we would like to assign zero weight to the irrevlevant features.\n",
    "\n",
    "\n",
    "<img src=\"images/p128.png\">\n",
    "\n",
    "- `L2` regularization forces the weights to be small but does not make them zero and does non-sparse solution.\n",
    "- `L2` is not robust to outliers as square terms blows up the error differences of the outliers and the regularization term tries to fix it by penalizing the weights\n",
    "- Ridge regression performs better when all the input features influence the output and all with weights are of roughly equal size."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cef45018",
   "metadata": {},
   "source": [
    "**Example**\n",
    "\n",
    "<img src=\"images/p129.png\">\n",
    "\n",
    "This plot shows us a few important things:\n",
    "\n",
    "- **Y-axis:** Regularized coefficients for each variable (ie. coefficients after penalization is applied)\n",
    "- **X-axis:** Logarithm of the penalization parameter Lambda (). The higher value of lambda indicates more regularization (ie. reduction of the coefficient magnitude, or shrinkage)\n",
    "- **Curves:** Change in the predictor coefficients as the penalty term increases.\n",
    "- **Numbers on top:** The number of variables in the regression model. Since Ridge regression doesn’t do feature selection, all the predictors are retained in the final model.\n",
    "- **Red dotted line:** The minimum value of lambda (lambda.min) that results in the smallest cross-validation error. This is calculated by dividing the dataset in ten subsets, followed by the calculation of fit in 9/10 of the subsets and testing the predicted model on the remaining 1/10. We would use ideally use this lambda value (or lambda.1se below) as our penalization level for predicting outcomes in a new dataset.\n",
    "- **Blue dotted line:** The largest value of lambda (ie. more regularized) within the 1 standard error of the lambda.min. This lambda.1se value corresponds to a higher level of penalization (ie more regularized model) and can be chosen for a simpler model in predictions (less impact from from coefficients)\n",
    "- **Log Lambda = 0** corresponds to “no regularization” (ie. regular linear model with minimum residual sum of squares).\n",
    "\n",
    "\n",
    "The way we read the plot is as follows:\n",
    "\n",
    "- Among the variables in the data frame, **watched_jaws** has the strongest potential to explain the variation in the response variable, and this remains true as the model regularization increases. **swimmers** has the second strongest potential to model the response, but it’s importance diminishes near zero as the regularization increases."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "985b44e2",
   "metadata": {},
   "source": [
    "### L1 Regularization – Lasso Regression\n",
    "\n",
    "$$min_\\theta \\; J(\\theta) = min_\\theta \\; \\frac{1}{2n} [ \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2 + \\lambda \\sum_{i=1}^n |\\theta_j| ]$$\n",
    "\n",
    "\n",
    "- This regularization is referred to as least absolute shrinkage or selection operator(Lasso).\n",
    "- L1 norm shrinks the parameters to zero.\n",
    "    - When input features have weights closer to zero that leads to sparse L1 norm. In Sparse solution majority of the input features have zero weights and very few features have non-zero weights.\n",
    "- Not all input features have the same influence on the prediction. L1 norm will assign a zero weight to features with less predictive power.\n",
    "- L1 regularization does feature selection. It does this by assigning insignificant input features with zero weight and useful features with a non-zero weight.\n",
    "- The intersection is at the corner of the diamond.\n",
    "    - Lasso regression gives us sparse solution.\n",
    "\n",
    "<img src=\"images/p128.png\" height=500px width=500px>\n",
    "\n",
    "\n",
    "**Example:**\n",
    "\n",
    "- **Y-axis:** Regularized coefficients for each variable (i.e. coefficients after penalization is applied)\n",
    "- **X-axis:** Logarithm of the penalization parameter Lambda. The higher value of lambda indicates more regularization (i.e. reduction of the coefficient magnitude, or shrinkage)\n",
    "- **Curves:** Change in the predictor coefficients as the penalty term increases.\n",
    "- **Numbers on top:** The number of variables in the regression model. Since Ridge regression doesn’t do feature selection, all the predictors are retained in the final model.\n",
    "- **Red dotted line:** The minimum value of lambda that results in the smallest cross-validation error.\n",
    "- **Blue dotted line:** The largest value of lambda within the 1 standard error of the lambda.min. This lambda.1se value corresponds to a higher level of penalization (i.e. more regularized model) and can be chosen for a simpler model in predictions (less impact from coefficients)\n",
    "- **Log Lambda = 0** corresponds to `no regularization` (i.e. regular linear model with minimum residual sum of squares).\n",
    "\n",
    "**temp and stock_price get eliminated quickly.**\n",
    "\n",
    "<img src=\"images/p130.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaafd93d",
   "metadata": {},
   "source": [
    "#### Differences\n",
    "\n",
    "**L1 Regularization**\n",
    "- L1 penalizes sum of absolute value of weights.\n",
    "- L1 has a sparse solution\n",
    "- L1 has multiple solutions\n",
    "- L1 has built in feature selection\n",
    "- L1 is robust to outliers\n",
    "- L1 generates model that are simple and interpretable but cannot learn complex patterns\n",
    "\n",
    "**L2 Regularization**\n",
    "- L2 regularization penalizes sum of square weights.\n",
    "- L2 has a non sparse solution\n",
    "- L2 has one solution\n",
    "- L2 has no feature selection\n",
    "- L2 is not robust to outliers\n",
    "- L2 gives better prediction when output variable is a function of all input features\n",
    "- L2 regularization is able to learn complex data patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02cfe5f5",
   "metadata": {},
   "source": [
    "### Elastic Net Regression \n",
    "\n",
    "Hybrid version: Both `L1` and `L2` penalties.\n",
    "\n",
    "$$min_\\theta \\; J(\\theta) = min_\\theta \\; \\frac{1}{2n} [ \\sum_{i=1}^n \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right)^2 + \\lambda_1 \\sum_{i=1}^n |\\theta_j| + \\lambda_2 \\sum_{i=1}^n \\theta_j^2 ]$$\n",
    "\n",
    "- Due to the corners included in the solution, regularized solution will have some weights equal to zero.\n",
    "    - Solution is sparse in general, and is therefore biased.\n",
    "- Ridge and Lasso are special cases of elastic net regression.\n",
    "- Combines the strength of both but require tuning of hyper-parameters $\\lambda_1$ and $\\lambda_2$ using validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef6be7bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of lasso regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8e6800f7",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Theta :  [0.98666667]\n",
      "Cost :  [3.21609375 2.76611146 2.38109537 2.05166598 1.76979795 1.52862462\n",
      " 1.32227069 1.14570911 0.99463861 0.86537891 0.75478108 0.66015081\n",
      " 0.57918279 0.50990452 0.45062831 0.3999101  0.35651433 0.31938382\n",
      " 0.28761403 0.26043101 0.23717253 0.21727199 0.2002446  0.18567554\n",
      " 0.17320988 0.16254395 0.15341792 0.14560946 0.13892834 0.13321181\n",
      " 0.12832061 0.12413557 0.12055475 0.11749091 0.11486941 0.11262639\n",
      " 0.1107072  0.1090651  0.10766008 0.1064579  0.10542929 0.10454919\n",
      " 0.10379615 0.10315183 0.10260053 0.10212883 0.10172523 0.1013799\n",
      " 0.10108443 0.10083161 0.1006153  0.10043022 0.10027185 0.10013635\n",
      " 0.10002042 0.09992122 0.09983634 0.09976372 0.09970158 0.09964842\n",
      " 0.09960293 0.09956401 0.0995307  0.09950221 0.09947783 0.09945696\n",
      " 0.09943912 0.09942384 0.09941078 0.0993996  0.09939003 0.09938184\n",
      " 0.09937484 0.09936885 0.09936372 0.09935933 0.09935558 0.09935237\n",
      " 0.09934962 0.09934727 0.09934526 0.09934353 0.09934206 0.0993408\n",
      " 0.09933972 0.0993388  0.09933801 0.09933734 0.09933676 0.09933626\n",
      " 0.09933584 0.09933548 0.09933517 0.0993349  0.09933468 0.09933448\n",
      " 0.09933432 0.09933418 0.09933405 0.09933395 0.09933386 0.09933378\n",
      " 0.09933372 0.09933366 0.09933362 0.09933358 0.09933354 0.09933351\n",
      " 0.09933348 0.09933346 0.09933344 0.09933343 0.09933341 0.0993334\n",
      " 0.09933339 0.09933338 0.09933338 0.09933337 0.09933337 0.09933336\n",
      " 0.09933336 0.09933335 0.09933335 0.09933335 0.09933335 0.09933334\n",
      " 0.09933334 0.09933334 0.09933334 0.09933334 0.09933334 0.09933334\n",
      " 0.09933334 0.09933334 0.09933334 0.09933334 0.09933334 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333 0.09933333\n",
      " 0.09933333 0.09933333 0.09933333 0.09933333]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Define the cost function for Lasso regularization\n",
    "def cost_function(X, y, theta, alpha):\n",
    "    m = len(y)\n",
    "    h = X @ theta\n",
    "    cost = 1/(2*m) * np.sum((h-y)**2) + alpha * np.sum(np.abs(theta))\n",
    "    return cost\n",
    "\n",
    "# Define the gradient descent function\n",
    "def gradient_descent(X, y, theta, alpha, iterations, learning_rate):\n",
    "    m = len(y)\n",
    "    cost_history = np.zeros(iterations)\n",
    "    for i in range(iterations):\n",
    "        h = X @ theta\n",
    "        theta = theta - learning_rate * (1/m) * (X.T @ (h-y)) - learning_rate * alpha * np.sign(theta)\n",
    "        cost_history[i] = cost_function(X, y, theta, alpha)\n",
    "    return theta, cost_history\n",
    "\n",
    "# Set the hyperparameters\n",
    "alpha = 0.1\n",
    "iterations = 1000\n",
    "learning_rate = 0.01\n",
    "\n",
    "# Initialize the model parameters\n",
    "theta = np.zeros(X.shape[1])\n",
    "\n",
    "# Run gradient descent to fit the model\n",
    "theta, cost_history = gradient_descent(X, y, theta, alpha, iterations, learning_rate)\n",
    "\n",
    "print(\"Theta : \", theta)\n",
    "print(\"Cost : \", cost_history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0e15ad9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of ridge regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "6d6d7bce",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Define the cost function for ridge regularization\n",
    "def cost_function(X, y, theta, alpha):\n",
    "    m = len(y)\n",
    "    h = X @ theta\n",
    "    cost = 1/(2*m) * np.sum((h-y)**2) + alpha * np.sum(theta**2)\n",
    "    return cost\n",
    "\n",
    "# Define the gradient descent function\n",
    "def gradient_descent(X, y, theta, alpha, iterations, learning_rate):\n",
    "    m = len(y)\n",
    "    cost_history = np.zeros(iterations)\n",
    "    for i in range(iterations):\n",
    "        h = X @ theta\n",
    "        theta = theta - learning_rate * (1/m) * (X.T @ (h-y)) - learning_rate * alpha * np.sign(theta)\n",
    "        cost_history[i] = cost_function(X, y, theta, alpha)\n",
    "    return theta, cost_history\n",
    "\n",
    "# Set the hyperparameters\n",
    "alpha = 0.1\n",
    "iterations = 1000\n",
    "learning_rate = 0.01\n",
    "\n",
    "# Initialize the model parameters\n",
    "theta = np.zeros(X.shape[1])\n",
    "\n",
    "# Run gradient descent to fit the model\n",
    "theta, cost_history = gradient_descent(X, y, theta, alpha, iterations, learning_rate)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2268ad39",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
    "\n",
    "\n",
    "# Fit a linear regression model\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X_train, y_train)\n",
    "\n",
    "# Fit a Ridge model with alpha=0.1\n",
    "ridge = Ridge(alpha=0.1)\n",
    "ridge.fit(X_train, y_train)\n",
    "\n",
    "# Fit a Lasso model with alpha=0.1\n",
    "lasso = Lasso(alpha=0.1)\n",
    "lasso.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dfd3f39",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30a73a38",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d71c54e1",
   "metadata": {},
   "source": [
    "## **Project:** Medical Insurance Cost Prediction\n",
    "\n",
    "- Basic Understanding of Data\n",
    "- Exploratory Data Analysis\n",
    "- Data Preprocessing and Feature Engineering\n",
    "- Train Test Split\n",
    "- Model Building and Model Evaluation \n",
    "- Hyper-Parameter Tuning\n",
    "- Prediction on new Data\n",
    "- Model Deployment \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39ad9279",
   "metadata": {},
   "source": [
    "**Which Factors Influence the Price/Cost of Health Insurance?**\n",
    "\n",
    "<img src=\"images/p5.jpg\" height=400px width=400px>\n",
    "\n",
    "Many factors that affect how much you pay for health insurance are not within your control. Nonetheless, it's good to have an understanding of what they are. Here are some factors that affect how much health insurance premiums cost\n",
    "\n",
    "- **age:** age of primary beneficiary\n",
    "- **sex:** insurance contractor gender, female, male\n",
    "- **bmi:** Body mass index, providing an understanding of body, weights that are relatively high or low relative to height, objective index of body weight $(\\frac {kg} {m^2})$ using the ratio of height to weight, ideally 18.5 to 24.9\n",
    "- **children:** Number of children covered by health insurance / Number of dependents\n",
    "- **smoker:** Smoking\n",
    "- **region:** The beneficiary's residential area in the US, northeast, southeast, southwest, northwest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "998153c1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "bc65065c",
   "metadata": {},
   "outputs": [],
   "source": [
    "url =\"https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv\"\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "246d180e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>bmi</th>\n",
       "      <th>children</th>\n",
       "      <th>smoker</th>\n",
       "      <th>region</th>\n",
       "      <th>charges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19</td>\n",
       "      <td>female</td>\n",
       "      <td>27.900</td>\n",
       "      <td>0</td>\n",
       "      <td>yes</td>\n",
       "      <td>southwest</td>\n",
       "      <td>16884.92400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>18</td>\n",
       "      <td>male</td>\n",
       "      <td>33.770</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>southeast</td>\n",
       "      <td>1725.55230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28</td>\n",
       "      <td>male</td>\n",
       "      <td>33.000</td>\n",
       "      <td>3</td>\n",
       "      <td>no</td>\n",
       "      <td>southeast</td>\n",
       "      <td>4449.46200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>33</td>\n",
       "      <td>male</td>\n",
       "      <td>22.705</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>northwest</td>\n",
       "      <td>21984.47061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32</td>\n",
       "      <td>male</td>\n",
       "      <td>28.880</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>northwest</td>\n",
       "      <td>3866.85520</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age     sex     bmi  children smoker     region      charges\n",
       "0   19  female  27.900         0    yes  southwest  16884.92400\n",
       "1   18    male  33.770         1     no  southeast   1725.55230\n",
       "2   28    male  33.000         3     no  southeast   4449.46200\n",
       "3   33    male  22.705         0     no  northwest  21984.47061\n",
       "4   32    male  28.880         0     no  northwest   3866.85520"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(url)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1b23125",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8994b19",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0625aea2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2715d89e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b819a1ae",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5426bae",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3b51f74",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a93afd80",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "16887c94",
   "metadata": {},
   "source": [
    "### Interview questions of Linear regression in machine learning\n",
    "Here are some common interview questions on linear regression in machine learning:\n",
    "- What is linear regression and how does it work?\n",
    "- What is the difference between simple linear regression and multiple linear regression?\n",
    "- How do you evaluate the performance of a linear regression model?\n",
    "- What is the ordinary least squares (OLS) method and how is it used in linear regression?\n",
    "- How can you prevent overfitting in linear regression?\n",
    "- Can you give an example of how linear regression can be used in a real-world problem?\n",
    "- How do you decide which variables to include in a linear regression model?\n",
    "- How do you handle collinearity in a linear regression model?\n",
    "- What are some common assumptions made in linear regression?\n",
    "- Can you discuss the limitations of linear regression?\n",
    "- What is gradient descent and how does it work?\n",
    "- How is gradient descent used in linear regression?\n",
    "- What is the purpose of the learning rate in gradient descent?\n",
    "- Can you explain the difference between batch gradient descent and stochastic gradient descent?\n",
    "- How do you choose the initial values of the model parameters in gradient descent?\n",
    "- Can you discuss the convergence of gradient descent and how to monitor it?\n",
    "- How do you implement gradient descent in practice?\n",
    "- Can you discuss the limitations and potential challenges of using gradient descent in linear regression?\n",
    "- Can you give an example of how gradient descent can be used to solve a linear regression problem?\n",
    "- Can you explain the mathematical concept of a straight line and how it is used in linear regression?\n",
    "- How do you represent a linear regression model mathematically?\n",
    "- Can you explain the mathematical concept of an objective function and how it is used in linear regression?\n",
    "- How do you optimize the objective function in linear regression using an iterative optimization algorithm like gradient descent?\n",
    "- Can you explain the mathematical concept of a gradient and how it is used to update the model parameters in gradient descent?\n",
    "- How do you represent the residuals (errors) of a linear regression model mathematically?\n",
    "- Can you discuss the mathematical concept of an orthogonal projection and how it is used in linear regression?\n",
    "- Can you explain the concepts of overfitting and underfitting in linear regression?\n",
    "- How do you detect overfitting or underfitting in a linear regression model?\n",
    "- What are some common causes of overfitting or underfitting in linear regression?\n",
    "- How do you prevent overfitting or underfitting in linear regression?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acbc6ca4",
   "metadata": {},
   "source": [
    "### Applications of Linear Regression:\n",
    "\n",
    "Linear regression is a widely used machine learning technique and has many applications in various industries. Here are a few examples of industry use cases of linear regression:\n",
    "\n",
    "- **Financial analysis:** Linear regression can be used to model and predict stock prices, analyze the relationship between different financial indicators, and identify trends in financial data.\n",
    "- **Healthcare:** Linear regression can be used to predict patient outcomes based on various factors such as age, medical history, and treatment options. It can also be used to analyze the effectiveness of different medical treatments.\n",
    "- **Marketing:** Linear regression can be used to predict customer behavior and identify the factors that influence customer decisions. It can also be used to optimize marketing campaigns and predict the success of a product launch.\n",
    "- **Sales forecasting:** Linear regression can be used to predict sales figures based on various factors such as advertising spend, economic indicators, and competition.\n",
    "- **Energy consumption:** Linear regression can be used to predict energy consumption patterns and identify ways to reduce energy use.\n",
    "- **Agriculture:** Linear regression can be used to predict crop yields and identify the factors that influence crop production.\n",
    "- **Sports analytics:** Linear regression can be used to predict the performance of sports teams or individual players based on various factors such as player statistics, team dynamics, and external factors.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3aba36d3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}