线性回归.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 线性回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性回归是一种最简单的机器学习模型，试图学习一个通过属性的线性组合来进行预测的函数，对于线性组合来说，有两种求解方式\n",
    "\n",
    "1. 最小二乘法\n",
    "2. 梯度下降"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import make_regression\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 构造数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x10f0bd240>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG/lJREFUeJzt3X+MHOddx/HP1+dNckmBSxWTOhubGOQ6ShpiK6cQMKAm\nLXWg0FwTaF0BLaKqQYRfVWVkU4kEQVQLUyJRoGCgalFDU9MkrkVSTFJbVKpwkzN2mjixwW2axNvQ\nXEmuJfIlnM9f/rhZe29vZmdmZ+Zmd+b9kk6+nZ2bfVaJ5jvP93me72PuLgBAvS0ruwEAgPIRDAAA\nBAMAAMEAACCCAQBABAMAgAgGAAARDAAAIhgAACQtL7sBSV1yySV+xRVXlN0MABgqhw4d+ra7r4g7\nb2iCwRVXXKHJycmymwEAQ8XMnk1yHmkiAADBAABAMAAAiGAAABDBAACgIZpNBAB1sudwSzv3Hdc3\np2d02diotm5ap4kNzcI+j2AAAANmz+GWtt//hGZm5yRJrekZbb//CUkqLCCQJgKAAbNz3/GzgaBt\nZnZOO/cdL+wzCQYAMGC+OT2T6ngeCAYAMGAuGxtNdTwPBAMAGDBbN63TaGNkwbHRxoi2blpX2Gcy\ngAwAA6Y9SMxsIgCouYkNzUJv/t1ySROZ2SfM7EUze7Lj2J1m1jKzI8HPz3S8t93MTpjZcTPblEcb\nAAD9y2vM4JOSbg45fre7rw9+HpIkM7tK0mZJVwd/81dmNhLytwCAJZJLMHD3L0l6KeHpt0i6191f\nc/dnJJ2QdH0e7QAA9Kfo2US/ZWZfDdJIFwfHmpKe7zjnZHBsETPbYmaTZjY5NTVVcFMBoL6KDAYf\nl/SDktZLekHSR9NewN13ufu4u4+vWBG7axsAoE+FBQN3/5a7z7n7GUl/q3OpoJakVR2nXh4cAwCU\npLBgYGYrO16+U1J7ptFeSZvN7HwzWyNpraRHi2oHACBeLusMzOwzkt4s6RIzOynpDklvNrP1klzS\nNyT9miS5+1Ez2y3pKUmnJd3u7nNh1wUALA1z97LbkMj4+LhPTk6W3QwAGCpmdsjdx+POozYRAIBg\nAAAgGAAARDAAAIhgAAAQwQAAIPYzAFBxew63Mm8Sk8c1Bh3rDABU1p7DLW2//wnNzJ5b1zraGNFH\nbr0m8c087BqNZabXXbBc06dmBz44JF1nQM8AQGXt3Hd8wU1ckmZm57Rz3/HQm3dYDyDsGrNnXC+f\nmpUktaZntP3+JyRpYANCEowZAKisb07PJD7e7gG0pmfkOneTb0Vco1M7wAwzggGAyrpsbDTx8ahe\nxIhZos+KCjzDgmAAoLK2blqn0cbCXXVHGyPaumndonOjbuZz7ouuESYq8AwLggGAyprY0NRHbr1G\nzbFRmaTm2Gjk4HHUzbz9N+1rjI021BhZ2FuICjDDhNlEAKB0M4+Gaaops4kAIIX2zTzJTX5iQ3Ng\nb/79IhgAQKCKN/mkGDMAABAMAAAEAwCACAYAABEMAAAiGAAAxNRSAANkmBZzVQ3BAMBA6F4BXJXS\n0MMilzSRmX3CzF40syc7jr3ezB42s/8K/r24473tZnbCzI6b2aY82gBgaew53NLGHfu1ZtuD2rhj\nv/YcbuVy3V57D6B4eY0ZfFLSzV3Htkn6oruvlfTF4LXM7CpJmyVdHfzNX5lZfElAAKWLqvmfR0BI\ns/cA8pdLMHD3L0l6qevwLZI+Ffz+KUkTHcfvdffX3P0ZSSckXZ9HOwAUq8in9zR7DyB/RY4ZXOru\nLwS//7ekS4Pfm5IOdpx3Mji2iJltkbRFklavXl1QMwEklffTe+eA8fcFpaFn585VUq5CaehhsSRT\nS32+TnbqWtnuvsvdx919fMWKFQW0DEAaeT69d6ecpmdmJZcuvrARu/cA8ldkz+BbZrbS3V8ws5WS\nXgyOtySt6jjv8uAYgAG3ddO60Jr//Ty9R200f+F5y3X4D96Wua1Ip8iewV5J7wt+f5+kz3cc32xm\n55vZGklrJT1aYDsA5CTNzmFxGDAeLLn0DMzsM5LeLOkSMzsp6Q5JOyTtNrP3S3pW0rskyd2Pmtlu\nSU9JOi3pdnefC70wgIGTV83/y8ZG1Qq58TNgXI5cgoG7vyfirbdEnH+XpLvy+GwAwynPlBOyYwUy\ngFKk2WYSxSMYAChNnbeZHDRULQUAEAwAAAQDAIAIBgAAMYAM1AYbx6AXggFQEb1u9mwcgzikiYAK\n2HO4pa3/9PiCfQa2/tPjZ/cZYOMYxCEYABVw596jmj2zsDDw7BnXnXuPSqIOEOKRJgKGSFQqaHpm\nNvT89nHqACEOwQAYEr3y/nHyrAPEQHQ1EQyAIdEr73/xhQ29fGpx7+DiCxuS8qsDxEB0dREMgCHR\nK+9/97vXa+vnHl+wZWRjxHTHz1199nUedYB6BSSCwXBjABkYEr22nJzY0NTOn792waYzO3/+2txv\n0AxEVxc9A2BIxOX9l6ICKAPR1UXPABgSeW452a+tm9ZptDGy4Bgb0lQDPQNgiOT19N/vjCA2pKku\nggFQM1lnBLEhTTWRJgJqhtIUCEPPABgwRS/qYkYQwhAMgAESlsL54GePaPLZl/THE9fk8hlpZgSl\nDUysTh5epImAARKWwnFJ9xx87mwF0qySzghqB6bOSqjb738ish1pz8dgIRgAAyQqVeNSbjn9pFNU\n044tMBYx3AoPBmb2DTN7wsyOmNlkcOz1Zvawmf1X8O/FRbcDGAa9Fm/lldOPS+XsOdzSxh37Q1NJ\nvdrBWMRwW6qewY3uvt7dx4PX2yR90d3XSvpi8Bqova2b1ski3stjlW9cKqfz/Si9ymIU1W4Ur6w0\n0S2SPhX8/ilJEyW1AxgoExua+sUbVi8KCHmt8o1L5YS9n7QdrE4ebksRDFzSI2Z2yMy2BMcudfcX\ngt//W9KlYX9oZlvMbNLMJqemppagqUD5/njiGt397vWFlJ2IS+X0SunEtWMQymWgf0sxtfTH3b1l\nZt8v6WEzO9b5pru7mXnYH7r7Lkm7JGl8fDz0HKBsRUynLGqVb9S00rFg34Oo95tjo/rytptir8/q\n5OFVeM/A3VvBvy9KekDS9ZK+ZWYrJSn498Wi2wHEaQ+crtn2oDbu2J9oSuSwTafcummdGiOLRyVe\nefW09hxukeqpsUKDgZldZGbf0/5d0tskPSlpr6T3Bae9T9Lni2wHEKffm/ogTqfsFdQmNjR10XmL\nEwKzZ/zsBjWkeuqp6DTRpZIeMLP2Z/2ju/+LmT0mabeZvV/Ss5LeVXA7gEh7Drf0od2Pa84XZiKT\n7ODVz3TKIlfpJilC952ZxdtjdraZVE89FRoM3P3rkq4NOf4/kt5S5GcDSbRvnt2BoC1ujnzazV6K\n3kM4ybaUbFCDMKxARq3FTaWMu0GmzbEXnVZK0lNhXABhKFSHWuv15D/aGNGNV67Qxh37I1M6aTd7\nifq81vSM9hxuZe4dJHnqZ4MahCEYoNaibp4jZrrtuqbuO9SKTenE5dg7xwiWmUWmpPJIF8Xtk5y0\nzagf0kSotaiUyXt+ZJU+85XnM6d0umcpRQWCfq4dhtlA6Bc9A9RaZ8qkNT2jETPNzM7pnoPPKeq2\nnabwWtyYRJZrR+GpH/2gZ4Dam9jQPNtDaD+591runmbWTdqbOzN6UBZ6BoCSP8GnnXUTNSZx8YUN\nvTp7Jja334ldxFAkegaAkj3Bj5ilzr+HjUmYpLf/8Mqzuf32tdtjBmGrnoet7AWGD8EA0LlCbVFG\nGyP66LuuTf0kPrGhqduuay4oSe2S7js0fxPvTk9F3eQHsewFqoU0EWpvz+GWXnn1dOT7zYwpmQPH\nphaNQXTeyONWDEvsIobiEQxQezv3HdfsmcVDxmOjDR25422JrtErn9/Pjbz7PUpIoGikiVB7UTfl\nqIJu3eLy+b22g0y6VSQlJFA0ggFqL+vevXH5/F438qQ3eRaToWikiVBJaaZhJi3hECUuDZSkFlCS\ntrKYDEUiGKByepWJlqJvvP3O4U9aHK7X3sHc5FE2ggEqJyptc+feo3rt9JnIwnP93pCz9iyAQcCY\nASonKm0zPTNbyFx98vmoAnoGqJyotE0UisMBBAMMgKSDvUnO23O4pVP/t3gB2WhjRBc0lunlU4un\ni3bPGqIGEOqIYIBSJd0TOMl53ee0jY02dOc7rpakRe9bcK2NO/afzfEXuUdxd6C58coVOnBsisCD\n0hEMUKokG7gnPS+q8uhF5y9fdK3W9IxM50pVt2/6FzSWJWpPP8IC2qcPPnf2/bwDD5AGA8goVdJS\nDb32Dt64Y7/WbHswcpyg828nNjT15W03qTk2GlovKCyN1Ovz00hSJpvicygLwQClSrr6N+q8dpon\nzWY0ew63Ug0wS/M9iI079mcqGZ00oFB8DmUgGKBUScsxRO0L0CsIhF2rnarpR9Y9BJKWt6D4HMpQ\nWjAws5vN7LiZnTCzbWW1A+VKOkc/7LxegSDqWmn3JO6WJY0TFtC6sVgNZSllANnMRiT9paSfknRS\n0mNmttfdnyqjPShX3Bz97hk4d797vSY2NLVxx/7QdE9zbFRf3nZT6LXySMH0e42wshfMJsKgMPe4\njnYBH2r2o5LudPdNwevtkuTuH4n6m/HxcZ+cnFyiFmJQhE0XbSwzve6C5Xr51OyiVFH7dTPiRtue\nSdRtbLSxoFRF57W6hQUb1iZgUJnZIXcfjzuvrDRRU9LzHa9PBscWMLMtZjZpZpNTU1NL1jgMjrC0\nzuwZPzvrx6WzW0p2TxX99MHnFu0xcOOVK0LHKO58x9WL0lA/9kOvD23TjVeuWPCa/YlRBQO9zsDd\nd0naJc33DEpuDkqQJCXjmt9Qfi6mlzszO6cDx6b0kVuviXyK73ya37hjf+h1Dhxb+GCSdK0EMMjK\nCgYtSas6Xl8eHAMWSFpnKC4QtH1zeiZxHaGsayCYIophUlaa6DFJa81sjZmdJ2mzpL0ltQUDLMkM\nHGm+Z5BEmmmbWddAMEUUw6SUYODupyX9pqR9kp6WtNvdj5bRFgy27imlY6MNNUYW3vhHGyN6z4+s\nUmNZ74CQdtpmljUQTBHFsCltzMDdH5L0UFmfj2QGYZZMd1onrE2S9NnHno+6hCSl3mMg6Q5oWXdK\nAwZBKVNL+8HU0qUXNq1ztDEykBu3RK05aOu19gCosqRTSwd6NhHK1c8smbQ9iTTn9zq312AtKRsg\nHsEAkdLOkkm6N0E/58ed22vW0W3XsQsZEIdCdYiUdpZMr55E1vPjzt26aZ2iho+71wVE2XO4dbYc\ndtYKpcCwIRjURD83urSzZKKezJPsMxB3PO7ciQ3NyMJ1Seb7s4oYdUcwqIF+b3RJK4q2Rc31jzqe\npueR5Nxmhvn+aXs1QNUwZlADUTe6D+1+XB/87JGeA7dJV+tK0auAo45v3bQudLbSjVeu0MYd+xcM\nFEed29lLSXJOFFYRo+7oGdRA1A1tzj2XlEg7BRUl6ok9rOdx23VN3XeotagXIym2l5K2J9OJVcSo\nO9YZ1EDcHPy2fubih61F6JR2XUI/exTkYZjWVABpDHoJayyhpPV9+kmJ9No5LM2TeVwbik7XZOlV\nAFXAmEENdJdLWBZR7rmflEjUTdqkvp7ko9YLRLWteyFalp3D0oyPAFVDMKiJzhtdVEok6Srdzhtw\nnoFFSjcIHLYQ7dMHnzv7ftyiNwDnEAwqKK7EQ5bCat034LBAkKX8Q5q2Jdncnk1mgGQIBhWTtMRD\nvymRqBvwiJnOuOdSsTPr5jP9ngfUGcGgYoregjHqxnrGXc/seHvm66eRdBc0pocC8ZhNVDFFz8bJ\ncz5+1lpASWZJUbEUSIZgUDFFL57Ka1evPGoBhU0H/aUbVjM9FOgDi84qJmymUGPEdNF5y/Wdmdlc\ncvp57H5W1uIyoG7Y3KamumfjjF3Y0Cuvntb0zKyk/qZbht38s96wqQUEDBbSRBU0saGpL2+7Sc/s\neLsuPG+5Zs8s7P2lqcZZVGlnagEBg4WeQcUleQLvlfZJOjspbeqo1+KyPNJQANIhGFRcXHmHuHUJ\nSYNJmu0uO4933/Qnn31J9xx87uxGNawiBpYGaaKKi5v9E7epS5J0Tr8bw3Sms9pjEJ2BIM21AGRD\nMKi4uGqccU/+SaaS5jUYvHPf8UxbVwLoX2FpIjO7U9IHJLV3I/99d38oeG+7pPdLmpP02+6+r6h2\noHd5h7g0UpJaQWkrjUbpdcNnYBkoVtFjBne7+592HjCzqyRtlnS1pMskPWJmb3T33hXHUIgkVULj\nagVl2W6yU1RQseAzABSnjDTRLZLudffX3P0ZSSckXV9COyorTZmHPDZ1yWtjmLCUlEn6xRtWM3gM\nFKzonsFvmdl7JU1K+pC7vyypKelgxzkng2OLmNkWSVskafXq1QU3tRr6ndmT9Wab1zWk/kprA8gm\nUzkKM3tE0htC3vqw5m/435bkkv5I0kp3/1Uz+wtJB93908E1/l7SF9z9c70+i3IUyVDmAUCnJSlH\n4e5vTdiYv5X0z8HLlqRVHW9fHhxDDpLO7GFhF4BOhY0ZmNnKjpfvlPRk8PteSZvN7HwzWyNpraRH\ni2pH3SRZF1BUiQkAw6vIMYM/MbP1mk8TfUPSr0mSux81s92SnpJ0WtLtSz2TqEpPxWEbwt93qLVo\nZs+NV67Qxh37I/ctZntIoN4KCwbu/ss93rtL0l1FfXYv/QywLqU0gSrsu9x3qKXbrmvqwLGpyAAR\ntm+xxMIuoM5qV5uo6G0hs0gbqKK+y4FjUwsGizfu2B+7cbwUnWKqUk8KQLjalaMY5Dr6aWv8JP0u\nSb5b1CIxxheAeqhdzyCv0glFiLu5dz+hj13Y0MunZhed3/1dor7ziJnOuPd82h/knhSA/NSuZ5DX\nHr5F6DUTKOwJ/ZVXT6sxYgvODfsuUd/5o++69mzF0Kgb+yD3pADkp3bBIK/SCWlKPiTVK1CFPaHP\nnnFddN7y2O+S5TuzIxlQD7VLE0nZSycUNSOpVzmGD372SOjfTM/M6qLz4/8z9vud8ypCB2Cw1TIY\nZFVkHj3qpt2romf7eF6b3Xf+LfWCgHogGPShjDx62BO6SZG7giW5WSft4eRRhA7AYKvdmEEeysij\nh+X9s+4K1u92lQCqh55BH8rKo3c/oUdVKE0alNJOZSU9BFQXwaAPg5JHTxuU0qxTGPSyHQDyRTDo\n0yDk0dMEpbCbe2OZqTFimp07l3DqNZWVxWZAdREM+jBI6ZOkQSlqncLYaEMXnb888VRWFpsB1UQw\nSGkY0idhwSrqJv6dmVkdueNti44PctkOAPljNlFKec3AKWIFc/u6YYXlvm+0EXp+1M19kMt2AMgf\nPYOU8lhjUGTvIipYXdBYptHGSOLB5kEZJAewNAgGKeWRPilycDYqKE2fmtXd716f6uY+CIPkAJYG\nwSClPNYYFLmCuVew4uYOIApjBinlUfW0yBXMSXP9RY1ZABhO9Az6kPUJu8gVzEly/cMwIwrA0iIY\nlKDowdm4YMWCMgDdCAYlKTN/z+5lALoxZlBD7F4GoBvBoIZYUAagW6ZgYGa/YGZHzeyMmY13vbfd\nzE6Y2XEz29Rx/DozeyJ478/NzBZfGUXKax9oANWRdczgSUm3SvqbzoNmdpWkzZKulnSZpEfM7I3u\nPifp45I+IOkrkh6SdLOkL2RsR2UsVRE81hwA6JSpZ+DuT7t7WFGeWyTd6+6vufszkk5Iut7MVkr6\nXnc/6O4u6R8kTWRpQ5VE1RViDQCAohU1ZtCU9HzH65PBsWbwe/fxUGa2xcwmzWxyamqqkIYOErah\nBFCW2DSRmT0i6Q0hb33Y3T+ff5POcfddknZJ0vj4eNSWv7kqc68CpnwCKEtsMHD3t/Zx3ZakVR2v\nLw+OtYLfu48PhLJX5rKHAICyFJUm2itps5mdb2ZrJK2V9Ki7vyDpu2Z2QzCL6L2SCu1dpFF2moYp\nnwDKkmk2kZm9U9LHJK2Q9KCZHXH3Te5+1Mx2S3pK0mlJtwcziSTpNyR9UtKo5mcRDcxMorLTNOwh\nAKAsmYKBuz8g6YGI9+6SdFfI8UlJb8ryuUUZhDQNUz4BlIEVyB1I0wCoKwrVdSBNA6CuCAZdSNMA\nqKPaBIMy1w8AwKCrRTAoe/0AAAy6Wgwgl71+AAAGXS2CQdnrBwBg0NUiGLCzFwD0VotgwPoBAOit\nFgPIrB8AgN5qEQwk1g8AQC+1SBMBAHojGAAACAYAAIIBAEAEAwCAKj6biOJ0AJBMZYMBxekAILnK\npokoTgcAyVU2GFCcDgCSq2wwoDgdACRX2WBAcToASK6yA8gUpwOA5CobDCSK0wFAUpnSRGb2C2Z2\n1MzOmNl4x/ErzGzGzI4EP3/d8d51ZvaEmZ0wsz83M8vSBgBAdlnHDJ6UdKukL4W89zV3Xx/8/HrH\n8Y9L+oCktcHPzRnbAADIKFMwcPen3T3xxH0zWynpe939oLu7pH+QNJGlDQCA7IqcTbQmSBH9m5n9\nRHCsKelkxzkng2OhzGyLmU2a2eTU1FSBTQWAeosdQDazRyS9IeStD7v75yP+7AVJq939f8zsOkl7\nzOzqtI1z912SdknS+Pi4p/17AEAyscHA3d+a9qLu/pqk14LfD5nZ1yS9UVJL0uUdp14eHIt16NCh\nb5vZs2nb0odLJH17CT5nENTpu0r1+r581+pK+31/IMlJhUwtNbMVkl5y9zkz+0HNDxR/3d1fMrPv\nmtkNkr4i6b2SPpbkmu6+ooi2djOzSXcfjz9z+NXpu0r1+r581+oq6vtmnVr6TjM7KelHJT1oZvuC\nt35S0lfN7Iikz0n6dXd/KXjvNyT9naQTkr4m6QtZ2gAAyC5Tz8DdH5D0QMjx+yTdF/E3k5LelOVz\nAQD5qmxtogx2ld2AJVSn7yrV6/vyXaurkO9r89P9AQB1Rs8AAEAwCGNmO83smJl91cweMLOxsttU\nlKj6UlViZjeb2fGgHta2sttTJDP7hJm9aGZPlt2WopnZKjM7YGZPBf8P/07ZbSqKmV1gZo+a2ePB\nd/3DvD+DYBDuYUlvcvcflvSfkraX3J4i9aovNfTMbETSX0r6aUlXSXqPmV1VbqsK9UnVp97XaUkf\ncverJN0g6fYK/7d9TdJN7n6tpPWSbg6m6OeGYBDC3f/V3U8HLw9q4UK5SklbX2oIXS/phLt/3d3/\nT9K9km4puU2FcfcvSXop9sQKcPcX3P0/gt//V9LT6lHeZpj5vFeCl43gJ9cBX4JBvF8VayGGWVPS\n8x2ve9bDwnAysyskbdD8YtZKMrORYO3Wi5Iedvdcv2ulN7fpJUnNJTP7sOa7ovcsZdvy1md9KWAo\nmNnrNL+u6Xfd/btlt6co7j4naX0whvmAmb3J3XMbG6ptMIiruWRmvyLpZyW9xYd8/m0/9aUqpCVp\nVcfrxPWwMPjMrKH5QHCPu99fdnuWgrtPm9kBzY8N5RYMSBOFMLObJf2epHe4+6my24NMHpO01szW\nmNl5kjZL2ltym5CDYJfEv5f0tLv/WdntKZKZrWjPajSzUUk/JelYnp9BMAj3F5K+R9LD3dt2Vk2P\n+lKVEEwE+E1J+zQ/wLjb3Y+W26rimNlnJP27pHVmdtLM3l92mwq0UdIvS7qpY4vdnym7UQVZKemA\nmX1V8w84D7v7P+f5AaxABgDQMwAAEAwAACIYAABEMAAAiGAAABDBAAAgggEAQAQDAICk/wetR8jz\nwcRNiAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10eef77f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X,y = make_regression(n_samples=100,noise=20,n_features=1)\n",
    "plt.scatter(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用最小二乘法来求解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 最小二乘法是一种求解释一种基于均方误差来求解的一种方法，在线性回归中，最小二乘法就是试图找到一条直线，使所有的样本到直线上的欧式距离最小\n",
    "\n",
    "$Ax = b$\n",
    "\n",
    "$A^TAx = A^Tb$\n",
    "\n",
    "则\n",
    "\n",
    "$x = (A^TA)^{-1}A^Tb$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 在这里，x就是我们所要求得的参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "ones = np.ones(X.shape[0])\n",
    "A = np.insert(X, 0, 1, axis=1)  # 特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = y.reshape(-1,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用最小二乘法求解\n",
    "\n",
    "$Ax=b$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linalg.inv(A.T @ A) @ A.T @ b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 那么X为"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.86962816],\n",
       "       [ 52.04612077]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 也就是我们需要求解的参数，我们将该方程画图出来"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1104cafd0>]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUnFW55/HvQ6cJDQhBCBg6CR3XwgDhFuhBmHBcAkrw\nGEiDCEFHmcMlXpA5cjCEHB3AESeJAVHRIwYvyMggKKYJUYxAmCNGAyYkXALEEyBgOkiAJIAnndCX\nZ/6oqk5d3rfqrcvbdft91mLRteutql3Kep/aez/72ebuiIhIc9ut2h0QEZHqUzAQEREFAxERUTAQ\nEREUDEREBAUDERFBwUBERFAwEBERFAxERAQYUe0ORHXAAQd4R0dHtbshIlJXVq1a9bq7jy50Xd0E\ng46ODlauXFntboiI1BUzeynKdZomEhERBQMREVEwEBERFAxERAQFAxERoY6yiUREmkn36h4WLF3H\npm29HDyqjVlTJ9I1uT22z1MwEBGpMd2re5jzq6fo7RsAoGdbL3N+9RRAbAFB00QiIjVmwdJ1Q4Eg\npbdvgAVL18X2mQoGIiI1ZtO23qLaK0HBQESkxhw8qq2o9kpQMBARqTGzpk6krbUlo62ttYVZUyfG\n9plaQBYRqTGpRWJlE4mINLmuifvR9eAq+JcL4cADY/+8ikwTmdmPzWyzmT2d1nadmfWY2ZrkP/+Y\n9twcM1tvZuvMbGol+iAi0jAWLIC99oKrroKHHhqWj6zUmsFtwBkB7Te5+7HJf34DYGZHADOAScnX\n/JuZtQS8VkSkuQwMwO23J4IAwOc+BzNmDMtHV2SayN1/b2YdES+fDvzc3XcCL5rZeuAE4E+V6IuI\nSN3ZuBEuuQTefBN+97vEvz/zGdh992HrQtzZRJeb2ZPJaaT9km3twF/TrtmYbMthZjPNbKWZrXzt\ntddi7qqIyDDr74eTT4Zx42DpUpg4Eczg8suHNRBAvMHg+8B7gWOBV4Abi30Dd1/o7p3u3jl6dMFT\n20RE6sddd0FrKyxfnnh8yy1w222w995V6U5s2UTu/mrqbzO7FViSfNgDjEu7dGyyTUSk8fX3wx/+\nsGstYNo0uPde2K26275i+3QzG5P28GwglWm0GJhhZiPNbAJwKPBYXP0QEakJW7fCWWfBEUfAccfB\nzTfD3/8O991X9UAAFRoZmNmdwAeBA8xsI3At8EEzOxZwYAPwGQB3X2tmdwPPAP3AZe4+EPS+IiJ1\nzx0uuCAxLQQwdSrs2AFf+EJ1+5XF3L3afYiks7PTV65cWe1uiIhE98gj8IEP7Hp87bVw3XXD2gUz\nW+XunYWu0w5kEZFK6+uDl16CD34w8fiYY+DRR2HkyKp2K5/qT1SJiDSK3l4491yYMAH22ANuvRW2\nbIE1a2o6EICCgYhIZXzpS7DnnnDPPYlSEu+8AxddBPvtV/i1NUDTRCIi5Vi3Dg47bNfjSy6BhQsT\nm8fqiIKBiEgp3nkHtm+HzuTa7P77wwsvwD77VLdfJdI0kYhIMfr6Er/6R46EH/wAfvQj2LQJXn+9\nbgMBKBiISIPrXt3DlHnLmHD1r5kybxndq4sveJB6j559DsysGXT00XDeeTBmTPiL64SCgYg0rO7V\nPcz51VP0bOvFgZ5tvcz51VNFBYTu1T2s/vJ8ls85jfa3dxXM7LzmN0z498GSA0yt0ZqBiDSsBUvX\n0duXWeCgt2+ABUvXBR4h2b26J+OoyWuO3puuj76frrRrPvJP3+HZA98LOweBXQEGiPVYyrgpGIhI\nw9q0rTdye2oU0ds3AO4sn3NaxvP/69RL+fF/mR74fvkCTL1QMBCRhnXwqDZ6Am78B49qy2lLjSIe\nueVixr35asZzHbOX5FyfLSzw1AutGYhIw5o1dSJtrZmn6ra1tjBr6sScazv/eD8b5k/LCARHffEu\nOmYvyXmPIEEBpp5oZCAiDSs1bZO+DjBr6sTM6Zy//Q3GjOHbaa/752lXcu+kUwBoT74m9R77trXy\nn+/00zewq8hnWICpJ6paKiLNK2uX8Mpxkzj3E/OHHre1tjD3nKNy1gKyF5pzAkwNUdVSEZEwU6bA\nH/+Y2TY4yMY1m2iPcJPvmtxeszf/UikYiEjzWLECTjops+2pp+DII4HGvMlHpQVkEWl8O3YkpoTS\nA8EVVyROIUsGgmankYGINLag6qF1slY6nDQyEJHGdOWVuYFgxw4FghAaGYhIY3n5ZTjkkMy2pUvh\n9NOr0586oWAgIo3BHXbLmuw48UT405+q0586o2AgIvVP6wJl05qBiNSMos8euP323ECwebMCQQk0\nMhCRmpBRNZQCpaHfegv23TezbeFCuPTS4ehqQ6rIyMDMfmxmm83s6bS2d5vZA2b2H8l/75f23Bwz\nW29m68xsaiX6ICLDoxInhwXJd/ZABrPcQOCuQFCmSk0T3QackdV2NfCQux8KPJR8jJkdAcwAJiVf\n829mVrgkoIhUXSVODgtT8OyBD3wgd0poYEBTQhVSkWDg7r8HtmQ1Twd+mvz7pzB0WNB04OfuvtPd\nXwTWAydUoh8iEq/Iv95LEFYC+oxt6xNB4JFHdjU+8URw9pCULM41g4Pc/ZXk338DDkr+3Q6sSLtu\nY7Ith5nNBGYCjB8/PqZuikhUxZwcFkV69c9921ppbbGh0tAjBvpZf0NX5guuvBJuuKGkz5L8hmUB\n2d3dzIoey7n7QmAhJEpYV7xjIlKUYk4OKyR7wXhbbx+tuxn77dnK6msDlhI1HRSrOMdYr5rZGIDk\nvzcn23uAcWnXjU22iUiNK+bksEKCppyuX/Kt3ECwfbsCwTCIMxgsBi5M/n0hcG9a+wwzG2lmE4BD\ngcdi7IeIVEjX5HbmnnMU7aPaMBKngAUd/hJF+tTSpFefZ8P8aZz/1AO7Lli0KBEE2ur7OMl6UZFp\nIjO7E/ggcICZbQSuBeYBd5vZxcBLwHkA7r7WzO4GngH6gcvcfSDwjUWk5lSq5v/Bo9ro2bqdDd84\nM6O9591jaH9jU9nvL8XRsZciUh0BJSQO/8r9JY80JFjUYy+VlyUiw2v27JxAcNLnbmPK3IcUCKpI\n5ShEZHhs2gTtWTf6adPgvvtQXdHqUzAQkfipqmjN0zSRiMRnxIjcQNDXp0BQgxQMRKTyfvnLRBAY\nSEsUXLYsEQRGaEKiFun/FRGpnB07gvcFaCRQ8xQMRJpEeh2gg0e1MWvqxMpm7mhdoK4pGIg0iHw3\n+6IOjinWqFHw5puZbZs3w+jR5b2vDCutGYg0gO7VPcz6xRMZ5wzM+sUTQ+cMxFJ6euXKxGggPRDM\nn58YDSgQ1B2NDEQawHWL19I3mDkl0zfoXLd4LV2T2ytbejrsHAFNCdU1BQOROhI2FbStty/w+lR7\nxUpPa12gYWmaSKROlHPkZNmlp9///pxAMOOCuUyZ+1DFzkCW6tLIQKRO5Jv332/PVrZuzx0d7Ldn\nK7BrkbjobKING2DChJzmjtlLEn9UciFaqkrBQKRO5Jv3v+n8Y5n1yyeGjowEaG0xrj1z0tDjoktP\nB0wJTZn7UM50UyogKRjUN00TidSJsPn9g0e10TW5nQXnHpNx6MyCc48p7QZtlhsIenvBveJnIEvt\n0MhApE7MmjoxY68AZM77l33ozMyZcOutmW1XXAHf/ObQw0qegSy1RcFApE6UPO9fSG8v7LlnbntA\nllChgCT1S8FApI5U6sjJVIrq8jmn5T6ZJ1U0toAkVadgINJkulf30HXcWLqy2h/o/j0fnv4PBV9f\nqYAktUULyCLN5Fvfouu4sRlNvSNG0jF7Cdc9G7xxTZqDRgYiNSa26qIBqaJD+wVQRlCzUzAQqSFB\n1UWvuGsNK1/awvVdR5X2pkFB4Kr7ctqDMoKKDUyxl8mW2GiaSKSGBO0yduCOFS8XX/YhaL/ATTfR\n/fhG2nbP/B0YlBFUbPmLcsplSPUpGIjUkLCpGofo5aYffzy8oNwXv0jX5HbmnnNUxga1uecclfML\nvtiy17GUyZZhE/s0kZltAN4GBoB+d+80s3cDdwEdwAbgPHffGndfRGpd2KYuiDinH6GqaKGpnNTz\nxfZDu5Pr23CNDE5x92PdvTP5+GrgIXc/FHgo+Vik6c2aOpGA2zlQYJdv0JTQ668HBoJ8Uznpz4fJ\nVxaj6H5LzajWNNF04KfJv38KOSnPIk2pa3I7nzxxfE5ACN3le9BBuUFgxIhEENh//5zLC03lBD0f\nqR9UoEy2VNVwBAMHHjSzVWY2M9l2kLu/kvz7b8BBQS80s5lmttLMVr722mvD0FWR6ru+6yhuOv/Y\n/HP6vb2JILB5c+aL3aEvfL9AoamcfFM6YWsLKVHXIqQ2DUdq6cnu3mNmBwIPmNlz6U+6u5tZ4P53\nd18ILATo7OzUcUpSk+JIp8y7y7eM08bC1iRGJc89CHu+fVQby68+teD7a3dy/Yp9ZODuPcl/bwYW\nAScAr5rZGIDkvzeHv4PI8Ohe3cOUecuYcPWvmTJvWaSUyGFNpwxaF1i8uKhjJ2dNnUhrS24w+fuO\nfrpX92iqp4nFGgzMbC8ze1fqb+B04GlgMXBh8rILgXvj7IdIIaXe1IclnfK668JHA2eemdOcL6h1\nTW5nr91zJwT6Bn3ogBpN9TSnuKeJDgIWWeI/5BHA/3X335rZn4G7zexi4CXgvJj7IRKqe3UPV979\nBANZv7CjnOBVSjplUdNKRU4JBe1gzj6W8s3e4DWFVJ811dOcYg0G7v4CcExA+xtAQO1ckeGVunlm\nB4KUQjnyxR72EuVmDQQHgcHB4PY0+UYqqffXATUSRDuQpakVSqUsdIMsdo694LRS0LrAuecmRgMF\nAgFEG6loXUCCqFCdNLV8v/zbWls45bDRTJm3LHRKp9jDXsI+711/eQasuINmgkT51a8DaiSIgoE0\ntbCbZ4sZHzu+nXtW9RSc0ik0x56+RrCbWc6U1Ib503JfVGQQSIl6LKXWBSSbpomkqYVNmVzw/nHc\n+ehfy84Uys5SSg8EG+ZPywkEZ/7rL0oOBKCNX1I6jQykqaVPmfRs66XFjN6+Ae5Y8TJht+RiCq8F\nrRH89keXcdjrL+Vc2zF7CRa+fBGZfvVLKRQMpOmlbpzp0yv5fpsXk3WTHjh27+/jLzeenXNN+mlj\nyuiRalEwEKFwVlFKsVk3qTWJoHWBw79yf8G5/XQ6RUzipGAgQrSpnxazouffl8/JzRD64rQr2fvi\nC5l7yLtzpqdS6xHZnxF5f4JIibSALMKuQm1h2lpbuPG8Y6LfeG++OfQA+u5Jp3DPqkSJiNQCdmph\nOawMhk4Rk7hpZCBNr3t1D3/f0R/6fHuxUzIhQSBd+o280I5h0CliEj8FA2l6C5auo28wd8l4VFsr\na649PdJ7dK/uoeu4sblP9Pcz4cu/DXxNvht59nMqISFx0zSRNL2wm3JYQbccZjmBYPXYw+l+fCO0\ntOQ9DjLqUZEqISFxUzCQplfy2b1PPx06JXT2JxcMTQPlu5FHvclrM5nETdNE0pCKScOMWsIhQ4R1\ngfSS0JC/FlCUvmozmcRJwUAaTr40TAi/8UYKHgFB4IIrfsKfdh+d055dHC7f2cG6yUu1KRhIwwlL\nw7xu8Vp29g+G5urnvSEfdFDu4fMA7py/uoc1xY4sRGqM1gyk4YQtCG/r7Ss+V7+/PzEayA4E7kMF\n5TSfL41AIwNpOGFpmGFCUzyLOHJSUz1S7xQMpOqiLvZGua57dQ/b38ndQNbW2sIerbuxdXtuumhO\n1lBQEJg/H666qrgvJlJHFAykqqLW3IlyXfY1KaPaWrnurEkAOc9b8r2mzFvG97f8gaMXXJvTx8O/\ncj9zP3wUXRX6vukB7ZTDRvPwc6+p+JxUnYKBVFWUA9yjXhdWeXSvkSNy3qtnWy/GrlLVQQXlhlJF\nA/pTiqCA9rMVLw89r+JzUk0KBlJVUWvuhF2X+lW/KXmSWKH3Ss3tT5m3LLS09KFfWkRfS2bhukrU\nAIpSJjsoEIoMB2UTSVVF3f0bdl1qmqeYw2i6V/ewfM5pgYGgY/aSnEAAiRHElHnLcqqJFiNqQFHx\nOakGBQOpqqjlGIKuS5/mCZP9Xr9bvDywoFzH7CU5O4izhZWXjipqUTkVn5NqqFowMLMzzGydma03\ns6ur1Q+prqg5+kHX5QsEge9lxunTT864LkoQSFfOGQJBAS2bNqtJtVRlzcDMWoDvAR8GNgJ/NrPF\n7v5MNfoj1VUoRz87A+em84/NmPfP1j6qjeVXn7qrISBVtOtTN7Lm4NJuuqVO4wSVvVA2kdQK85BN\nNLF+qNlJwHXuPjX5eA6Au88Ne01nZ6evXLlymHootSIoXbR1N2PvPUawdXtfzlRR6nH7qDYWffcS\nDux5Mec9g0YCo9paM0pVpL9Xtpxgg84nltplZqvcvbPQddXKJmoH/pr2eCPw/uyLzGwmMBNg/Pjx\nw9MzqSlBGTh9gz60eczZddMe+rcPBqaKHv6V+/nY8e20rerJqSOU2oeQfkPv2L+N5c9vyXmfUw7L\nLEqn84mlEdR0aqm7LwQWQmJkUOXuSBVEmZJxEofVD7iHZggB0DfAw8+9xtxzjgr9FZ9+854yb1ng\n5z383GsZj6PulRCpZdUKBj3AuLTHY5NtIhmi1hl6ft5Hc9p+2Dmd60+7NKNt07beyHWEyt0DoRRR\nqSfVyib6M3ComU0ws92BGcDiKvVFalihDJyznvl/oaOB7EAAxaVtlrsHQimiUk+qMjJw934z+wKw\nFGgBfuzua6vRF6lt2Rk4+7a18p/v9NM3UGBKKECxaZtRT0Ar6aQ0kRpTlWyiUiibqDpqMksmIFV0\n8R//gyuX/IW+gfD/nr+VTEktRiUrqopUQ9RsIgUDCRWU1tnW2lK9g1uCSksDuIfuOUgJSgcVaQZR\ng4HKUUiofFkyYbpX9zBl3jImXP3rSLV8Il3f0xMYCLof3zh02Ey+xVpN2YgUVtOppVJdxWbJFJtv\nH+n6gCCQWhdoS7s2X9bRx47XKWQihWhkIKGKzZIpdiSR93qznEBw3ifmZSwQp7/3rKkTCZlEytkX\nEKbYUY1II1EwaBKl3OiiVhRNCftlHtYeNMK4ccmNgbuHJ8xewmPjjgx9j67J7ZHOMwiTGqWkymGX\nW6FUpN4oGDSBUm90USuKprSELPCGtWeMMJK7hz+29uHMi9zBPdIopb2MfP9S1kdEGonWDJpA2I3u\nyruf4Iq71uRNhYy6WxdgICQzLaw9lZ//7PUfyXluytyHEmma85Yxa+rESLn85eT7axexNDuNDJpA\n2A1twL0iUyKpKagwYb/Yu44bmxMI/vjRT3D4V+7PGcUABUcpxY5k0mkXsTQ7jQyaQJT6PqUWVgva\ni5Au8Jf5smVwWu66AO7MmreM3qy+pvq2/OpTC/avmJFMOu0ilmankUETiHLCFpQ2JZLvkPfAX+Zm\nuYEguS6Qrw9xT9eUM6oQaQQaGTSB7Po+uyXLPWcrZUok7CZtUPC0Md58E/bZJ6cPQaOYsL5ll4Eo\n5+SwUkcVIo1AwaBJpN/owspMRJ0SSb8BFwwseUpIBClmuiZo09rPVrw89LwOmRGJTsGgARUqmhZ0\nFm/UX9DZN+CgQNDW2sKc/zomOBAUqIVVTN/yTVGl6JAZkWgUDBpM1JIQpU6JhN2AW8wYTO4HWD7n\nNLg+64IiCiKWe/hMqdeJNDMtIDeYuDdPhd1YB915cf603N3DS5cWFQiKEXWNQ+mhIoVpZNBg4s7G\nCVrg/eaSGzkne+cwFAwC5Z4BELS+kE3poSLRaGTQYOLePJWdprph/rTcQJCWKhqmErWAgtJB/9uJ\n45UeKlICHW7TYIIyhVpbjL12H8GbvX0VOYWre3UPXceNzX2iiP+Wwg6j0SE0IpWlw22aVPav5f32\nbAWHbb19Jf8CT694illuIJg1q+h1AdUCEqktWjNoQOnZOFPmLWPr9r6M54tJt0yNNI55fjXL7/zX\n3AtKHFkWu7lMROKlYNDgovwCz7eQu2DputCqounTOcUuBufbXKbD5UWGn4JBgyv0CzzvvoTjxrI8\n63WTL7+DrXvui2UFk2KOu0xvz77pr3xpC3eseHnooBrtIhYZHlozaHCFTisL2pewYsG5OesCaw98\nLx2zl7B1z32BzOmcUvc2dE1uZ/nVp/LivI8OjTLSA0Ex7yUi5dHIoMEVKu+QPl205zu9PHPTx3Pe\n4/Cv3J+3VlClFoMXLF1X1tGVIlK62IKBmV0HXAqkTiP/V3f/TfK5OcDFwADwP9x9aVz9kPzlHVLT\nSBvmT8t9Mrk4PLfAHH6lFoPz3fC1sCwSr7hHBje5+w3pDWZ2BDADmAQcDDxoZu9z9/wVxyQWQYfP\nX/SJ/81ZX/o0XcnHhWoFVepgmLCgYsnPEJH4VGPNYDrwc3ff6e4vAuuBE6rQj4aVvi9gyrxlwXsK\nvv3twKqiU+Y+lAgERSzWVupgmKD1DQM+eeJ4LR6LxCzukcHlZvZpYCVwpbtvBdqBFWnXbEy25TCz\nmcBMgPHjx8fc1cYQKbMnT2np7OyhqCpxMEw5pbVFpDxllaMwsweB9wQ89WUSN/zXAQe+Boxx94vM\n7LvACnf/WfI9fgTc7+6/zPdZKkcRTd4yDwFTQgwOhh9AIyJ1L2o5irJGBu7+oYiduRVYknzYA4xL\ne3pssk0qIGgRdt0NXYwc6M9s+6cvcNH7zmbTnN/oF7iIxLdmYGZj0h6eDTyd/HsxMMPMRprZBOBQ\n4LG4+tFs0rNuOrb0sGH+tJxA0P34RrraP1pWxVARaSxxrhl8w8yOJTFNtAH4DIC7rzWzu4FngH7g\nsuHOJGqkcgdBB8Lfs6ontITEpm297Hb3EznHVep4SJHmFlswcPdP5Xnu68DX4/rsfEopnTCciglU\nQd/l+rOPzjlx8vqf/YE7nnub3uQUUtC5xaCNXSLNrOnKUcR9LGQ5ij3wJf27/M+Hbs3dOHbmmeDO\n/X/dWfDgeAjf2BUpVVVE6lrTlaOo5Tr6+QJV0Ohg07ZeWgYHeH7B9Nw3S/v1H+W7hW0Sq/WRlIhU\nRtMFg1quo18oUGVPIb0YUEKiY/aSRBppWlvYd24xY9A973RUsQFKROpT0wWDSpVOiEO+QJX+C/0n\nv7iWU15YlXHNqZfcwgv7jw38LmHfOcou4VoeSYlI5TRdMKjULtc4MpLyBaoFS9cx8aVn6P4/V2a8\nZmvbPky7ZhGbtvXSHtKPcr5zLY+kRKRymi4YQPmlE+KaR8930w46gL5jdmIfX5RPLPU71/JISkQq\np6xyFMOplspR5C35kHYUZEUElIrouOq+oXaDjDMAok7/pEQZ4TTSvgyRZjMs5Sia1bDMo//kJ3DR\nRRlNnz3/q/y24/ihx9mBAEo77L7QCKcSRehEpLY13T6DSgibL6/IPPqbbyZ+9WcFAtw5Y/bFGWWi\nyz0VrJb3XIjI8NLIoASxzaPnKS0Nub/Qw6arogalYlNZNT0k0rgUDEpQ8br7BxwAb7yR2fb227D3\n3nlfVmxQyr65j9qzla3b+3Kuy05lBW02E2l0CgYlqsg8+pNPwjHHZLZ973vw+c9H7gNEC0pBN/fW\n3YzWFqNvYNfoIz2VVZvNRJqHgkEJyp4+GRyElpbc9hIyu6IGpaCbe9+gM6qtlb1Gjsj5LlfctSbw\nfbTZTKQxKRgUqezpkwLrApUQFKzCbuJv9vax5trTc9q12UykuSibqEglZ+B89as5geDs2XcyYfaS\nilYCDat8um9ba+D1YTf3oMPptdlMpHFpZFCkovcYvPoqvCfzmOhnP/MvnDP6w7EszoYFqz1ad6Ot\ntSXyYrMOpxdpLgoGRSpq+iRkSuiSecuGDppJqdTibFhQ2ra9j5vOP7aom7s2m4k0DwWDIkVK59x/\nf9iyJfOFg4NDwSHOHcz5gpVu7iISRmsGReqa3M7cc47K2Ak8VAvovmTNoPRA8MgjiQXitFFCnDuY\no8716/QyEUmnkUEJcn5h79iROyV0xhlw//2Br4+zEmiUuX5tKBORbAoG5SohVTTuxdlC00HaUCYi\n2RQMSvW1r8E112S27dgBI0dGenk15+91epmIZNOaQbGeeSYxGkgPBD/4QWI0EDEQVFusVVdFpC4p\nGESVWgSeNGlXW1dXon3mzOr1qwTaUCYi2coKBmb2cTNba2aDZtaZ9dwcM1tvZuvMbGpa+/Fm9lTy\nue+YBU261xgz2C3rfyp3WLSoOv0pU96MKBFpSuWuGTwNnAP8IL3RzI4AZgCTgIOBB83sfe4+AHwf\nuBR4FPgNcAYQnHZTbXfdBTNmZLa9+ioceGBsHzlcZwhoz4GIpCtrZODuz7p7UFGe6cDP3X2nu78I\nrAdOMLMxwD7uvsIThy/fDnSV04dYvP56YjSQHghuuCExGog5EATVFdIeABGJW1zZRO3AirTHG5Nt\nfcm/s9sDmdlMYCbA+PHjK9/L4A/NfNzeDhs3Bl9bYUr5FJFqKTgyMLMHzezpgH+mx905d1/o7p3u\n3jl69Oh4P+yaa3ICwclff4Du+x6N93PTKOVTRKql4MjA3T9Uwvv2AOPSHo9NtvUk/85ur56nnoKj\nj85oOvWSW3hh/7Hw1s5h3ZmrMwREpFriSi1dDMwws5FmNgE4FHjM3V8B3jKzE5NZRJ8G7o2pD/n1\n9ydGAmmB4MazLqdj9pJEIEiKdFZBhSjlU0SqpdzU0rPNbCNwEvBrM1sK4O5rgbuBZ4DfApclM4kA\nPg/8kMSi8vNUI5Po3HOhNe2wlzFjwJ3vHj418PLhmqZRyqeIVEtZC8juvggITLZ3968DXw9oXwkc\nWc7nlmzNGpg8ObOttxf22AOojWkapXyKSDU0xw7knTvhG9/IDASPPppIFU0GAtA0jYg0r8YvVPfG\nG3DAAYm/zzwTrroKTj458FId9Sgizarxg8HAABxyCNx8cyIYFKBpGhFpRo0fDA48EDZsSJR5mLdM\nv/hFRAI0fjBAJ3uJiBTSFAvI+co8iIhIkwQDlXkQEcmvKYKBTvYSEcmvKYKB9g+IiOTXFAvI2j8g\nIpJfUwQD0P4BEZF8mmKaSERE8lMwEBERBQMREVEwEBERFAxERIQGzybqXt2jdFIRkQgaNhioOJ2I\nSHQNO01kjBvGAAAELklEQVSk4nQiItE1bDBQcToRkegaNhioOJ2ISHQNGwxUnE5EJLqGXUBWcToR\nkegaNhiAitOJiERV1jSRmX3czNaa2aCZdaa1d5hZr5mtSf5zS9pzx5vZU2a23sy+Y2ZWTh9ERKR8\n5a4ZPA2cA/w+4Lnn3f3Y5D+fTWv/PnApcGjynzPK7IOIiJSprGDg7s+6e+TEfTMbA+zj7ivc3YHb\nga5y+iAiIuWLM5toQnKK6N/N7B+Sbe3AxrRrNibbApnZTDNbaWYrX3vttRi7KiLS3AouIJvZg8B7\nAp76srvfG/KyV4Dx7v6GmR0PdJvZpGI75+4LgYUAnZ2dXuzrRUQkmoLBwN0/VOybuvtOYGfy71Vm\n9jzwPqAHGJt26dhkW0GrVq163cxeKrYvJTgAeH0YPqcWNNN3heb6vvqujavY73tIlItiSS01s9HA\nFncfMLP3klgofsHdt5jZW2Z2IvAo8Gng5ijv6e6j4+hrNjNb6e6dha+sf830XaG5vq++a+OK6/uW\nm1p6tpltBE4Cfm1mS5NPfQB40szWAL8EPuvuW5LPfR74IbAeeB64v5w+iIhI+coaGbj7ImBRQPs9\nwD0hr1kJHFnO54qISGU1bG2iMiysdgeGUTN9V2iu76vv2rhi+b6WSPcXEZFmppGBiIgoGAQxswVm\n9pyZPWlmi8xsVLX7FJew+lKNxMzOMLN1yXpYV1e7P3Eysx+b2WYze7rafYmbmY0zs4fN7Jnkf8P/\nXO0+xcXM9jCzx8zsieR3/WqlP0PBINgDwJHufjTwF2BOlfsTp3z1peqembUA3wM+AhwBXGBmR1S3\nV7G6jeap99UPXOnuRwAnApc18P+3O4FT3f0Y4FjgjGSKfsUoGARw99+5e3/y4QoyN8o1lGLrS9Wh\nE4D17v6Cu78D/ByYXuU+xcbdfw9sKXhhA3D3V9z98eTfbwPPkqe8TT3zhL8nH7Ym/6nogq+CQWEX\nob0Q9awd+Gva47z1sKQ+mVkHMJnEZtaGZGYtyb1bm4EH3L2i37WhD7fJJ0rNJTP7Momh6B3D2bdK\nK7G+lEhdMLO9Sexr+qK7v1Xt/sTF3QeAY5NrmIvM7Eh3r9jaUNMGg0I1l8zsvwPTgNO8zvNvS6kv\n1UB6gHFpjyPXw5LaZ2atJALBHe7+q2r3Zzi4+zYze5jE2lDFgoGmiQKY2RnAVcBZ7r692v2RsvwZ\nONTMJpjZ7sAMYHGV+yQVkDwl8UfAs+7+zWr3J05mNjqV1WhmbcCHgecq+RkKBsG+C7wLeCD72M5G\nk6e+VENIJgJ8AVhKYoHxbndfW91excfM7gT+BEw0s41mdnG1+xSjKcCngFPTjtj9x2p3KiZjgIfN\n7EkSP3AecPcllfwA7UAWERGNDERERMFARERQMBARERQMREQEBQMREUHBQEREUDAQEREUDEREBPj/\nGtlEw5G5gk4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1102c6e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X,y)\n",
    "\n",
    "y_pred = A @ x\n",
    "\n",
    "plt.plot(X,y_pred,'r--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 最小二乘法是非常常用的算法来求解 线性回归问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用梯度下降来求解"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LinearRegression(object):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    def fit(self,X,y):\n",
    "        X = np.insert(X, 0, 1, axis=1)\n",
    "        y = y.reshape(-1,1)\n",
    "        self.w = np.random.randn(X.shape[1],1)\n",
    "        \n",
    "        for _ in range(50):\n",
    "        \n",
    "            y_pred = X @ self.w  # 100 * 1\n",
    "            \n",
    "            \n",
    "            mse = np.mean(0.5*(y_pred-y)**2)\n",
    "\n",
    "            grad_w = X.T @ (y_pred - y)\n",
    "            \n",
    "            self.w -= 0.01 * grad_w\n",
    "            \n",
    "            print(_,mse,self.w[0][0],self.w[1][0])\n",
    "            \n",
    "    def predict(self,X):\n",
    "        X = np.insert(X, 0, 1, axis=1)\n",
    "        return X @ self.w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 [[ 1734.75699056]] 0.955185053008 60.3923330382\n",
      "1 [[ 18445.5964029]] 1.90492609609 44.6473973352\n",
      "2 [[ 10180.76495859]] 1.56316980672 48.774385072\n",
      "3 [[ 12077.05088666]] 1.66203080946 47.6911839011\n",
      "4 [[ 11560.95950818]] 1.6351668849 47.9756328586\n",
      "5 [[ 11695.19197897]] 1.64231165918 47.9009223224\n",
      "6 [[ 11659.849021]] 1.64042618749 47.9205464437\n",
      "7 [[ 11669.12627968]] 1.64092231823 47.9153916634\n",
      "8 [[ 11666.68896292]] 1.64079191042 47.9167457126\n",
      "9 [[ 11667.32916289]] 1.64082617423 47.9163900319\n",
      "10 [[ 11667.16099373]] 1.640817173 47.916483462\n",
      "11 [[ 11667.2051682]] 1.64081953752 47.9164589198\n",
      "12 [[ 11667.19356445]] 1.6408189164 47.9164653665\n",
      "13 [[ 11667.19661252]] 1.64081907956 47.9164636731\n",
      "14 [[ 11667.19581185]] 1.6408190367 47.9164641179\n",
      "15 [[ 11667.19602217]] 1.64081904796 47.9164640011\n",
      "16 [[ 11667.19596692]] 1.640819045 47.9164640318\n",
      "17 [[ 11667.19598144]] 1.64081904578 47.9164640237\n",
      "18 [[ 11667.19597763]] 1.64081904557 47.9164640258\n",
      "19 [[ 11667.19597863]] 1.64081904563 47.9164640253\n",
      "20 [[ 11667.19597836]] 1.64081904561 47.9164640254\n",
      "21 [[ 11667.19597843]] 1.64081904562 47.9164640254\n",
      "22 [[ 11667.19597841]] 1.64081904562 47.9164640254\n",
      "23 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "24 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "25 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "26 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "27 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "28 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "29 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "30 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "31 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "32 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "33 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "34 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "35 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "36 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "37 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "38 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "39 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "40 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "41 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "42 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "43 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "44 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "45 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "46 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "47 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "48 [[ 11667.19597842]] 1.64081904562 47.9164640254\n",
      "49 [[ 11667.19597842]] 1.64081904562 47.9164640254\n"
     ]
    }
   ],
   "source": [
    "lr.fit(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = lr.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x11092c400>]"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYXFWZ7/Hvm6YTOqA0SATSIQQRwtVJpI144pwxgCYg\nShNUgjrq4Bg4ggjyBJIDSjyICcYBRRk1c7yLQATSRmImXJI5jnm4dQi3BDIGuaUIEC6RITSh03nP\nH1WVVFftXbXrsntXV/0+z8ND16pdu1Y96H73Xutd7zJ3R0REmtuwpDsgIiLJUzAQEREFAxERUTAQ\nEREUDEREBAUDERFBwUBERFAwEBERFAxERATYLekORLXvvvv6uHHjku6GiMiQsnr16pfcfVSp44ZM\nMBg3bhw9PT1Jd0NEZEgxs6ejHKdhIhERUTAQEREFAxERQcFARERQMBAREYZQNpGISDPpXpNiwfL1\nPLell9HtbcyaOp6uiR2xfZ+CgYhInelek2LOrY/Q29cPQGpLL3NufQQgtoCgYSIRkTqzYPn6nYEg\nq7evnwXL18f2nQoGIiJ15rktvWW114KCgYhInRnd3lZWey0oGIiI1JlZU8fT1toyoK2ttYVZU8fH\n9p2aQBYRqTPZSWJlE4mINLmuiR2xXvzz1WSYyMx+ZmYvmtmjOW1zzSxlZg9m/jk55705ZrbBzNab\n2dRa9EFERCpXqzmDXwDTAtqvcfcJmX/+CGBmRwIzgKMyn/lXM2sJ+KyIiAySmgQDd/8T8ErEw08F\nbnT3be7+JLABmFSLfoiISGXizib6ipk9nBlG2jvT1gE8m3PMxkxbATObaWY9ZtazefPmmLsqItK8\n4gwGPwLeBUwANgH/Uu4J3H2hu3e6e+eoUSV3bRMRkQrFFgzc/QV373f3HcC/sWsoKAUcmHPomEyb\niIgkJLZgYGYH5Lw8DchmGi0BZpjZCDM7GDgUuC+ufoiISGk1WWdgZjcAHwL2NbONwOXAh8xsAuDA\nU8DZAO6+1swWAeuA7cC57t4fdF4RkablDn19MHz4oHydufugfFG1Ojs7vaenJ+luiIjE76674Ior\n4D3vgWuvrepUZrba3TtLHafaRCIi9eLmm8EMTjwR1q6Fo44atK9WOQoRkaQ98wwcdNCu1yNGwPr1\nsM8+g9YFPRmIiCTl5ZfhppvgsMN2tT3+OLz55qAGAlAwEBEZfC+8ANOnw7veBb/6FZx/Pjz9dHrS\neHx8ZaqL0TCRiMhg2boVJk2CdevSr08+Gb7znUGdGwijJwMRkbjt2AG//S3sueeuQHD11bB0aV0E\nAtCTgYg0uO41qao3ian4HO6wfDncdx9cfjmMHQtdXfC976WzhuqI1hmISMPqXpNizq2P0Nu3a11r\nW2sL86YfEzkgBJ2jdZix5+67seWNvvDg8P3vwwUXpP8+9VQ4+2yYOhWGDe6ATNR1BnoyEJGGtWD5\n+gEXcYDevn4WLF8fGAyCngCCztG3w3n1jT4AUlt6mXPrI0Bmu8pbb4XTT9918NVXw7nnDtpK4kop\nGIhIw3puS2/k9vwngOxFPj8QBOnt6+fni/5M1w+Ww89/nm4cPjydIbT//pX/gEGkYCAiDWt0exup\ngAv/6Pa2grawp4gWM/qLDKe3977Gl+/+HS2+A9beDuedl34SOPzw6n/AIFIwEJGGNWvq+MA5g1lT\nC3P5w54i+t1pa20pCBTv2LqFVT8+i923v0W/DePGv/8EpFKw996B56l3Si0VkYbVNbGDedOPoaO9\nDQM62ttCJ4+DnhbI+Uz2HPu39PPUVaew+oefZfftbwFw+j//gD2+d/WQDQSgbCIRESBC5lFfX3r8\n/5Vd270vnDSdX552bkXpqoNF2UQiImXIXswL1hNMGF24JuC88+Daa5lpxswE+hoHBQMRkYyuiR0D\n7/CDFob19cFujXfp1JyBiEi+s84qDASpVHpFcQMGAlAwEBHZ5YIL0kEgu1YA4Pbb00Fg9Ojk+jUI\nFAxERLK1gr7//V1ty5alg8CHP5xcvwZRYz7viIhEsXIlHH/8wLazz4Yf/ziZ/iRIwUBEmk/+NpOQ\nfjLYsSOZ/tQBBQMRaR69vTByZGH7EFlvFScFAxGpG7XYeyCQe3Dp6B076m5fgaQoGIhIXQirGgpU\nFxCCLvZbtwY/ITSxmmQTmdnPzOxFM3s0p20fM7vDzP6S+ffeOe/NMbMNZrbezKbWog8iMji616SY\nPH8FB89eyuT5K+hek6rJeYvtPVCRd7+7MBBs2JB+SlAgKFCr1NJfANPy2mYDd7n7ocBdmdeY2ZHA\nDOCozGf+1cxaatQPEYlR9u49taUXZ9fdey0CQjl7DxT16U+ng8ATT+xqy6aJHnJIFT1sbDUJBu7+\nJ+CVvOZTgV9m/v4l0JXTfqO7b3P3J4ENwKRa9ENE4lXzu/ccYVVDw9oL/PrX6SBwww272n7+83QQ\nmJZ/ryr54pwz2M/dN2X+fh7YL/N3B3BPznEbM20FzGwmpOtAjR07NqZuikhUNbt7z8idMN6rrZXW\nFqOvf1dmT9jeAwP8x3/AlCkD2z72MViypKI+NatBmUB2dzezsnO33H0hsBDSJaxr3jERKUs5O4eV\nkj9hvKW3j9Zhxt4jW4tvNJ+1aVNhiYj99oPnny+7LxJvMHjBzA5w901mdgDwYqY9BRyYc9yYTJuI\n1Llydg4rJWyj+ZHDd2PNNz4S/kGtFYhFnLWJlgCfz/z9eeD3Oe0zzGyEmR0MHArcF2M/RKRGytk5\nrJSyh5zc03MC+YFgxw4FghqoyZOBmd0AfAjY18w2ApcD84FFZvZF4GngUwDuvtbMFgHrgO3Aue7e\nH3hiEak7BTX/K1TWkFPQWoHXXoO3va3qfkharbKJznT3A9y91d3HuPtP3f1ldz/B3Q919xPd/ZWc\n469090Pcfby7L6tFH0RkaJk1dTxtrQOzyguGnCZNKgwEf/lL+klAgaCmVMJaRBJRdMjpjDPSQeD+\n+3d94I470kHg3e9OrM+NTOUoRCQxBUNOc+fCe7858KCf/ARmNspOw/VLwUBEkrd4MUyfPrDthBPg\nzjuT6U8TUjAQkeQ8+igcc0xhu7KDBp2CgYgMvq1bYc89C9sVBBKjYCAig0f7CtQtBQORJhHbxjFR\nBV3sX34Z9tln8PogoRQMRBpEsYt9bBvHRBEUBNasgQkT4v1eKYvWGYg0gO41KWb97qEB+wzM+t1D\nO/cZiLP0dKiDDioMBNddlx4qUiCoOwoGIg1g7pK19O0YOPnat8OZu2QtUPvS00VdcEE6CDzzzK62\nj340HQS+/OXaf5/UhIaJRIaQsKGgLb19gcdn22tZejrUT34C55xT2K4MoSFBwUBkiCg27l9KLUtP\n5wekKzve4EOf+1jhgQoCQ4qCgcgQUWzcf++Rrbz6RuHTwd4jW4Fdk8TVZhPlBqSRb/Wyas4phQcp\nCAxJCgYiQ0Sxcf9rzpjArJsfGrBlZGuLcfnHjtr5uhalpxcsX0/vW9t56jsBTwL9/cFrCGRIUDAQ\nGSKKjfvX6s6/lFVzTihoe9+5v+alPffmSQWCIU3BQGSIKDXuX6tNZwIFrBWY/tkFPNBxBJAuPy1D\nm4KByBAxWHf/AwQEgd8fcwJfPfnCna8rnYiW+qJgIDKE1Oruv2RpirA6Qe74mhQdSZa1kFgoGIg0\nmaKlKa74SnpvgXw5GUKxDkdJYhQMRJpMUIpq53/dT9d7Tyo8WGmiTUPBQKTOxF1dNDdF9e1vvs7D\n359ReJCCQNNRMBCpI0FDOBfe9CA9T7/Ct7oCdgSrwOj2NlKvvhG8VmD7dmhpGdCfcgJT4mWypWIK\nBiJ1JGgIx4Hr73mGzoP2qcmFNWitwD+c/2su/MIUuvICQTllrxMtky1V0yoRkToStsrYofpy02YF\nWUJXf/AzTJ53VzoQ5F2wyy17nUiZbKmZ2J8MzOwp4L+BfmC7u3ea2T7ATcA44CngU+7+atx9Eal3\nYauMoYpy0wFpom+2juCIr90SOJSTHeoptx+DWiZbam6wngymuPsEd+/MvJ4N3OXuhwJ3ZV6LNL1Z\nU8cTthNw2eWmA54EAI64bBmHf+2WnZvgzLn1kZ2b4GSHesICQbF+lNsu9SWpYaJTgV9m/v4l0JVQ\nP0TqStfEDj5z3NiCgFDWKt8JE4IXjbkzed5dRYdygoZ6ovZj1tTxtLW2RD5e6stgBAMH7jSz1WY2\nM9O2n7tvyvz9PLBf0AfNbKaZ9ZhZz+bNmwehqyLJ+1bXMVxzxgQ62tsw0nV/5k0/pvQk7DXXpIPA\nQw8NbHffmSpaaiin2JBOqX50Texg3vRjyu+31IXByCb6oLunzOydwB1m9njum+7uZhaY1OzuC4GF\nAJ2dnUp8lroURzplWat8n3kmvd9wvoC1AmFzEu2ZfQ/C3u9ob2PV7ONLdkWrk4eu2J8M3D2V+feL\nwGJgEvCCmR0AkPn3i3H3Q6SU7jUpJs9fwcGzlzJ5/oqd4+ilPpMdYw8ag4+Ve/pJID8QbNsWumhs\n1tTxtLYUDiG9/uZ2utekNNTTxGINBma2h5m9Lfs38BHgUWAJ8PnMYZ8Hfh9nP0RKqfSinlg6pVnh\nRjL33gvudK/dHBrUuiZ2sMfwwgGBvh3OguXrNdTTxOIeJtoPWGzpyazdgN+6+7+b2f3AIjP7IvA0\n8KmY+yESqntNiosWPUR/3t109qJe7EJYSTplVcNKQRPDM2bADTfsPHephV9/6y3cHjO3zxrqaU6x\nBgN3/yvwdwHtLwOFyyBFBln24pkfCLJK5cgX232s2PeVvUq3SEnpXMWeVLLnL7fP0hy0AlmaWqlU\nylIXyHLH2MseVgpZK5CbIZQrypOK5gUkiIKBNLVid/5trS1MOXxU0UnlcsfYw74vtaV34Lnb28sK\nAllRFn5pXkCCmA+RUrWdnZ3e09OTdDekwUyevyJwyKTFjDPffyC3rE4V7Dlc7oUzd45gmFnokFRb\naws3vXkv7/nu3MI3I/7/NH8YqtI+S+Mws9U51R9C6clAmlrYkMmZ7z+QG+59tupMofwspbBA0PG3\nF3nsWycVBoISTwL5dNcvlVIJa2lquZvMp7b00mJGb18/19/zDGGX4HIKr5Wak8A9eF+Bbdtg+PDI\n35NL2UBSCT0ZSNPrmtix8wkhe+de7F68nKybYoHjqatOKQgEX/nn76afBCoMBCKV0pOBCBHu4DPK\nzboJSuN86qpTCo679agpXHraxcybHr6bmXYRkzgpGIgQbeinxazs8fdZU8fvnNANCgIA4y65jRYz\n+nPmI/K/Q7uISdwUDERIF2p79Y3glblQeUZO18QOut47JvC97gc2ps8d4SIfZTGZSDUUDKTpda9J\n8fqb20Pf76h0SCZk1fC4S25LnzfzFBDlIq9dxCRuCgbS9BYsX0/fjsIp4/a2Vh68/CORzpE7nn/5\nfTfwhZXXFxyTDQJZxS7k+e+phITETcFAml7YRTmsoFu+7Hj+O156jid//MWC9yfPu6vohTzKRT53\n7iFLJSSklpRaKk2v2r17F/z74zz2rZP4c14gmPJ/loF70VpAUesEaTGZxE1PBtKQyknDrOqu24xV\neU2f/dQV/PngidjW9PlyF7aF9SdKX7WYTOKkYCANp1gaJoRfeMvK4Q+YHL53zFGc8Zmrdr7OLw5X\nbO9gXeQlaQoG0nDC0jDnLlnLtu07QtM4I12QQzKEjrhsmcbzZUjTnIE0nLAJ4S29fZUXniuxr4DG\n82Wo05OBNJywNMwwRXP1I+4wpqEeGeoUDCRxUSd7oxzXvSbFG28VLiBra21h99ZhgauM87OGutek\n4Jxz6LrvtoJjyyknLTKUKBhIoqLW3IlyXNDGLpBePDb340cBFLxvmXNNnr+CWVPHM/LZp+g69YOF\n/XxgY03u/PMD2pTDR7Hy8c0qPieJUzCQREWtuRPluLDKo3uM2K3gXKktvRi7SlWntvQG1hA6/Gs3\n82br7nTUoAZQUED7zT3P7HxfxeckSQoGkqioNXeK7R08ef4KnsvsJFbqXNmx/dztLoOqic487VJu\nP+wDJb+/HFHKZKv4nCRFwUASFbXmTthx2WGeUt+Rq3tNitSW3sAg8NiocZx01g8L2h12DiVVeqGO\nGlBUfE6SoNRSSVTUcgxBx+UO84TJP1f3mhRd7x0TGAjGXXJbYCDIyg7jdK9JlfjWYFHLW6j4nCQh\nsWBgZtPMbL2ZbTCz2Un1Q5IVteZO0HHFAkHgucwC5wXGXXJbQUXRMJHXJQQICmj5tFhNkpLIMJGZ\ntQDXAR8GNgL3m9kSd1+XRH8kWaVy9PMzcK45Y0LBuH+ujvY2Vs0+fldDiX0FylXpME5Q2QtlE0m9\nME8gb9rMPgDMdfepmddzANx9XthnOjs7vaenZ5B6KPUiKF20dZix5+678eobfQVDRdnXHe1tLPjz\nT/kfS39bcM6gINDe1jqgVEXuufIVBBu0P7HULzNb7e6dpY5LagK5A3g25/VG4P35B5nZTGAmwNix\nYwenZ1JXgjJw+nb4zsVjzq6Ldvbf+7/2Equu+kLBuY64bBmnH9tB2+pUQR2h7DqE3Av6uHe0seqJ\nVwrOM+XwUQNea39iaQR1nU3k7guBhZB+Mki4O5KAKEMyTnqz+n73wInh7FoB+vpZ+fhm5k0/JvQu\nPvfiPXn+isDvW/n45gGvtT+xNIKkgkEKODDn9ZhMm8gAUesMPTH/owVt//SJy1l5yPsGtD23pTdy\nHaFq10AoRVSGkqSyie4HDjWzg81sODADWJJQX6SOlcrAeeqqUwqeBp7YZwzjLrmtIBBAeWmbUXdA\nq3anNJF6kMiTgbtvN7PzgOVAC/Azd1+bRF+kvuVn4OzV1srWt7bzl28XPglA8QyhctM2o+6Apv2J\npREkkk1UCWUTJaPusmRC0kS7H9jIrJsfoq8//H/P38ukpJajlhVVRZIQNZtIwUBCBaV1trW2JLNx\nS4l9BcLWHGQFpYOKNIOowUDlKCRUsSyZMN1rUkyev4KDZy9l8vwVJUs3lDz+wgsDA8HkeXfR/cDG\nna+LTdZqyEaktLpOLZVklZslU26+fdHj99kO48YVfGbnnEDeuYtlHZ1+rHYhEylFTwYSqtwsmXKf\nJMKO73rvmIJAcOLcwvpBueeeNXU8IQNJBesCwpT7VCPSSBQMmkQlF7qoFUWzwu7Mw9rznzCC0kT5\nwx/AnSdCRoGy5+ia2BFpP4Mw2aeUVGZfhGorlIoMNRomagKVlksIKqxWLEsmuwo4qD1IdmgnaNUw\nI0bAm28WHBt0jqyOiHsjBNEqYml2CgZNIOxCd9Gih7jwpgeLXuSjrtYFAgNBsfZVc04IbL9s8cPp\nSp6zl+7sW5Rc/mry/bWKWJqdhomaQNgFrd+9JkMi2SGoMB35d+ZmoRlCly1+mFtWpwqGa4CS+x5E\n3RshiFYRS7PTk0ETiFLfp9IhkaC1CLkG3JmXWCuwivR6gbDhmlWzjy/Zv3KeZHJpFbE0Oz0ZNIEo\nO2xBZUMixTZ533ln/p+3BAcC952BoFQf4h6uqeapQqQR6MmgCeRPBA8LmeitZEgk7CJtwKpPvxsO\nKtxmMj8A5PehnEng/DIQ1ewcVulThUgjUDBoErkXurAyE1GHRHIvwGGB5cmrToGr8hq3boWRI4ue\nu5zhmqAsqd/c88zO97XJjEh0CgYNqFTRtHJTRvPPnXsBzg8EgWmiv/sdfOITkfpeTt+KDVFlKT1U\nJBoFgwYTdU1BpUMiYRfgwCAwbBj0F79YB6l285lKjxNpZgoGDSbuxVNBq4YDDUI13Ki7oCk9VKQ0\nBYMGE3c2TtFVw1BWEKh2D4Cg+YV8Sg8ViUbBoMGUm41Trj9f+mFsx46C9u4HNpZ1Ia+0REauoPmF\narKJRJqZNrdpMEGZQq0txh7Dd+NvvX2VXyAXLoSzzy5onjzvrorOF7YZjTahEamtqJvb6MmgweTf\nLbePbOX1N7ezpbcPqOAO/Pnn4YADCttzVg1XQrWAROqLViA3oK6JHayafTxPzv8oI4fvRt+OgU9/\npXYr28msIBAc/rWbOeKyZVWXdlYtIJH6oieDBhflDjx/Ijeomug/feJyVh7yvvSLgOykcieDiy0u\n0+byIoNPwaDBlZpQzp1jCMoQWvGuTs765NyC9vxgUu5kcNjisp6nX+H6e57ZuVGNVhGLDA4FgwZX\nqrzDguXreexbJwV/2J2vz18BJbKTKl3bkL+4rHtNakAgKOdcIlIdBYMGV7S8g1ngBPC4S27DgCeJ\nViuoVpPBC5avr2rrShGpXGzBwMzmAl8CsruR/293/2PmvTnAF4F+4Hx3Xx5XPySgvMN558F11xUc\nl7vhfPbOP0qtoFqtbSh2wdfEski84n4yuMbdv5vbYGZHAjOAo4DRwJ1mdpi7l1/ERspz000wY0ZB\n8xGXLSt651+qVlCtNoYJCyqW+Q4RiU8SqaWnAje6+zZ3fxLYAExKoB8NK7sN5cGzlzJ5/gqW/mld\nOk00PxBkNpepdlOXWm0ME7QJjwGfOW6s5gtEYhb3k8FXzOxzQA9wkbu/CnQA9+QcszHTVsDMZgIz\nAcaOHRtzVxtDfmZP4KbzefsK1GJTl1qdAyorrS0i1akqGJjZncD+AW9dCvwIuALwzL//BTirnPO7\n+0JgIaTLUVTT12aRzewJLCR3991w3HGD36kyaLcxkWRUFQzc/cQox5nZvwHZ2ckUcGDO22MybVID\nQU8CV3/wM/xg8pk8mRMItLBLRHLFmU10gLtvyrw8DXg08/cS4LdmdjXpCeRDgfvi6kfTCNhw/qH9\nD+XUz18DpMfxs2pRMVREGkuccwbfMbMJpIeJngLOBnD3tWa2CFgHbAfOHexMoka6K37u+GmMXlmY\nmZubJtrW2sKUw0cxef6K0H2LtbBLpLnFFgzc/R+LvHclcGVc311Mvd8VRw5UCxbAxRczOq/5iMuW\ncfqxHXTk1PSfcvgoblmdCt23OEsLu0SaV9OtQI57W8hqRApUDz4IEycWfHbnk0BfPysf3zxgT4DJ\n81eU3Dgewhd2NdKTlIgEa7pgUM919IsGqiPeAW2FF+vc4aCs/N8S5beFLRKr9ycpEamNptvPoJ7r\n6IddtFfNOaEgEHzoimVM/GZwFY/83xL221rMSi4SKxagRKRxNN2TQa1KJ8QhvxxD0FqByef8jNRe\n74TX+2kdtoPWFqOvf9ccQNBvCfvNUVYJ1/OTlIjUTtMFg1qtco1jHD170Q4qKf31M7/Or8e+f0Bb\n3w6nva2VPUbsVrQf1fzmWhWhE5H61nTBAKpf5RrXOHrX/zqdrnvvHdD23JRpjF6xjN/MXhr4mS29\nfewxovR/xkp/cz0/SYlI7TTdnEEt1Hwcff789KKx3EAwejS4M3rFsvTLkDtxIx2MnF1BqZz9ifOL\n2uV/tlZF6ESkvjXlk0G1ajaOfvvtMHVqYXvAOoCgO3SDqnYFi/qEo3pBIo1PTwYVqDoj6YUX0k8C\n+YEgU1I6SNAderW7gilTSESy9GRQgYrH0d96C0aMKGwPCQD58u/QJ89fUdXkbqknHC02E2keCgYV\nqCg7J6CQHH19sFvl/wnKDUr5F/f2ka28+kZfwXGj29u02EykySgYVCjyOHpQEHjxRRg1qiZ9gGhB\nKeji3jrMQtcp1HPZDhGpPQWDCkQaPjnkEPjrXwe2rVsHRxxR075EDUpBF/di6xQuvOnBwPNosZlI\nY1IwKFPJ4ZMTToAVKwZ+6Oqr4cILB7WP+cEq7CL+t94+Hrz8IwXtWmwm0lyUTVSmsOGTJ78xLz0k\nlBsIzj8/PTkcEAhK5fdXKhus8tce7NXWGnh82MU9aHN6LTYTaVx6MihT/h32sRvXccv1FxceWCRD\nKM7J2bBgtXvrMNpaWyJPNmtzepHmomBQpuzwyTu2bmH1Dz9beECENNE4J2fDhoO2vNHHNWdMKOvi\nrsVmIs1DwaBMF59wCKdOGlfQ3v3AxsgXzjgrgRYb69fFXUTCaM6gHGYFgeDvr7y9rEAA8e6pEHWs\nP645CxEZmvRkEMXuu8O2bQPbXn0V2tv5zwpOF2cl0Chj/VpQJiL5FAyKOfpoWLt2YNvdd8Nxx1V1\n2rgnZ0sNB2lBmYjkUzAIcuWVcNllA9uWLoWTT67ZVyQ5fq/dy0Qkn+YMcv3hD+m1ArmBYM6cdIZQ\nDQNB0up5H2gRSYaCAcDmzekg8PGP72rr6koHgW9/O7l+xUQLykQkX1XBwMw+aWZrzWyHmXXmvTfH\nzDaY2Xozm5rTfqyZPZJ571qzoEpug+S119JB4J3v3NV24IHpILB4cWLdipt2LxORfNXOGTwKTAd+\nkttoZkcCM4CjgNHAnWZ2mLv3Az8CvgTcC/wRmAYsq7If5envLywdfdFF8N3vDmo3ggzWHgJacyAi\nuaoKBu7+GEDAzf2pwI3uvg140sw2AJPM7Cng7e5+T+ZzvwK6GMxgMHo0bNo0sK2/H4YlP2KmlE8R\nSUpcV8AO4Nmc1xszbR2Zv/PbA5nZTDPrMbOezZs3V9eje+9NDwnlBoKtW9NDQnUQCEDbUIpIcko+\nGZjZncD+AW9d6u6/r32XdnH3hcBCgM7Ozmh7QwZZtAjOOCP99+jRsHo17B/0k5Ld6lEpnyKSlJLB\nwN1PrOC8KeDAnNdjMm2pzN/57fGaNg2++U346ldhr71CD0t6mEZ7CIhIUuIaH1kCzDCzEWZ2MHAo\ncJ+7bwJeM7PjMllEnwNifboA4O1vh298o2gggOSHaZTyKSJJqWoC2cxOA34AjAKWmtmD7j7V3dea\n2SJgHbAdODeTSQTwZeAXQBvpiePBzSQqIulhGu0hICJJqTabaDEQmJDv7lcCVwa09wBHV/O9camH\nYRqlfIpIEuojjaZOaJhGRJqVCtXl0DCNiDQrBYM8GqYRkWbUNMEgyfUDIiL1rimCQdLrB0RE6l1T\nTCAnvX5ARKTeNUUwSHr9gIhIvWuKYKCdvUREimuKYKD1AyIixTXFBLLWD4iIFNcUwQC0fkBEpJim\nGCYSEZHiFAxERETBQEREFAxERAQFAxERocGziVScTkQkmoYNBipOJyISXcMOE6k4nYhIdA0bDFSc\nTkQkuoaBNa+LAAAEHUlEQVQNBipOJyISXcMGAxWnExGJrmEnkFWcTkQkuoYNBqDidCIiUVU1TGRm\nnzSztWa2w8w6c9rHmVmvmT2Y+efHOe8da2aPmNkGM7vWzKyaPoiISPWqnTN4FJgO/CngvSfcfULm\nn3Ny2n8EfAk4NPPPtCr7ICIiVaoqGLj7Y+4eOXHfzA4A3u7u97i7A78Cuqrpg4iIVC/ObKKDM0NE\n/8/M/j7T1gFszDlmY6YtkJnNNLMeM+vZvHlzjF0VEWluJSeQzexOYP+Aty5199+HfGwTMNbdXzaz\nY4FuMzuq3M65+0JgIUBnZ6eX+3kREYmmZDBw9xPLPam7bwO2Zf5ebWZPAIcBKWBMzqFjMm0lrV69\n+iUze7rcvlRgX+ClQfieetBMvxWa6/fqtzaucn/vQVEOiiW11MxGAa+4e7+ZvYv0RPFf3f0VM3vN\nzI4D7gU+B/wgyjndfVQcfc1nZj3u3ln6yKGvmX4rNNfv1W9tXHH93mpTS08zs43AB4ClZrY889b/\nBB42sweBm4Fz3P2VzHtfBv4vsAF4AlhWTR9ERKR6VT0ZuPtiYHFA+y3ALSGf6QGOruZ7RUSkthq2\nNlEVFibdgUHUTL8Vmuv36rc2rlh+r6XT/UVEpJnpyUBERBQMgpjZAjN73MweNrPFZtaedJ/iElZf\nqpGY2TQzW5+phzU76f7Eycx+ZmYvmtmjSfclbmZ2oJmtNLN1mf8NfzXpPsXFzHY3s/vM7KHMb/1m\nrb9DwSDYHcDR7v4e4L+AOQn3J07F6ksNeWbWAlwHnAQcCZxpZkcm26tY/YLmqfe1HbjI3Y8EjgPO\nbeD/ttuA493974AJwLRMin7NKBgEcPfb3X175uU9DFwo11DKrS81BE0CNrj7X939LeBG4NSE+xQb\nd/8T8ErJAxuAu29y9wcyf/838BhFytsMZZ72euZla+afmk74KhiUdhZaCzGUdQDP5rwuWg9LhiYz\nGwdMJL2YtSGZWUtm7daLwB3uXtPf2tCb2xQTpeaSmV1K+lH0+sHsW61VWF9KZEgwsz1Jr2u6wN1f\nS7o/cXH3fmBCZg5zsZkd7e41mxtq2mBQquaSmX0BOAU4wYd4/m0l9aUaSAo4MOd15HpYUv/MrJV0\nILje3W9Nuj+Dwd23mNlK0nNDNQsGGiYKYGbTgIuBj7v7G0n3R6pyP3ComR1sZsOBGcCShPskNZDZ\nJfGnwGPufnXS/YmTmY3KZjWaWRvwYeDxWn6HgkGwHwJvA+7I37az0RSpL9UQMokA5wHLSU8wLnL3\ntcn2Kj5mdgNwNzDezDaa2ReT7lOMJgP/CByfs8XuyUl3KiYHACvN7GHSNzh3uPtttfwCrUAWERE9\nGYiIiIKBiIigYCAiIigYiIgICgYiIoKCgYiIoGAgIiIoGIiICPD/AQrqaYslsfJuAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11092c518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X,y)\n",
    "\n",
    "plt.plot(X,y_pred,'r--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 可以发现，使用梯度下降算法得到了与最小二乘法差不多的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在周老师的书中，还提出了最优解的闭式解\n",
    "\n",
    "* 参见西瓜书 54页"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_mean = np.mean(X)\n",
    "Y = y.reshape(-1,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = np.sum(Y*(X-x_mean))/(np.sum(X**2)-1./X.shape[0]*(np.sum(X))**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = np.sum(Y-w*X)/X.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.869628157291537, 52.046120767103673)"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b,w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 可见，这三种方式都是求线性规划的方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "* 使用最小二乘法来求解 $A^TAx=A^Tb$\n",
    "\n",
    "* 使用梯度下降算法\n",
    "\n",
    "* 使用闭包解\n",
    "\n",
    "## 参考\n",
    "\n",
    "* 54、55页周志华老师的西瓜书"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}