Assignment2.Exercise3.Multivariate Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment2.Exercise3: Multivariate Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exe3.Question1. Read data and shuffle the rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.loadtxt('breast_cancer.csv' ,delimiter=',')\n",
    "\n",
    "np.random.shuffle(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exe3.Question2. Replace the response 2 qnd 4 with 0 and 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = lambda x:1 if x[0]==4 else 0\n",
    "\n",
    "data[:,9] = np.apply_along_axis(f,1,data[:,9].reshape(-1,1)) \n",
    "\n",
    "size = data.shape[0]\n",
    "train, test = data[:int(0.70*size),:], data[int(0.70*size):,:]\n",
    "# 70%  training and 30% testing\n",
    "\n",
    "X = train[:,:9]\n",
    "y=train[:,9]\n",
    "\n",
    "Xtest = test[:,:9]\n",
    "ytest=test[:,9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason which I choose 70% and 30%:\n",
    "Generally, k-fold cross validation; e.g. 10-fold cross validation, is the best.\n",
    "Based on most of Literature reviews about this issue splitting the dataset; few training data causes that  parameter estimates will have greater variance, whereas with few test data, our performance statistic will have greater variance. Therefore, we need to have compromise about that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exe3.Question3. Normalize the Training data and Test data and Print the hyperparameters and Plot the Cost Function over iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize(X):\n",
    "    means = X.mean(axis=0)\n",
    "    stds = X.std(axis=0)\n",
    "    \n",
    "    def normal_(x):\n",
    "        return (x - means) / stds\n",
    "    \n",
    "    return normal_\n",
    "\n",
    "\n",
    "func_normal=normalize(X)\n",
    "normalX = func_normal(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xe = np.c_[np.ones((normalX.shape[0],1)),normalX]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logistic(x):\n",
    "    return np.exp(x)/(1 + np.exp(x))\n",
    "\n",
    "\n",
    "def CostFunc(theta, X, y):\n",
    "  \n",
    "    J = - (1 / X.shape[0]) * (np.log(logistic(X @ theta)).T @ y + np.log(1 - logistic(X @ theta)).T @ (1 - y))\n",
    "   \n",
    "    return J\n",
    "\n",
    "def gradient_descent(X, y, theta,N, rate):\n",
    "    costList = []\n",
    "    for i in range(N):\n",
    "        prediction = logistic(X@ theta)\n",
    "        error = prediction - y\n",
    "        costList.append(CostFunc(theta,X,y))\n",
    "        theta = theta - (rate * (X.T @ error))\n",
    "\n",
    "    return theta,costList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "init_theta = np.zeros(Xe.shape[1])\n",
    "niter=1000\n",
    "rate = 0.0001\n",
    "\n",
    "theta,costList = gradient_descent(Xe, y, theta=init_theta, N=niter, rate=rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcHWWd7/HP95zTSzrpEJI0AcISggEmMCwaENfBERhGHXAbBXSUGa/oncH1qgN6r87gLM7odZl54cJ10BlGRUWRqCiOCu5AAkIkMEDYQyAJhOxJb+d3/6g6J5XDOd2nk64+6T7f9+tVr1P11FN1nupK+ttVTy2KCMzMzAAKrW6AmZntOxwKZmZW5VAwM7Mqh4KZmVU5FMzMrMqhYGZmVQ4Fswkk6Q2SftTqdpg14lCwlpF0oqRbJW1PP08coe5sSddI2ibpYUnn18w/Py3fJuk7kmY3s6ykgyQtlbRGUkhasBfbc1q6jstqyn8p6QKAiPhKRJyZmReSnrWn3zmeJF0u6R5J5Up7a+a/R9ITkjZJukJSV2beAkk3pPvyvyWd3uyytm9xKFhLSOoErgX+E9gf+Hfg2rS8nsuAAWAe8Abgc5KOTdd1LPAF4M/S+duBzzazLFAGfgi8Zpw2bRvwpr0Jl2ZJKo3zKu8A/hK4rc53/RFwMfBSYAGwEPjbTJWvAb8F5gAfAq6W1NfksrYviQgPHkYcgGcDtwCbSX6RzwceBrQX6zwTeCy7DuAR4Kw6daeT/FI/KlN2JfCxdPwfgK9m5h2Z1u8dbdlMWQkIYMFebNNpwGrgX4EvZcp/CVyQjl8A/DId/3n6nduArcDr0/JXALcDG4FfA8dn1vUQ8NfACqAfKOWwv6vtzZR9FfiHzPRLgSfS8aPStvRm5v8CePtoy3rY9wYfKdiIJE0DvgtcDswl+eX5beAHkf4Pr7PMxhGGi9NqxwIrataxIi2vdRQwHBH3ZsruyNQ9Np0GICLuJw2CJpbNw98Dr5F09EiVIuLF6egJETEjIr4u6dnAFcDbSP7q/gKwtOZ0y3nAy4FZETFUu15JK0b4+X+2tn6TdvsZp+PzJM1J5z0QEVtq5tfdPzXL2j5mvA8/beo5leTfyRURUZb0JeCbwEcaLRARs5pY7wxgU03ZJpK/7sdad6T5w2P4nnEREU9I+jxwKfD6MS7+VuALEXFzOv3vkj5Ish9+lpb9S0Q8OsL3Hz/WNjeh9mdcGe+tM68yf34Tyz41vs20veVQsNHMAx6OiHI6fQfJqY6f7uV6twIza8pmAlv2oO5I88tj+J7x9E/A/ZJOGONyhwNvlvSOTFkncHBmumEg5Kj2Z1wZ31JnXmV+o/2TXdb2MT59ZKNZS9IRXHEMIGCw0QKSto4wfDCtthI4XpIyix6flte6FyhJWpQpOyFTd2U6Xfn+hUBXutxoy+YiIp4CPg18dIyLPgr8fUTMygw9EfG17OpHWoGklSP8/D8/xvZU7PYzTsfXptu5Elgoqbdmft39U7Os7Wta3anhYd8egGnAOuAF6fR1JB2jz9nL9XaSdFa/i+QX+EXpdGeD+leRXOEyHXgBySmIY9N5x5J0gr8onf+fwFXNLJvO707nBXA00D1Cu78MfLnBvNOA1ZnpmcCTJKdILkjLLiDtaE6nnwDOzEwvIQmG55KE73SS/oPedP5DwOk57evO9GfxK5LTWN1AIZ13VtrWxSR/JPyUTGc9cBPwiXSZV5F0kvc1s6yHfWtoeQM87PsD8AfAcuDO9Bfsq0n+An/BXq73JOBWYAfJZZAnZeZ9kKQzuzI9G/hOGkiPAOfXrOv8tHwbyRVSs8ewbNQOI7T5J8BbG8zbLRTSsg+k67wgna4NhbcDj6e/RF+Xlp0FLEvLHifpw5mIULixzs/itMz895IcOW4GvgR0ZeYtSJffAdxT28aRlvWwbw1Kd5iZjSK9h+IOkktEG54+M5vMHApmZlbljmYzM6tyKJiZWZVDwczMqibdzWtz586NBQsWtLoZZmaTyq233vpkRPSNVm/ShcKCBQtYvnx5q5thZjapSHq4mXo+fWRmZlUOBTMzq3IomJlZVa6hIOms9PV+qzLP0c/O/5Sk29PhXkkb82yPmZmNLLeOZklFktcgnkHyNqplkpZGxF2VOhHxnkz9d5A8C8fMzFokzyOFU4BVEfFARAyQPEjtnBHqn0fyJEszM2uRPENhPru/DGQ1u97EtBtJhwNH0ODFLZIulLRc0vL169ePe0PNzCyRZyioTlmjp++dC1wdEcP1ZkbE5RGxJCKW9PWNeu9FXcse2sD//dE9DA6XR69sZtam8gyF1cChmelDgDUN6p5LzqeObnv4af71p6sYGHIomJk1kmcoLAMWSToifQ79ucDS2kqSjiZ5G9NvcmwLpWKyqUPDflS4mVkjuYVCRAyRvGLxeuBu4BsRsVLSpZLOzlQ9j+TVibn+tu4oJmezBss+UjAzayTXZx9FxHUk7/TNln24Zvpv8mxDRangIwUzs9G0zR3NpcqRgjuazcwaaptQqJw+Gir7SMHMrJG2CYVdp498pGBm1kjbhEK1o9l9CmZmDbVNKFSPFHz1kZlZQ+0TCj5SMDMbVduEQkfRfQpmZqNpm1AoFXz1kZnZaNonFNIjBd+nYGbWWNuEQvU+BfcpmJk11Dah4KuPzMxG1zah4PsUzMxG1zahUH10to8UzMwaap9QKPhIwcxsNG0TCh1+yY6Z2ajaJhRK1aek+vSRmVkjbRMKHYXKfQo+UjAza6RtQqF6pOCb18zMGmq/UPBjLszMGmqbUNh1+shHCmZmjbRNKBQKoiBffWRmNpK2CQVIbmAb9NVHZmYNtVUodBTkIwUzsxHkGgqSzpJ0j6RVki5uUOd1ku6StFLSV/NsT6lY8NVHZmYjKOW1YklF4DLgDGA1sEzS0oi4K1NnEXAJ8IKIeFrSAXm1B5KH4g366iMzs4byPFI4BVgVEQ9ExABwFXBOTZ23ApdFxNMAEbEux/ZQKvhIwcxsJHmGwnzg0cz06rQs6yjgKEm/knSTpLPqrUjShZKWS1q+fv36PW5Qqeg+BTOzkeQZCqpTVvsbuQQsAk4DzgO+KGnWMxaKuDwilkTEkr6+vj1uUEexwICPFMzMGsozFFYDh2amDwHW1KlzbUQMRsSDwD0kIZGLzmLBN6+ZmY0gz1BYBiySdISkTuBcYGlNne8ALwGQNJfkdNIDeTWos1RgYMihYGbWSG6hEBFDwEXA9cDdwDciYqWkSyWdnVa7HnhK0l3ADcD7I+KpvNrUWfLpIzOzkeR2SSpARFwHXFdT9uHMeADvTYfcdRZ9pGBmNpK2uqPZp4/MzEbWdqHQ71AwM2uo7ULBfQpmZo21VSh0uU/BzGxEbRUK7lMwMxtZ+4WCTx+ZmTXUXqFQLNA/6FAwM2ukvULBRwpmZiNqu1AYLgfDfqeCmVldbRcKgDubzcwaaK9QKDoUzMxG0lah0JUeKfQPD7e4JWZm+6a2CgWfPjIzG1lbhUJXqQg4FMzMGmmrUKgeKfiyVDOzutorFNzRbGY2ovYKBfcpmJmNyKFgZmZVbRkK/e5TMDOrq71CwX0KZmYjaqtQ6PLpIzOzEbVVKLhPwcxsZO0ZCu5TMDOrq71CwX0KZmYjyjUUJJ0l6R5JqyRdXGf+BZLWS7o9Hf5Hnu3x6SMzs5GV8lqxpCJwGXAGsBpYJmlpRNxVU/XrEXFRXu3I8ukjM7OR5XmkcAqwKiIeiIgB4CrgnBy/b1SV00f9PlIwM6srz1CYDzyamV6dltV6jaQVkq6WdGi9FUm6UNJyScvXr1+/xw2SRGex4NNHZmYN5BkKqlNW+3Lk7wILIuJ44MfAv9dbUURcHhFLImJJX1/fXjWqs1Sgf8gv2TEzqyfPUFgNZP/yPwRYk60QEU9FRH86+f+A5+TYHgC6O4rsHPSRgplZPXmGwjJgkaQjJHUC5wJLsxUkHZSZPBu4O8f2ADCts8DOQR8pmJnV09TVR5KeDyzI1o+I/xhpmYgYknQRcD1QBK6IiJWSLgWWR8RS4J2SzgaGgA3ABXuyEWMxraPIjgGHgplZPaOGgqQrgSOB24HKb9MARgwFgIi4DriupuzDmfFLgEvG0N69Nq2jyA4fKZiZ1dXMkcISYHFE1HYST0rTOh0KZmaNNNOncCdwYN4NmSg+fWRm1lgzRwpzgbsk3QJUrhQiIs7OrVU58pGCmVljzYTC3+TdiInU7SMFM7OGRg2FiPiZpHnAyWnRLRGxLt9m5aens+hLUs3MGhi1T0HS64BbgD8FXgfcLOm1eTcsL9M6imz3kYKZWV3NnD76EHBy5ehAUh/JIymuzrNhealckhoRSPWexGFm1r6aufqoUHO66Kkml9sndXcWAT8p1cysnmaOFH4o6Xrga+n066m5IW0y6elIQmHHwDDd6biZmSWa6Wh+v6TXAC8gefLp5RFxTe4ty8m09Ehh++Aw+7e4LWZm+5qmnn0UEd8CvpVzWyZEd+ZIwczMdtcwFCT9MiJeKGkLu78HQUBExMzcW5eDaWko+LJUM7NnahgKEfHC9LN34pqTv8rpI9/VbGb2TM3cp3BlM2WTRU+lT8Gnj8zMnqGZS0uPzU5IKjEBb0jLi/sUzMwaaxgKki5J+xOOl7Q5HbYAa4FrJ6yF48x9CmZmjTUMhYj4x7Q/4eMRMTMdeiNiTvpynEnJfQpmZo01c/roFkn7VSYkzZL0yhzblKuejqRv3X0KZmbP1EwofCQiNlUmImIj8JH8mpSv7s5kk3cMDLW4JWZm+56mnn1Up6ypm972RV2lIp3FAlv7faRgZlarmVBYLumTko6UtFDSp4Bb825YnmZ0l9jaP9jqZpiZ7XOaCYV3AAPA14FvAjuBv8qzUXmb0VVi606fPjIzq9XMA/G2ARdPQFsmzIyuElv7HQpmZrWauaP5KEmXS/qRpJ9WhmZWLuksSfdIWiWpYbBIeq2kkLRkLI3fUzO6S2z2kYKZ2TM002H8TeDzwBeBpntnJRWBy4AzgNXAMklLI+Kumnq9wDuBm5td997q7Srx+KadE/V1ZmaTRjOhMBQRn9uDdZ8CrIqIBwAkXQWcA9xVU++jwD8D79uD79gjvd0l7lvnIwUzs1rNdDR/V9JfSjpI0uzK0MRy84FHM9Or07IqSScBh0bE90ZakaQLJS2XtHz9+vVNfPXIkquPHApmZrWaOVJ4c/r5/kxZAAtHWU51yqrvZZBUAD4FXDBaAyLicuBygCVLlsQo1Uc1o6vDVx+ZmdXRzNVHR+zhulcDh2amDwHWZKZ7geOAGyUBHAgslXR2RCzfw+9sSm93iYHhMv1Dw3SV/J5mM7OKUUNB0pvqlUfEf4yy6DJgkaQjgMeAc4HzM8tvAuZmvudG4H15BwIkl6QCbNk5RNcMh4KZWUUzp49Ozox3Ay8FbgNGDIWIGJJ0EXA9UASuiIiVki4FlkfE0j1s816rhMLWnUPMndHVqmaYme1zmjl99I7sdPrE1KbevBYR1wHX1ZR9uEHd05pZ53jo7U5DwZ3NZma7aebqo1rbgUXj3ZCJNKN71+kjMzPbpZk+he+y66qhArAY+Eaejcpbb1cH4CMFM7NazfQpfCIzPgQ8HBGrc2rPhNh1pOAnpZqZZTUMBUmnRsRNEfGziWzQRJiZhsLmHQ4FM7OskfoUPlsZkfSbCWjLhNlvWnL66OntDgUzs6yRQiF7R3J33g2ZSKVigd7uEhu3D7S6KWZm+5SR+hQKkvYnCY7KeDUoImJD3o3L0/49nWz06SMzs92MFAr7kbx2sxIEt2XmNfPso33a/j0dPn1kZlajYShExIIJbMeEm9XTydM+fWRmtps9uXltSkiOFBwKZmZZbRsKs3o62ejTR2Zmu2nbUNi/p5MtO4cYGi63uilmZvuMUUNB0jMeflevbLKZ1ZPcq+ArkMzMdmnmSOHY7ISkIvCcfJozcaqh4H4FM7OqhqEg6RJJW4DjJW1Ohy3AOuDaCWthTvbv6QR8V7OZWVbDUIiIf4yIXuDjETEzHXojYk5EXDKBbczF7OlpKGzzkYKZWUUzp4++J2k6gKQ3SvqkpMNzblfuKm9ce3KrQ8HMrKKZUPgcsF3SCcAHgIcZ5VWck8GcGcmRwrotO1vcEjOzfUczoTAUEQGcA3wmIj4D9ObbrPx1FAvMnt7J+i39rW6Kmdk+o5mX7GyRdAnwZ8CL0quPOvJt1sQ4oLfLoWBmltHMkcLrgX7gLyLiCWA+8PFcWzVB+nq7WL/VoWBmVjFqKKRB8BVgP0mvAHZGxKTvUwDom9HFus0OBTOzimbuaH4dcAvwp8DrgJslvTbvhk2EypFC0mViZmbN9Cl8CDg5ItYBSOoDfgxcnWfDJkJfbxcDQ2U27xyqvqLTzKydNdOnUKgEQuqpJpdD0lmS7pG0StLFdea/XdLvJN0u6ZeSFjfZ7nHR15vcq+DOZjOzRDO/3H8o6XpJF0i6APg+8IPRFkqvUroM+GNgMXBenV/6X42I34+IE4F/Bj45ptbvpQN6k1dPr93sexXMzKCJ00cR8X5JrwZeSPJqzssj4pom1n0KsCoiHgCQdBXJvQ53Zda9OVN/OslrPifM/FnTAHhs446J/Fozs31Ww1CQ9CxgXkT8KiK+DXw7LX+xpCMj4v5R1j0feDQzvRp4bp3v+SvgvUAn8IcN2nIhcCHAYYcdNsrXNu/A/bqRYI1DwcwMGPn00aeBLXXKt6fzRqM6Zc84EoiIyyLiSOCvgf9db0URcXlELImIJX19fU18dXM6SwUO6O1yKJiZpUYKhQURsaK2MCKWAwuaWPdq4NDM9CHAmhHqXwW8son1jquDZ01jzUb3KZiZwcih0D3CvGlNrHsZsEjSEZI6gXOBpdkKkhZlJl8O3NfEesfVwbOmuU/BzCw1Uigsk/TW2kJJbwFuHW3FETEEXARcD9wNfCMiVkq6VNLZabWLJK2UdDtJv8Kbx7wFe2l+Ggq+gc3MbOSrj94NXCPpDewKgSUkHcKvamblEXEdcF1N2Ycz4+8aU2tzMH/WNAaGyjy5daB634KZWbtqGAoRsRZ4vqSXAMelxd+PiJ9OSMsmyGGzewB4ZMN2h4KZtb1m7lO4AbhhAtrSEkfMnQ7Ag09u4zmH79/i1piZtVZTj6uYyg7Zfxqlgnhg/dZWN8XMrOXaPhRKxQKHzenhwSe3tbopZmYt1/ahALBw7nSHgpkZDgUg6Vd48MltlMu+LNXM2ptDAVjYN4P+oTKP+2mpZtbmHArsugLJnc1m1u4cCsCzDpgBwL1rHQpm1t4cCsDcGV0c0NvFyjWbWt0UM7OWciikFh88k7vWbB69opnZFOZQSB178ExWrdtK/9Bwq5tiZtYyDoXU4oP2Y6gc3Od+BTNrYw6F1OKDZwK4X8HM2ppDIXX47B5mdJX43WMOBTNrXw6FVKEgTjpsFssferrVTTEzaxmHQsaSw2dzz9otbNox2OqmmJm1hEMh4+QF+xMBtz3iowUza08OhYwTD5tFsSCWP7Sh1U0xM2sJh0JGT2eJ4w6eyS0POhTMrD05FGo8/1lzue2RjWze6X4FM2s/DoUaLzn6AIbLwa/ue7LVTTEzm3AOhRrPPmwWvd0lbrxnfaubYmY24XINBUlnSbpH0ipJF9eZ/15Jd0laIeknkg7Psz3NKBULvGjRXG68dx0RfhObmbWX3EJBUhG4DPhjYDFwnqTFNdV+CyyJiOOBq4F/zqs9Y/HSY+axdnM/v310Y6ubYmY2ofI8UjgFWBURD0TEAHAVcE62QkTcEBHb08mbgENybE/Tzjh2Hp3FAt+9Y02rm2JmNqHyDIX5wKOZ6dVpWSNvAX5Qb4akCyUtl7R8/fr8z/XP7O7gtKP7+P6Kxxku+xSSmbWPPENBdcrq/oaV9EZgCfDxevMj4vKIWBIRS/r6+saxiY39yQkHs25LPzc98NSEfJ+Z2b4gz1BYDRyamT4EeMb5GEmnAx8Czo6I/hzbMyZnLJ7HrJ4OvnLzw61uipnZhMkzFJYBiyQdIakTOBdYmq0g6STgCySBsC7HtoxZd0eRP33OIfxo5VrWbd7Z6uaYmU2I3EIhIoaAi4DrgbuBb0TESkmXSjo7rfZxYAbwTUm3S1raYHUt8YbnHs5QOfjaLY+OXtnMbAoo5bnyiLgOuK6m7MOZ8dPz/P69tWDudP7gqD6uvOkhLnzxQqZ1FlvdJDOzXPmO5lFc9IfP4smtA+5bMLO24FAYxckLZvO8hXP4ws8fYOfgcKubY2aWK4dCE959+iLWb+nni794oNVNMTPLlUOhCc9dOIezjj2Qy264nzUbd7S6OWZmuXEoNOlDL/89yhF89Ht3tbopZma5cSg06dDZPbzzpYv4wZ1PcO3tj7W6OWZmuXAojMHbXryQkw6bxf/5zp0+jWRmU5JDYQxKxQKfet2JlAPeduWtvhrJzKYch8IYLZg7nU+//kTuXLOJD1y9wi/iMbMpxaGwB05fPI/3nXk0S+9Yw999/24Hg5lNGbk+5mIq+8vTjmT9ln7+7ZcP0lUq8P4/Ohqp3tPCzcwmD4fCHpLER/5kMf1DZT574/08vX2Qj55zLKWiD77MbPJyKOwFSfzDq45j/54OPnvj/azdvJNPn3siM7s7Wt00M7M94j9r95IkPnDWMfzdK4/j5/eu5+X/8gvueHRjq5tlZrZHHArj5I2nHs7X3/Y8ymV47ed/zaf+6176h3zJqplNLg6FcfScw/fn++98IS/7/YP4zE/u42Wf+QW/XvVkq5tlZtY0h8I4m9XTyWfOPYkv//nJ9A+VOf+LN/OmK27hzsc2tbppZmaj0mS7xn7JkiWxfPnyVjejKTsHh7nyNw9z2Y2r2Lh9kNN/7wDe8sKFnLpwti9fNbMJJenWiFgyaj2HQv427Rjkil8+yJU3PcyGbQMce/BM3njq4bz8+IN8pZKZTQiHwj5o5+Aw1/z2Mb70qwe5d+1WukoFzjz2QF554sG84Flz6e7wO6DNLB8OhX1YRLBi9Sa+ddtqrr19DZt2DNLTWeTFi/o4Y/E8XnLMAcye3tnqZprZFOJQmCT6h4b5zf1P8eO71/Lju9bxxOadABxzYC+nLpzD846cw3OPmM2sHoeEme05h8IkFBH87rFN/OK+J7npgadY9tAGdg6WAVgwp4fjD5nF8Yfsx/GHzOK4+TPp6fQN6WbWnH0iFCSdBXwGKAJfjIiP1cx/MfBp4Hjg3Ii4erR1TuVQqDUwVOaO1Ru55cENrFi9kRWrN/H4puRIQoLDZvew6IBeFs2bwVHzZrDogF6O7JvBtE73TZjZ7poNhdz+1JRUBC4DzgBWA8skLY2I7EuOHwEuAN6XVzsms85SgZMXzObkBbOrZeu27GTFo5v43WObWLVuK/eu3cKN96xjqLwr3OfN7OKw2T0cOruHwzLDobN7mDuji2LBl8OaWX15nn84BVgVEQ8ASLoKOAeohkJEPJTOK+fYjinlgN5uTl/czemL51XLBobKPPzUNu5du5X712/l0Q3beWTDdm66/ymu+e1jZA8GC4K+3i4OnNnNvHQ4cL/KeBezp3cyZ3ry2VnyvY1m7SbPUJgPPJqZXg08N8fva1udpQKL5vWyaF7vM+b1Dw3z2NM7eGTDdlY/vYO1m3fyxKadPLF5Jw89tY2bH9zAph2Dddfb21Vi9oxOZk/vZHZP+jmjkznTO9lvWgczuzuYmX72dpeYOS357PDjw80mrTxDod45ij3qwJB0IXAhwGGHHbY3bWo7XaUiC/tmsLBvRsM6OwaGWbt5J2s372TDtgGe2jbAhpphzaadrFyzmQ3bBhgYHvnArqezmAZGqRoc07tKTO8s0tNZYnpXzWdnkZ4G87tLRQo+3WU2YfIMhdXAoZnpQ4A1e7KiiLgcuBySjua9b5plTesssmDudBbMnT5q3Yhg28Awm3cMsnnnIJt3DGXGB9m8c2i3eVv6B1m3ZSfbnxxm28AQ2/uTz/IY9mJ3R4HujiQgKuNdHUW6S4XqZ3fHrnndmXldpUxZR4GuUpHOUoGOougqFegoFtLpAp3peGexQEflsyg/ksTaSp6hsAxYJOkI4DHgXOD8HL/PJoAkZnSVmNFV4mCm7dE6IoL+oTLb+ofYPpCExLb+YbYPJNPbM9Nb+4fZOZgdyuwcHKZ/KPnctGOQddl5Q7vGx0slHDozIZKUFaoBU5lXCZpiQXQUC5QKolQUpUIh/RSlYoGOgiimZR3FZLyjUq+yTGX5dF3FtLy6/pp1Zr+roygKBVGUKBbSQfJRl40qt1CIiCFJFwHXk1ySekVErJR0KbA8IpZKOhm4Btgf+BNJfxsRx+bVJts3SKr+9T4np++oBE//YJn+oV2B0T9YZmB4mIGhYHC4zMBQOflMxweGywxWPoeTdexWb2hX3V3lwcBQmS2DQzyVlg+Xg8FymaHhYKgcDA1nxsvJMq0gUQ2HUiYoSoXMZxoklbJssOy+HJQKhbQOFAsFigXSugWKYlf9mmDKrjs7Xki/v6BkPZKq60nGk3mFat0645nlC1KyzdnxuvWT7alse0Fkvm/3dRTS9tSOF9N6teuYbHK9+ykirgOuqyn7cGZ8GclpJbNxlQ0e2PceOhgRlAMGh8sMlYPh4WyIZD7LwdBw7AqamvKhdPlK0AynATQ4HJQjmR6OZP3DEZTLSTBlx6ufaf1K2XDAcDn53uFyOh6k9cuUy7BjeHhX/fKu76t+T52ycjqd/e6p7Jkh88wgU7Y8rVsJqUImbN750kWcfcLBubbXt8SatUDlL+BiwTcaAtWgKEdQLpN81owPRxBBNbyy4+Wov0w5SOvWGU/rRCShV06Da7d1pesbjgbrKMdu3/WMdWTGq+0v7xqvLB+kfyhU2175w2HX8hEwa1r+f+A4FMys5QoFUah7waJNNF9QbmZmVQ4FMzOrciiYmVmVQ8HMzKocCmZmVuVQMDOzKoeCmZlVORTMzKxq0r2jWdJ64OE9XHwu8OQ4Nmcy8Da3B29ze9ibbT48IvpGqzTpQmFvSFrezDtKpxJvc3vwNreHidhmnz4yM7Mqh4KZmVW1Wyhc3uoGtIC3uT14m9tD7tvcVn0KZmY2snY7UjAzsxE4FMzMrKpWMtY7AAAGGUlEQVQtQkHSWZLukbRK0sWtbs94kXSopBsk3S1ppaR3peWzJf2XpPvSz/3Tckn6l/TnsELSs1u7BXtOUlHSbyV9L50+QtLN6TZ/XVJnWt6VTq9K5y9oZbv3lKRZkq6W9N/p/n7eVN/Pkt6T/ru+U9LXJHVPtf0s6QpJ6yTdmSkb836V9Oa0/n2S3rw3bZryoSCpCFwG/DGwGDhP0uLWtmrcDAH/KyJ+DzgV+Kt02y4GfhIRi4CfpNOQ/AwWpcOFwOcmvsnj5l3A3ZnpfwI+lW7z08Bb0vK3AE9HxLOAT6X1JqPPAD+MiGOAE0i2fcruZ0nzgXcCSyLiOKAInMvU289fBs6qKRvTfpU0G/gI8FzgFOAjlSDZI5G+e3SqDsDzgOsz05cAl7S6XTlt67XAGcA9wEFp2UHAPen4F4DzMvWr9SbTAByS/mf5Q+B7gEju8izV7nPgeuB56XgpradWb8MYt3cm8GBtu6fyfgbmA48Cs9P99j3gj6bifgYWAHfu6X4FzgO+kCnfrd5Yhyl/pMCuf1wVq9OyKSU9XD4JuBmYFxGPA6SfB6TVpsrP4tPAB4ByOj0H2BgRQ+l0druq25zO35TWn0wWAuuBL6WnzL4oaTpTeD9HxGPAJ4BHgMdJ9tutTO39XDHW/Tqu+7sdQqHe28Cn1HW4kmYA3wLeHRGbR6pap2xS/SwkvQJYFxG3ZovrVI0m5k0WJeDZwOci4iRgG7tOKdQz6bc5Pf1xDnAEcDAwneT0Sa2ptJ9H02gbx3Xb2yEUVgOHZqYPAda0qC3jTlIHSSB8JSK+nRavlXRQOv8gYF1aPhV+Fi8Azpb0EHAVySmkTwOzJJXSOtntqm5zOn8/YMNENngcrAZWR8TN6fTVJCExlffz6cCDEbE+IgaBbwPPZ2rv54qx7tdx3d/tEArLgEXpVQudJJ1VS1vcpnEhScC/AXdHxCczs5YClSsQ3kzS11Apf1N6FcOpwKbKYepkERGXRMQhEbGAZF/+NCLeANwAvDatVrvNlZ/Fa9P6k+ovyIh4AnhU0tFp0UuBu5jC+5nktNGpknrSf+eVbZ6y+zljrPv1euBMSfunR1hnpmV7ptWdLBPUkfMy4F7gfuBDrW7POG7XC0kOE1cAt6fDy0jOpf4EuC/9nJ3WF8mVWPcDvyO5sqPl27EX238a8L10fCFwC7AK+CbQlZZ3p9Or0vkLW93uPdzWE4Hl6b7+DrD/VN/PwN8C/w3cCVwJdE21/Qx8jaTPZJDkL/637Ml+Bf4i3fZVwJ/vTZv8mAszM6tqh9NHZmbWJIeCmZlVORTMzKzKoWBmZlUOBTMzq3IoWNuRtDX9XCDp/HFe9wdrpn89nus3y5tDwdrZAmBMoZA+dXcku4VCRDx/jG0yaymHgrWzjwEvknR7+uz+oqSPS1qWPq/+bQCSTlPy3oqvktw0hKTvSLo1fd7/hWnZx4Bp6fq+kpZVjkqUrvtOSb+T9PrMum/UrnclfCW9gxdJH5N0V9qWT0z4T8faUmn0KmZT1sXA+yLiFQDpL/dNEXGypC7gV5J+lNY9BTguIh5Mp/8iIjZImgYsk/StiLhY0kURcWKd73o1yV3JJwBz02V+ns47CTiW5Hk1vwJeIOku4FXAMRERkmaN+9ab1eEjBbNdziR5tsztJI8gn0PyQhOAWzKBAPBOSXcAN5E8jGwRI3sh8LWIGI6ItcDPgJMz614dEWWSR5UsADYDO4EvSno1sH2vt86sCQ4Fs10EvCMiTkyHIyKicqSwrVpJOo3kKZ7Pi4gTgN+SPHtntHU30p8ZHyZ5icwQydHJt4BXAj8c05aY7SGHgrWzLUBvZvp64H+mjyNH0lHpy2xq7Ufy6sftko4heRVqxWBl+Ro/B16f9lv0AS8meXBbXek7MvaLiOuAd5OcejLLnfsUrJ2tAIbS00BfJnkP8gLgtrSzdz3JX+m1fgi8XdIKklci3pSZdzmwQtJtkTzSu+IaktdH3kHyZNsPRMQTaajU0wtcK6mb5CjjPXu2iWZj46ekmplZlU8fmZlZlUPBzMyqHApmZlblUDAzsyqHgpmZVTkUzMysyqFgZmZV/x/zXoGFiSLJkgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(1, len(costList) + 1), costList)\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Cost Function')\n",
    "plt.title('α ={} , Niter = {}'.format(rate,niter))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exe3.Question4: Training Error and Training accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = logistic(Xe @ theta)\n",
    "labels = y_pred >= 0.5\n",
    "\n",
    "\n",
    "compare = (labels == (y==1))\n",
    "\n",
    "ff = lambda x:1 if x[0]==True else 0\n",
    "\n",
    "compare = np.apply_along_axis(ff,1,compare.reshape(-1,1)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training error = 15\n",
      "training accuracy = 0.9686192468619247\n"
     ]
    }
   ],
   "source": [
    "correct = np.count_nonzero(compare)\n",
    "non_correct = compare.shape[0] - correct\n",
    "accuracy = correct / (correct+non_correct)\n",
    "\n",
    "print(\"training error = {}\".format(non_correct))\n",
    "\n",
    "print(\"training accuracy = {}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exe3.Question5: Test Error and Test Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "func_normal=normalize(Xtest)\n",
    "normalXt = func_normal(Xtest)\n",
    "Xte = np.c_[np.ones((normalXt.shape[0],1)),normalXt]\n",
    "\n",
    "yt_pred = logistic(Xte @ theta)\n",
    "labels_t = yt_pred >= 0.5\n",
    "\n",
    "compare_t = (labels_t == (ytest==1))\n",
    "\n",
    "compare_t = np.apply_along_axis(ff,1,compare_t.reshape(-1,1)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test error = 5\n",
      "test accuracy  = 0.975609756097561\n"
     ]
    }
   ],
   "source": [
    "correct_t = np.count_nonzero(compare_t)\n",
    "non_correct_t = compare_t.shape[0] - correct_t\n",
    "accuracy_t = correct_t / (correct_t+non_correct_t)\n",
    "\n",
    "print(\"test error = {}\".format(non_correct_t))\n",
    "\n",
    "print(\"test accuracy  = {}\".format(accuracy_t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exe3.Question6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the results with shuffling aren't the same when we repeat it. I don't think so, they would be the same even qualitatively, since with shuffling we are getting different train set and test set, therefore our model and Consequently, our prediction would not be the same. Also, It worth to mention that more training dataset have less error(more accuracy) in prediction in test set."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}