hw2_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "approved-psychology",
   "metadata": {
    "id": "approved-psychology"
   },
   "source": [
    "# Home Assignment No. 2 - part one\n",
    "\n",
    "To solve this task, you will write a lot of code to try several machine learning methods for classification and regression.\n",
    "* You are **HIGHLY RECOMMENDED** to read relevant documentation, e.g. for [python](https://docs.python.org/3/), [numpy](https://docs.scipy.org/doc/numpy/reference/), [matlpotlib](https://matplotlib.org/) and [sklearn](https://scikit-learn.org/stable/). Also remember that seminars, lecture slides, [Google](http://google.com) and [StackOverflow](https://stackoverflow.com/) are your close friends during this course (and, probably, whole life?).\n",
    "\n",
    "* If you want an easy life, you have to use **BUILT-IN METHODS** of `sklearn` library instead of writing tons of your own code. There exists a class/method for almost everything you can imagine (related to this homework).\n",
    "\n",
    "* You have to write **CODE** directly inside specified places marked by comments: **BEGIN/END Solution**. Do not create new cells.\n",
    "\n",
    "* In some problems you are asked to provide a short discussion of the results. For that find the specific place marked via **Your text answer: \\<write your answer\\>**.\n",
    "\n",
    "* For every separate problem or subproblem (if specified) you can get only 0 points or maximal points for this problem. There are **NO INTERMEDIATE scores**. So make sure that you did everything required in the task.\n",
    "\n",
    "* Your **SOLUTION** notebook **MUST BE REPRODUCIBLE**, i.e., if the reviewer decides to restart the notebook and run all cells, after all the computation he will obtain exactly the same solution (with all the corresponding plots) as in your uploaded notebook. For this purpose, we suggest fixing random `seed` or (better) define `random_state=` inside every algorithm that uses some pseudorandomness.\n",
    "\n",
    "* Your code must be clear to the reviewer. For this purpose, try to include necessary comments inside the code. But remember: **GOOD CODE MUST BE SELF-EXPLANATORY** without any additional comments.\n",
    "\n",
    "* Many `sklearn` algorithms support multithreading (Ensemble Methods, Cross-Validation, etc.). Check if the particular algorithm has `n_jobs` parameters and set it to `-1` to use all the cores.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cooperative-spokesman",
   "metadata": {
    "id": "cooperative-spokesman"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sklearn\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import cross_val_score\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "noted-bridal",
   "metadata": {
    "id": "noted-bridal"
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "guilty-contamination",
   "metadata": {
    "id": "guilty-contamination"
   },
   "source": [
    "## Task 1. Model and feature selection problem (3 points)\n",
    "\n",
    "Your goal in this task is to predict the price (```price_doc```) given some characteristics of a house. In order to do that, you will demonstrate your abilities in data preprocessing, finding feature importances, applying different ML models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "friendly-review",
   "metadata": {
    "id": "friendly-review"
   },
   "source": [
    "### Subproblem 1.1. Data preprocessing and feauture importances (1 point)\n",
    "\n",
    "The goal of this subproblem is to prepare the data for further usage. Complete all of the following subtasks:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rational-reggae",
   "metadata": {
    "id": "rational-reggae"
   },
   "source": [
    "**1.1.1) load the dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "nutritional-phoenix",
   "metadata": {
    "id": "nutritional-phoenix"
   },
   "outputs": [],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "df = pd.read_csv('data_fs.csv')\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "objective-working",
   "metadata": {
    "id": "objective-working"
   },
   "source": [
    "**1.1.2) preprocess the dataset by dropping the ```timestamp``` feature, filling ```NaN```s with 0, converting the categorical variables into dummy variables**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "another-keyboard",
   "metadata": {
    "id": "another-keyboard"
   },
   "outputs": [],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "df = df.drop('timestamp', axis=1)\n",
    "df = df.fillna(0)\n",
    "categorical_cols = [col for col in df.columns if df[col].dtype == ('O')]\n",
    "df = pd.get_dummies(df, columns=categorical_cols)\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dress-shanghai",
   "metadata": {
    "id": "dress-shanghai"
   },
   "source": [
    "**1.1.3) define ```x_train```, ```x_test```, ```y_train```, ```y_test```. The train-test split should be 70:30**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "outdoor-runner",
   "metadata": {
    "id": "outdoor-runner"
   },
   "outputs": [],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "seed = 1\n",
    "x, y = df.drop('price_doc', axis=1), df['price_doc']\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=.3, random_state=seed)\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "engaging-ready",
   "metadata": {
    "id": "engaging-ready"
   },
   "source": [
    "\n",
    "**1.1.4) plot the histogram of y values. Note that the mean of these values is very high and the distribution of the values is far from normal. Create ```y_train_log``` and ```y_test_log``` by applying $log_{y}$. Now plot the new histogram of the obtained values. You should use these log values for the next subtasks.**   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "inclusive-outreach",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 575
    },
    "id": "inclusive-outreach",
    "outputId": "6a70e6b1-b183-470a-a694-52e230c3da6b"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmoAAAEHCAYAAAAeUGpmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAuTElEQVR4nO3deZhcZZn38e8PQtiXEBASGqfDSIBAEggJBMK+C/MaM0EWkUVRQAFFGYagM4Iwvi8uo4MOoiCMhEEIq0ZhZMcIGAiBECAQyUCQLCymMSIYQsL9/nGeaoqmq7uqu5ZT3b/PdfXV5zxnu7uq++m7nuUcRQRmZmZmlj9rNDoAMzMzM+ucEzUzMzOznHKiZmZmZpZTTtTMzMzMcsqJmpmZmVlODWh0ALWw2WabRWtra1n7vvPOO6y11lq1DagKHGf1NEOM0Bxx5inG2bNn/ykiNm90HNVQSR1WC3l6X4s5rso4rso0Mq6u6q8+mai1trby6KOPlrXvkiVLGDp0aI0j6j3HWT3NECM0R5x5ilHSi42OoVoqqcNqIU/vazHHVRnHVZlGxtVV/eWuTzMzM7OccqJmZmZmllNO1MzMzMxyqk+OUTPLm3feeYdFixaxYsWKso9ZvXo1y5cvr2FUvdeIGNdZZx1aWlpyORjZrJE61jN5rUP6c1w9qb+cqJnVwaJFi9hwww1pbW1FUlnHrFy5koEDB9Y4st6pd4wRwbJly1i0aBHDhg2r23XNmkHHeiavdUh/jaun9Ze7Ps3qYMWKFQwePLjsJM06J4nBgwdX1DJp1l+4nsm3ntZfTtTM6sSVZ3X4dTQrzX8f+daT98eJmpmZmVlOeYyaWQO0TrmtqudbePERVT2fmTW/gWuvXd0TRlT3fFYWt6hVqNr/YM3y5utf/zp33313Tc590kkncdNNN9Xk3GZ9kpR9NYkNNtigV8cfeeSRPP/88xUd8+Mf/5ipU6f26rqlXHDBBXz3u9+t+LjXXnuNww47rCoxuEXNzNqtXr2aCy+8sNFhmFk/9PTTT7N69Wq22Wabso9ZtWoVp512Wg2j6pnNN9+cIUOG8OCDDzJhwoRencstamb9xMKFC9l+++057rjj2GGHHTjyyCN56623aG1t5dxzz2XMmDHceOON72v1mjVrFnvuuSejR49mt91244033mD16tWcc845jBs3jl133ZWf/OQnJa8ZEZxxxhlst912HHTQQbz66qvt2+655x522WUXRo4cyWc+8xnefvvtktc0s+YSEZxzzjnstNNOjBw5kmnTpgHw7rvvcuaZZ7L99ttz8MEHc/jhh7fXN9deey0TJ05sP8cGG2zAl7/8ZXbccUcOPPBAXnvtNQD2228/zjrrLMaOHcsll1zyvlavBQsWcNBBBzF69GjGjBnD//7v/wLwne98h3HjxjFq1CjOP//8LmP/5je/yfDhw9lrr72YP39+e/mcOXMYP348o0aNYtKkSbz++utdXvPjH/841157ba9fSydqPdA65TZ3gVpTmj9/Pl/4whd45pln2GijjfjRj34EwODBg3nsscc45phj2vdduXIlRx99NJdccglPPPEEd999N+uuuy5XXnklG2+8MbNmzeKhhx7iiiuu4IUXXuj0erfeeivz589n3rx5TJ06lYceegjIbiNw0kknMW3aNJ588klWrVrFZZddVvKaZtZcbrnlFubMmdP+d3zOOeewdOlSbrnlFl588UXmzZvHNddcw+9///v2Yx588EF23XXX9vU333yTsWPH8vTTT7PvvvvyjW98o33bypUrefTRRzn77LPfd93jjjuO008/nSeeeIKHHnqIIUOGcOedd/Lcc8/xyCOPMGfOHGbPns2MGTM6jXv27Nlcf/31zJkzh9tvv51Zs2a1bzvhhBP41re+xdy5cxk5cmR7PJ1dE2Ds2LH87ne/6/Vr6UTNrB/Zeuut25vhP/WpT/HAAw8AcPTRR39g3/nz5zNkyBDGjRsHwEYbbcSAAQO48847mTp1KjvvvDN77bUXy5Yt47nnnuv0ejNmzODYY49lzTXXZOjQoRxwwAHt5x42bBjDhw8H4MQTT2TGjBklr2lmzeWBBx5o/9vfYost2HfffZk1axYPPPAAkydPZo011mDLLbdk//33bz9m6dKlbL755u3ra6yxRnvdVFxfQed11htvvMHixYuZNGkSkD0FYL311uPOO+/kzjvvZJdddmHMmDE8++yzJeus3/3ud0yaNIn11luPjTbaiI997GMALF++nD//+c/su+++wHt1VqlrAnzoQx9iyZIlPX4NC1wDmvUjHe/hU1hff/31yz5HRPDDH/6QQw89NLd3GDez5rPuuut2eTPY4vqr0jrrvPPO49RTT+1VfJVasWJFVXoE3KJm1gALLz6i268/XHhwWftVcmuOP/7xj+1dDT//+c/Za6+9Su673XbbsXTp0vam/zfeeINVq1Zx6KGHctlll/HOO+8A8Ic//IE333yz03Pss88+TJs2jdWrV7N06VLuu+++9nMvXLiQBQsWAHDNNdew7777lrymmVVu5dtvZ7fUqNZXBfbee+/2v/3XXnuNGTNmsNtuuzFhwgRuvfVW3n33XV555RXuv//+9mN22GGH9joBsvFshfFr3dVXABtuuCEtLS384he/AODtt9/mrbfe4tBDD+Wqq67ir3/9KwCLFy9+33jZYvvssw+/+MUv+Nvf/sYbb7zBr371KwA23nhjBg0a1N6VWaizSl0Tsrpxp512quh164wTNbN+ZLvttuPSSy9lhx124PXXX+fzn/98yX0HDhzItGnTOPPMMxk9ejQHH3wwK1as4LOf/SwjRoxgzJgx7LLLLpx66qklk6lJkyax7bbbMmLECE444QT22GMPIOse+K//+i8+8YlPMHLkSNZYYw1OO+20ktdsBElXSXpV0lNFZZtKukvSc+n7oFQuST+QtEDSXEljio45Me3/nKQTG/GzmNXbpEmTGDVqFKNHj+aAAw7g29/+NltuuSWTJ09mq622YsSIEXzqU59izJgxbLzxxgAcccQR70vc1l9/fR555BF22mkn7r33Xr7+9a93e91rrrmGH/zgB4waNYo999yTl19+mUMOOYRPfvKT7LHHHowcOZIjjzyy5CSlMWPGcPTRRzN69Gg++tGPtg/DALj66qs555xzGDVqFHPmzGmPp7NrAtx3330ccUQV7nEZEX3ua9ddd41yLV68uOx9IyL+7txft3/VU6VxNkozxNmIGOfNm1fxMW+//XZVY3jhhRdixx13rOo5qx1juTp7PYFHo4r1CLAPMAZ4qqjs28CUtDwF+FZaPhz4H0DAeODhVL4p8Hz6PigtD+ru2pXUYbWQ17/jfhlXoT2rDB3/Lhr199mdZcuWRUTEn/70p9hmm21i6dKlERHx1ltvxe677x6rVq2KiIj111+/rnFV+/Xae++9o62t7QPlldZfHqNmZtaJiJghqbVD8URgv7R8NXA/cG4qn5oq3JmSNpE0JO17V0S0AUi6CzgMuK7W8Zvl1aRJk1i+fDkrV67kX//1X9lyyy2BbIzaN77xDRYvXsyHP/zhBkfZO6+99hpf+cpXGDRoUK/P5UTNrJ9obW3lqaee6n7HHnjyySc5/vjj31e29tpr8/DDD9fkeg20RUQsTcsvA1uk5a2Al4r2W5TKSpV/gKRTgFMAWlpaqjJbrKfa2toadu2u9Mu4xo/Pvpfx+7B69WpWrlxZcj0vfvOb37Dmmmu2rxfHWJgFunLlStra2moW/7Jlyz7w5ICI4I477mDw4MG9Pv/GG2/M4Ycf3mn8q1evrujv24maWZ1ExAdmXfYVI0eOZM6cOXW5VuTkeYMREZKqFkxEXA5cDjB27NgYOnRotU7dI42+fin9Lq6ZMwsX6HbX5cuXs9Zaa7XXM3mdlZ2HuIYMGcITTzzxvrJ6xBUR7bcrKpcnE5jVwTrrrMOyZctyk2Q0q4hg2bJlrLPOOo0K4ZXUpUn6Xpg6thjYumi/llRWqtys6lzP5FtP6y+3qJnVQUtLC4sWLWp/BEo5Vq9e/b7ugTxqRIzrrLMOLS0tdb1mkenAicDF6fsvi8rPkHQ9sDuwPCKWSroD+L+F2aHAIcB5dY7Z+omO9Uxe65D+HFdP6q+aJWqStgamko3hCODyiLhE0gXA54DCf6yvRsTt6ZjzgJOB1cAXI+KOVH4YcAmwJvDTiLi4VnGb1cJaa63FsGHDKjpmyZIlue3mKWiGGHtK0nVkkwE2k7QIOJ8sQbtB0snAi8BRaffbyWZ+LgDeAj4NEBFtki4CCs+hubAwscCs2jrWM3n9+3Rclalli9oq4OyIeEzShsDsNOMJ4PsR8d3inSWNAI4BdgSGAndLGp42XwocTDYQd5ak6RExr4axm1k/FxHHlth0YCf7BnB6ifNcBVxVxdDMrB+pWaKWZkYtTctvSHqGErOdkonA9RHxNvCCpAXAbmnbgoh4HiB1LUwEnKiZmZlZn1aXMWrpXkS7AA8DE8jGcpwAPErW6vY6WRI3s+iw4mnsHae3797JNXo0tb3SadWjNn23fbme0+fzOi29o2aIsxlihOaIsxliNDNrZjVP1CRtANwMnBURf5F0GXAR2bi1i4B/Bz7T2+v0Zmp7JfvObXu8R8dVQx77zjvTDHE2Q4zQHHE2Q4xmZs2qpomapLXIkrRrI+IWgIh4pWj7FcCv02pX09g9vd3MzMz6nZrdR03ZHfeuBJ6JiO8VlQ8p2m0SULhV+nTgGElrSxoGbAs8QjZbaltJwyQNJJtwML1WcZuZmZnlRS1b1CYAxwNPSpqTyr4KHCtpZ7Kuz4XAqQAR8bSkG8gmCawCTo+I1QCSzgDuILs9x1UR8XQN4zYzMzPLhVrO+nwA6Ox5Obd3ccw3gW92Un57V8eZmZn1aRL4iQP9kh8hZWZmZpZTTtTMzMzMcsqJmpmZmVlOOVEzMzMzyyknamZmZnmkzubjWX/jRM3MzMwsp5yomZmZmeVUXR7KbmZmZmVyl6cVcYuamZmZWU45UTMzMzPLKSdqZmZmZjnlMWplap1yW6NDMDMzs37GLWpmZmZmOeVEzczMzCynnKiZmZmZ5ZQTNTMzM7OccqJmZmZmllNO1MzMzMxyyomamZmZWU45UTMzMzPLKSdqZmZmZjnlRM3MzKwZSNmX9StO1MzMzMxyyomamVmFJH1Z0tOSnpJ0naR1JA2T9LCkBZKmSRqY9l07rS9I21sbHL6ZNREnamZmFZC0FfBFYGxE7ASsCRwDfAv4fkR8BHgdODkdcjLweir/ftrPzKwsTtTMzCo3AFhX0gBgPWApcABwU9p+NfDxtDwxrZO2Hyh5oJGZlceJmplZBSJiMfBd4I9kCdpyYDbw54hYlXZbBGyVlrcCXkrHrkr7D65nzGbWvAY0OgAzs2YiaRBZK9kw4M/AjcBhVTjvKcApAC0tLSxZsqS3p+yxtra2hl27K/0mrvHju96+xx5w883dnqbfvF5Vkte4nKiZmVXmIOCFiHgNQNItwARgE0kDUqtZC7A47b8Y2BpYlLpKNwaWdTxpRFwOXA4wduzYGDp0aM1/kK40+vql9Iu4Zs4s54JlnapfvF5VlMe4atb1KWlrSfdJmpdmR30plW8q6S5Jz6Xvg1K5JP0gzYyaK2lM0blOTPs/J+nEWsVsZlaGPwLjJa2XxpodCMwD7gOOTPucCPwyLU9P66Tt90ZE1DFeM2titRyjtgo4OyJGAOOB0yWNAKYA90TEtsA9aR3go8C26esU4DLIEjvgfGB3YDfg/EJyZ2ZWbxHxMNmkgMeAJ8nq0cuBc4GvSFpANgbtynTIlcDgVP4V3qvzzD7I80ysg5p1fUbEUrKBtkTEG5KeIRtUOxHYL+12NXA/WQU3EZiaPmnOlLSJpCFp37siog1A0l1k40Guq1XsZmZdiYjzyT5AFnue7MNkx31XAJ+oR1zWjxQSOjfO9nl1GaOWbvC4C/AwsEVK4gBeBrZIy+0zo5LCrKlS5R2v0aOBuOUOHhy16bsfKKvnYN+8DnLsqBnibIYYoTnibIYYzcyaWc0TNUkbADcDZ0XEX4pvHxQRIakqHwd6MxC3nH3ntj3eo+OqKY+DHDvTDHE2Q4zQHHE2Q4xmZs2qpvdRk7QWWZJ2bUTckopfSV2apO+vpvLCzKiCwqypUuVmZmZmfVotZ32KbBDtMxHxvaJNxTOgOs6MOiHN/hwPLE9dpHcAh0galCYRHJLKzMzMzPq0WnZ9TgCOB56UNCeVfRW4GLhB0snAi8BRadvtwOHAAuAt4NMAEdEm6SJgVtrvwsLEAjMzM7O+rJazPh8ASs0zPrCT/QM4vcS5rgKuql50ZmZmZvnnZ332QuuU2xodgpmZmfVhTtTMzMzMcsqJmpmZWbPykwz6PCdqZmZmZjnlRM3MzMwsp5yomZmZmeWUEzUzMzOznHKiZmZmZpZTTtTMzMzMcsqJmpmZmVlOOVEzMzMzyyknamZmZmY55UTNzMzMLKecqJmZmZnl1IBGB9AMWqfc1ugQzMysL/MzO60Et6iZmZmZ5ZQTNTMzs0Zya5p1wYmamZnlg+SkxawDJ2pmZmZmOeVEzczMzCynnKiZmZmZ5ZQTNTMzM7OccqJmZmZmllNO1MzMzMxyyomamZlZM/NtTfq0shI1SRPKKTMzy5ta1F+SNpF0k6RnJT0jaQ9Jm0q6S9Jz6fugtK8k/UDSAklzJY3pzbXNrH8pt0Xth2WWmZnlTS3qr0uA30TE9sBo4BlgCnBPRGwL3JPWAT4KbJu+TgEu6+W1zawf6fKh7JL2APYENpf0laJNGwFr1jIwM7PeqFX9JWljYB/gJICIWAmslDQR2C/tdjVwP3AuMBGYGhEBzEytcUMiYmlPYzCz/qPLRA0YCGyQ9tuwqPwvwJG1CsrMrApqVX8NA14D/kvSaGA28CVgi6Lk62Vgi7S8FfBS0fGLUtn7EjVJp5C1uNHS0sKSJUt6EWLvtLW1NebC48dn30v87A2Lqxu9jqvwc/dW4XWbPBluvrnvvl41kte4ukzUIuK3wG8l/SwiXqzkxJKuAv4BeDUidkplFwCfI6vkAL4aEbenbecBJwOrgS9GxB2p/DCyboY1gZ9GxMWVxGFm/VNv6q9uDADGAGdGxMOSLuG9bs7CtUNSVHLSiLgcuBxg7NixMXTo0GrF2yMNuf7MmYWLl9yl0a9LKb2Kq/Bz9z6I986Xlvvk61VDeYyruxa1grUlXQ60Fh8TEQd0cczPgP8EpnYo/35EfLe4QNII4BhgR2AocLek4WnzpcDBZJ9CZ0maHhHzyozbzKwn9VdXFgGLIuLhtH4TWaL2SqFLU9IQ4NW0fTGwddHxLanMzKxb5SZqNwI/Bn5K1uLVrYiYIam1zPNPBK6PiLeBFyQtAHZL2xZExPMAkq5P+zpRM7NyVVx/dSUiXpb0kqTtImI+cCBZnTQPOBG4OH3/ZTpkOnBGqr92B5Z7fJqZlavcRG1VRFRrptIZkk4AHgXOjojXycZrFLf9FsZwwAfHduze2Ul7Or6jnD7pUZu+W3JbvcaR5LXvvKNmiLMZYoTmiLMZYqS69VfBmcC1kgYCzwOfJptFf4Okk4EXgaPSvrcDhwMLgLfSvmZmZSk3UfuVpC8AtwJvFwojotJa+jLgIiDS938HPlPhOTrVm/Ed3e07t+3xHh9bTXnsO+9MM8TZDDFCc8TZBDFWq/5qFxFzgLGdbDqwk30DOL2n1zKz/q3cRO3E9P2corIAtqnkYhHxSmFZ0hXAr9NqV2M4PLbDzHqjKvWXmVkjlJWoRcSwalysw72DJgFPpeXpwM8lfY9sMsG2wCOAgG0lDSNL0I4BPlmNWMysf6hW/WVm1ghlJWppTNkHRETHGZ3Fx1xHdvPHzSQtAs4H9pO0M9mn2YXAqek8T0u6gWww7irg9IhYnc5zBnAH2e05roqIp8uJ2cwMelZ/WR9XeC5mVHQHFbOGKLfrc1zR8jpk4zAe44O33mgXEcd2UnxlF/t/E/hmJ+W3kw3GNTPriYrrLzOzvCi36/PM4nVJmwDX1yIgM7Nqcv1lZs2s3Ieyd/Qm2WNUzMyajesvM2sa5Y5R+xXZuDLIxortANxQq6DMzKrF9ZfliuSxcVaRcseoFT/yaRXwYkQsqkE8ZmbV5vrL+ofCJAnrU8rq+kwPN34W2BAYBKysZVBmZtXi+svMmllZiZqko8jua/YJsseiPCzpyFoGZmZWDa6/zKyZldv1+TVgXES8CiBpc+Bu4KZaBWZmViWuv8ysaZU763ONQiWXLKvg2D6tdcpttE65rdFhmFlprr/MrGmV26L2G0l3ANel9aPxTWjNrDm4/rJ88uB/K0OXiZqkjwBbRMQ5kv4R2Ctt+j1wba2DMzPrKddfZtYXdNei9h/AeQARcQtwC4CkkWnb/6lhbGZmvfEfuP6yPHJLmlWgu3EaW0TEkx0LU1lrTSIyM6sO119m1vS6S9Q26WLbulWMw8ys2jbpYpvrLzNrCt0lao9K+lzHQkmfBWbXJiQzs6pw/WVmTa+7MWpnAbdKOo73KraxwEBgUg3jMjPrrbNw/WVmTa7LRC0iXgH2lLQ/sFMqvi0i7q15ZGZmveD6y8z6grLuoxYR9wH31TgWM7Oqc/1lZs3Md+c2MzMzyyknamZmZmY55UTNzMzMLKecqJmZmfU1Ekye3OgorAqcqJmZmZnlVFmzPvur1im3NToEMzMz68fcomZmZmaWU07UqsStb2ZmZlZtTtTMzMzMcsqJmplZD0haU9Ljkn6d1odJeljSAknTJA1M5Wun9QVpe2tDAzezpuJEzcysZ74EPFO0/i3g+xHxEeB14ORUfjLweir/ftrPzKwsTtTMzCokqQU4AvhpWhdwAHBT2uVq4ONpeWJaJ20/MO1vZtatmt2eQ9JVwD8Ar0bETqlsU2Aa0AosBI6KiNdTpXUJcDjwFnBSRDyWjjkR+Jd02n+LiKsxM2us/wD+GdgwrQ8G/hwRq9L6ImCrtLwV8BJARKyStDzt/6fiE0o6BTgFoKWlhSVLltQy/i61tbU15sLjx2ffS/zsVYurm+tUqqK4Cteug7bhw6v2M1ZTw36/upHXuGp5H7WfAf8JTC0qmwLcExEXS5qS1s8FPgpsm752By4Ddk+J3fnAWCCA2ZKmR8TrNYzbzKwkSYUPoLMl7Vet80bE5cDlAGPHjo2hQ4dW69Q90pDrz5xZuHjJXaoSVxnXqVTZcRWuXSeN/j0qxXGVr2ZdnxExA+iYnhZ3AXTsGpgamZnAJpKGAIcCd0VEW0rO7gIOq1XMZmZlmAB8TNJC4HqyLs9LyOqtwoffFmBxWl4MbA2Qtm8MLKtnwNYgUvZl1gv1fjLBFhGxNC2/DGyRltu7BpJCt0Gp8g/oabdBV02dozZ9t6xzFNSyqyKvTbIdNUOczRAjNEeczRBjtUXEecB5AKlF7Z8i4jhJNwJHkiVvJwK/TIdMT+u/T9vvjYioc9hm1qQa9gipiAhJVausetNtUGrfuW2PVxRDrZtM89gk25lmiLMZYoTmiLMZYqyTc4HrJf0b8DhwZSq/ErhG0gKyXoZjGhSfmTWheidqr0gaEhFLU9fmq6m8vWsgKXQbLAb261B+fx3iNDPrVkTcT6qTIuJ5YLdO9lkBfKKugZlZn1Hv23MUugDgg10DJygzHlieukjvAA6RNEjSIOCQVGZmZtYcPE7NeqGWt+e4jqw1bDNJi8hmb14M3CDpZOBF4Ki0++1kt+ZYQHZ7jk8DRESbpIuAWWm/CyOi/w2KMTMz6wkJPCSyqdUsUYuIY0tsOrCTfQM4vcR5rgKuqmJoZmZmZk3BTyYwMzMzyyknamZmZmY55UTNzMysL/ONd5uaEzUzMzOznHKiZmZmZpZTTtTMzMzMcsqJmpmZmVlOOVEzMzOrRLMOzG/WuPs5J2pmZmZmOeVEzczMzCynnKiZmZmZ5VTNnvXZrFqn3AbAwouPaHAkZmZmVVY8Ts0Pa28KblEzMzMzyyknamZmZv2ZZ4Pmmrs+zczMqq1j8uNkyHrIiVoJhbFqZmZmfZKTx6bgrk8zMzOznHKiZmZmZpZTTtSKuLvTzMzM8sSJmpmZmVlOOVGrotYpt7lVzszMzKrGiZqZmZlZTjlRMzMzM8spJ2pmZmb9neT7quWUE7Ua8Dg1M7N+ygmPVZkTNTMzM7OccqJmZmZmllNO1MzMKiBpa0n3SZon6WlJX0rlm0q6S9Jz6fugVC5JP5C0QNJcSWMa+xOYWTNxomZmVplVwNkRMQIYD5wuaQQwBbgnIrYF7knrAB8Ftk1fpwCX1T9kM2tWDUnUJC2U9KSkOZIeTWX+NGpmuRcRSyPisbT8BvAMsBUwEbg67XY18PG0PBGYGpmZwCaShtQ3ajNrVo1sUds/InaOiLFp3Z9GzaypSGoFdgEeBraIiKVp08vAFml5K+ClosMWpTKz/PGM1dwZ0OgAikwE9kvLVwP3A+dS9GkUmClpE0lDiipEM7O6k7QBcDNwVkT8RUX/4CIiJEWF5zuF7MMoLS0tLFmypJrhVqStra0xFx4/Pvte4mevWlzdXKes44uOfV9chXPnQNvw4T07cI89su8331y9YIo07PerG3mNq1GJWgB3porsJxFxOZV/Gn1fotbTSq74jRm16bsV/yClVLuSzesvUEfNEGczxAjNEWczxFgLktYiS9KujYhbUvErhQ+RqWvz1VS+GNi66PCWVPY+qR68HGDs2LExdOjQmsVfjoZcf+bMwsVL7lKVuMq4TrfHdzi2Pa7CuXNiaG/iqeHvQKN/v0vJY1yNStT2iojFkj4E3CXp2eKNPfk02ptKrrDv3LbHK7lkWeespjz+AnWmGeJshhihOeJshhirSVnT2ZXAMxHxvaJN04ETgYvT918WlZ8h6Xpgd2C5ewTMrFwNGaMWEYvT91eBW4HdSJ9GAXryadTMrE4mAMcDB6QJUXMkHU6WoB0s6TngoLQOcDvwPLAAuAL4QgNitlqaPNlju6xm6t6iJml9YI2IeCMtHwJciD+NmlkTiIgHgFL/lQ/sZP8ATq9pUGbWZzWi63ML4NY08HYA8POI+I2kWcANkk4GXgSOSvvfDhxO9mn0LeDT9Q+5coXnfS68+IgGR2JmZnXhVjWrgbonahHxPDC6k/Jl9MFPo07YzMz6MClXMz2t7/GTCczMzMxyyomamZmZfZDk7twccKJmZmZmllNO1MzMzMxyyomamZmZlebuz4ZyomZmZmZd83i1hnGiZmZmZpZTTtTqpHA/NTMzM7NyOVEzMzMzy6lGPEKq3/JTCszM+oj+PF6r8LNHNDaOfsItamZmZmY55UTNzMzMLKecqJmZmZnllBM1MzMzK09/HpvXIE7UzMzMzHLKiZqZmZlZTvn2HGZmZh35FhTdK+4G9etUM07UGqD4KQW+p5qZmZmV4q5PMzMzs5xyomZmZmaWU+76xA9MNzOzMvjWFNYAblEzM7P+S3ICVg1+DWvGLWoN5okFZmbWJ3imbE24Rc3MzPKt0a1ejb6+9WtO1MzMLH8amRwVX9cJWuU6vn5OdHvFiVqOeFKDmZmZFfMYNTMz6586tvK41cdyyIlaznhygZlZD/R2ILuTtOrq7PUslI0fDzNnZsueeNAtd302AXeJmplViROyfOnq/fDYNqCJEjVJh0maL2mBpCmNjqceWqfcVjJJc/Jm1jz6Y/1VM/7n3fcUv6fdvb/98L1viq5PSWsClwIHA4uAWZKmR8S8xkZWP6f992zmtj3+vrLOkrWFFx/RXu6uU7PGc/3VA9X+Zyy5i60ZlHrfS71/nZX3wfe6KRI1YDdgQUQ8DyDpemAi4Iqug47JW7nJXMf9SiV5TgLNKtb89Vde/vlNnvzeclf/1CGL17fZaF5dTfTobLlUIteZ4t+N4uXFi3N5015FjoIpRdKRwGER8dm0fjywe0ScUbTPKcApaXU7YH6Zp98M+FMVw60Vx1k9zRAjNEeceYrx7yJi80YH0VE59Vcq72kdVgt5el+LOa7KOK7KNDKukvVXs7SodSsiLgcur/Q4SY9GxNgahFRVjrN6miFGaI44myHGZtHTOqwW8vq+Oq7KOK7K5DWuZplMsBjYumi9JZWZmeWd6y8z67FmSdRmAdtKGiZpIHAMML3BMZmZlcP1l5n1WFN0fUbEKklnAHcAawJXRcTTVTp9LroayuA4q6cZYoTmiLMZYmyoGtdftZLX99VxVcZxVSaXcTXFZAIzMzOz/qhZuj7NzMzM+h0namZmZmY51W8Ste4e4SJpbUnT0vaHJbU2IMxy4vyKpHmS5kq6R9Lf5S3Gov0mSwpJDZnuXE6cko5Kr+fTkn5e7xhTDN295x+WdJ+kx9P7fngDYrxK0quSniqxXZJ+kH6GuZLG1DtGq1xn76uki9J7OEfSnZKG5iGuom1np3plszzEJekCSYvT6zWnEX+fpWJL5WdKejbVcd/OQ1zpf23h9VooaU5O4tpZ0swU16OSdqt3XJ2KiD7/RTaA93+BbYCBwBPAiA77fAH4cVo+BpiW0zj3B9ZLy5+vd5zlxJj22xCYAcwExub0tdwWeBwYlNY/lNM4Lwc+n5ZHAAsbEOc+wBjgqRLbDwf+BxAwHni43jH6qzrvK7BR0fIXC/Vio+NK5VuTTcp4EdgsD3EBFwD/lNP3cn/gbmDttN6IOq67uuPfga/nIS7gTuCjaflw4P5Gv68R0W9a1Nof4RIRK4HCI1yKTQSuTss3AQdKdX/mSLdxRsR9EfFWWp1Jdk+mXMWYXAR8C1hRz+CKlBPn54BLI+J1gIh4tc4xQnlxBrBRWt4YWFLH+LIAImYAbV3sMhGYGpmZwCaShtQnOuupzt7XiPhL0er6ZL9/ddXF79v3gX+mATFBWX8HDVMits8DF0fE22mfutdxXb1m6X/sUcB1dQ2KknE1vK7tTH9J1LYCXipaX5TKOt0nIlYBy4HBdYmukxiSzuIsdjJZK0Y9dRtj6vbaOiI++KDR+inntRwODJf0YGruPqxu0b2nnDgvAD4laRFwO3BmfUKrSKW/u5Zjkr4p6SXgOODrjY4HQNJEYHFEPNHoWDpxRuouvkrSoEYHU2Q4sHcazvNbSeMaHVAHewOvRMRzjQ4kOQv4Tvrd/y5wXmPDyfSXRK3PkfQpYCzwnUbHUkzSGsD3gLMbHUsZBpB1f+4HHAtcIWmTRgZUwrHAzyKihaw5/pr0OpvVRER8LSK2Bq4Fzuhu/1qTtB7wVXKSNHZwGfD3wM7AUrKuvLwYAGxKNhzhHOCGBvQUdeVYGtCa1oXPA19Ov/tfBq5scDxA/0nUynmES/s+kgaQNXsuq0t0ncSQdPqoGUkHAV8DPlZo0q6j7mLcENgJuF/SQrIKYnoDJhSU81ouAqZHxDsR8QLwB7LErZ7KifNk4AaAiPg9sA7Zw4PzxI9J6puuBSY3OgiyRGgY8ESqV1qAxyRt2dCogIh4JSJWR8S7wBVkwxnyYhFwSxqS8AjwLjmpO9L/2X8EpjU6liInArek5RvJyXvZXxK1ch7hMp3sTQI4Erg30ojCOuo2Tkm7AD8hS9IaMaaqyxgjYnlEbBYRrRHRSjaO7mMR8Wie4kx+QdaaRppBNhx4vo4xQnlx/hE4EEDSDmSJ2mt1jbJ704ET0uzP8cDyiFja6KCscpKKP6xMBJ5tVCwFEfFkRHyoqF5ZBIyJiJcbHBodxmJOAjqdGd0gvyCbUICk4WQTlv7UyICKHAQ8GxGLGh1IkSXAvmn5ACAfXbKNns1Qry+yLqM/kM2w+1oqu5AsiYDsn9+NwALgEWCbnMZ5N/AKMCd9Tc9bjB32vZ8GzPos87UUWTftPOBJ4JicxjkCeJBsRugc4JAGxHgdWbfOO2T/JE8GTgNOK3otL00/w5ONes/9VZX39WayZGMu8CtgqzzE1WH7Qhoz67Oz1+ua9Ds/l+wDy5AcvZcDgf9O7+djwAF5iCuV/6xQf+To9doLmJ3q2oeBXRsVX/GXHyFlZmZmllP9pevTzMzMrOk4UTMzMzPLKSdqZmZmZjnlRM3MzMwsp5yomVlDdffA9w77NvwB9WZm9eREzcwa7WdAuY/v+hfghojYheyecz+qVVBmZnngRM1yS9KF6SkMtTj3zyQdWYtzW2Wik4cjS/p7Sb+RNFvS7yRtX9idHD402ZqbpL/28vibJG1T4TGnSTqhN9ft4twXSPqnHhy3uaTf1CIm67kBjQ7ArDOS1oyIPD7Xz+rjcrKbYT4naXeylrMDyB5Qf6ekM4H1ye5ubtYwknYE1oyIsp9qImlARPy4hmH1SES8JmmppAkR8WCj47GMW9Ss7iS1SnpW0rWSnkmfRteTtFDStyQ9BnyiuNVL0jhJD0l6QtIjkjaUtKak70ialcYrndrFNSXpPyXNl3Q38KGibQemMU9PpvFSa5e6Zq1fGwNJGwB7AjdKmkP2yLTCY3r8gHqrmVRPfEfSU6k+ODqVryHpR6neukvS7UUt8scBvyw6x18lfV/S05LukbR5Kr9f0n9IehT4UnGrl6SPSLo71TWPSfr7VH5OUf32jW5i/5qkP0h6ANiuqHxnSTPTOW6VNKira5I9duq4KrycViWu4KxRtgN+FBE7AH8BvpDKl0XEmIi4vrCjsmdgTgO+FBGjyVpR/kb2yI/lETEOGAd8TtKwEteblK45AjiBLBFA0jpkY6SOjoiRZK3Mn+/imlZ7awB/joidi752SNua4QH11rz+EdgZKPzNf0fZszz/EWglqz+OB/YoOmYC2WOHCtYHHo2IHYHfAucXbRsYEWMj4t87XPda4NJU1+wJLJV0CLAt2YPBdwZ2lbRPZ0FL2pVszObOZB9gxhVtngqcGxGjyB51VYjnA9dM5Y8Ce3d2HWsMJ2rWKC8VNa3/N9kz1iBLjjraDlgaEbMAIuIvEbEKOITsQeBzyJ7LNpisYuvMPsB1EbE6IpYA9xad+4WI+ENavzrtW+qaVmMR8RfgBUmfgPZWjtFpczM8oN6a1168V0+8QpZojUvlN0bEu5E9CP6+omOG8P7fwXd5rx4rrtugk/ottdRvFRG3AkTEioh4i6x+OwR4nOw5ndtTun7bG7g1It5Kfz/T07k3BjaJiN+m/a4G9unimgCvAkNLvkJWdx6jZo3S8SGzhfU3KziHgDMj4o7qhGSNIOk6YD9gM0mLyD7xHwdcJulfgLWA68kelHw2cIWkL5P9zpwUfmCxNdbfyD4wlFL8+1lp/fb/IuInPYqq59bBvQe54hY1a5QPSyp0H3wSeKCLfecDQySNg+wTqKQBwB1k3ZRrpfLhktYvcY4ZwNFpXNsQYP+ic7dK+khaP57sU3Spa1qVRcSxETEkItaKiJaIuDIiXoiIwyJidESMiIgL077zImJCKt85Iu5sdPzWp/yO9+qJzcla1x8BHgQmp7FqW5B9sCh4BvhI0foaQGH8Wnd1GxHxBrBI0scBJK0taT2y+u0zacwmkraS9KESp5kBfFzSuqm17P+kcy8HXpdU6Mo8HvhtF9cEGA50e09Dqx8natYo84HTJT0DDAIuK7VjRKwEjgZ+KOkJ4C6yT30/BeYBjym7WepPKN1KfCvwXNp/KvD7dO4VwKfJBq4/SdZt8eMurmlmfdetwFyy1tt7gX9OXZ03A4vI6o//JuuKXJ6OuY33J25vArulOukA4MIyrns88EVJc4GHgC3Th5CfA79PddNNQKcTmiLiMbJu1SeA/wFmFW0+kWys3VyyMWyFeD5wzVS+f/qZLCfkXgOrN0mtwK8jYqdGx2JmVg5JG0TEXyUNJmtlmxARL0tal2zM2oSIWC3prxGxQWOj7TlJM4CJEfF6o2OxjLtyzMzMuvdrSZsAA4GLUksbEfE3SecDW5FNdmlaqbv3e07S8sUtatanSBoJXNOh+O2I2L0R8ZiZVUtqzbunk00HRsSyesdj9eFEzczMzCynPJnAzMzMLKecqJmZmZnllBM1MzMzs5xyomZmZmaWU/8fYSk+zuNUPMcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "y_train_log,  y_test_log = np.log(y_train), np.log(y_test)\n",
    "\n",
    "fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 4))\n",
    "ax1.hist(y_train, bins=150, label='price_doc')\n",
    "ax2.hist(y_train_log, bins=150, color='r', label='log(price_doc)')\n",
    "ax1.grid(alpha=.4)\n",
    "ax2.grid(alpha=.4)\n",
    "ax1.set_ylabel('Count')\n",
    "ax2.set_ylabel('Count')\n",
    "ax1.set_xlabel('price_doc')\n",
    "ax2.set_xlabel('log(price_doc)')\n",
    "ax1.legend()\n",
    "ax2.legend()\n",
    "plt.show()\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "noticed-drunk",
   "metadata": {
    "id": "noticed-drunk"
   },
   "source": [
    "**1.1.5) print the number of features in the created ```x_train``` and ```x_test``` datasets. Answer the question: do you think all of these features are relevant for the adequate price prediction?**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "strategic-spending",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "strategic-spending",
    "outputId": "60227791-3f62-4d31-cbcd-a5a8aada6763"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features:\n",
      "Train : 560\n",
      "Test  : 560\n"
     ]
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "print(f'Number of features:\\nTrain : {len(x_train.columns)}\\nTest  : {len(x_test.columns)}')\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "restricted-anchor",
   "metadata": {
    "id": "restricted-anchor"
   },
   "source": [
    "**Your text answer (do not delete this comment)**:\n",
    "\n",
    "In this case, we may suspect that majority of features are not relevant in the predictions of price:\n",
    "In general, we may say that the following groups of features might be irrelevant:\n",
    "1. Redundant binary features created during OneHotEncoding e.g. `water_1line`, `railroad_1line`\n",
    "2. Features that had object type but in reality has float nature, e.g. `child_on_acc_pre_school`\n",
    "3. Columns, which has a lot of missing `NaN`'s values\n",
    "4. Strongly correlated features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "endangered-mineral",
   "metadata": {
    "id": "endangered-mineral"
   },
   "source": [
    "**1.1.6) use random forest to find the importance of the features. Plot the histogram of the importances.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "residential-tongue",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "id": "residential-tongue",
    "outputId": "3ed70854-fddc-439f-eef5-542e3de192e2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<AxesSubplot:title={'center':'importance'}>]], dtype=object)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAASPElEQVR4nO3df7DldX3f8edLVqC6dBeE7lBYuRhxUtA2hluxzbS5K6YBDC4zMRkoVqp0dqa1bRyTqRjbaZJpKrZjHGWc2B3JiIZxRdpkCWpTRG+dTActK4YVKGUhS2Sru8MPVxeJFvPuH+cDPa5795577rl77t3P8zFz5n6/n+/n+/l+3ve787rf+/2eezZVhSTp+PeCaU9AknRsGPiS1AkDX5I6YeBLUicMfEnqhIEvSZ0w8LVqJbkvydy05yEdL+L78KWjS7IX+CdV9flpz0VaDq/wpQUkWTftOUiTZOBr1UqyN8nrk/xGkk8n+f0k302yO8krkrw7yYEk30jyD4b2m0/y3iRfSfKdJDuTnDa0/Y3tdtG3W9+/cdgx35XkXuDpJJ8EXgr8UZJDSf5V6/fpJN9KcjDJl5JcMDTGx5J8OMln2ny/nOQnhrZfkOSOJE8m2Z/k11v7C5Jcl+ThJE8kuWV43tJyGfhaKy4HPgGcCtwD/DGDf79nAb8F/KfD+r8FeBtwJvAs8CGAJK8APgm8AzgD+CyDMD9xaN+rgDcAG6vqKuDPgcuran1V/YfW53PAecBfA74K3HzY8a8EfrPNdw/w2+34pwCfB/4r8NeBlwN3tn3+BXAF8LNt21PAh0f9BkmLqipfvlblC9gLvB74DeCOofbLgUPACW39FKAYBDTAPHD9UP/zgR8AJwD/BrhlaNsLgH3A3NAx33akeRxlnhvb8Te09Y8BHx3afhnwv9ryVcA9C4zzAHDx0PqZwP8F1k37XPg6Pl5e4Wut2D+0/AzweFX9cGgdYP1Qn28MLT8KvBA4ncGV86PPbaiqv2x9z1pg3x+T5IQk17dbL99h8AOBNv5zvjW0/L2huW0GHl5g6HOAP2i3mr7N4AfAD4FNR5uPNCoDX8erzUPLL2Vwpfw48H8YBCsASdL67hvqf/hb1w5f/4fAVga/fWwAZp4bboR5fQN42VG2XVpVG4deJ1fVvgX6S0ti4Ot49eYk5yd5EYN7/Le23whuAd6Q5OIkLwR+Ffg+8D+OMtZ+fjSkT2n7PAG8CPj3S5jX7cCZSd6R5KQkpyS5qG37CPDbSc4BSHJGkq1LGFs6KgNfx6tPMLiX/i3gZOBfAlTVg8CbgRsYXPFfzuCB7A+OMtZ7gX/dbrX8GvBxBreF9gH3A3eNOqmq+i7wc+243wIeAra0zR8EbgP+W5LvtnEvOtI40jj8wysdd5LMA79fVR+d9lyk1cQrfEnqhIEvSZ3wlo4kdcIrfEnqxKr4cKjTTz+9ZmZmxtr36aef5sUvfvFkJ7RGWLu196TXumHh2nft2vV4VZ0x6jirIvBnZma4++67x9p3fn6eubm5yU5ojbD2uWlPYyp6rb3XumHh2pM8+uO9F+YtHUnqhIEvSZ0w8CWpEwa+JHXCwJekThj4ktQJA1+SOmHgS1InDHxJ6sSaD/zd+w4yc91nmLnuM9OeiiStams+8CVJozHwJakTBr4kdcLAl6ROGPiS1AkDX5I6YeBLUicMfEnqhIEvSZ0w8CWpEwa+JHXCwJekThj4ktQJA1+SOmHgS1InDHxJ6oSBL0mdMPAlqRMGviR1YuTAT3JCknuS3N7Wz03y5SR7knwqyYmt/aS2vqdtn1mhuUuSlmApV/i/AjwwtP4+4ANV9XLgKeDa1n4t8FRr/0DrJ0maspECP8nZwBuAj7b1AK8Dbm1dbgKuaMtb2zpt+8WtvyRpilJVi3dKbgXeC5wC/Brwj4G72lU8STYDn6uqVyb5OnBJVT3Wtj0MXFRVjx825jZgG8CmTZsu3LFjx1gFHHjyIPufGSy/6qwNY42xVh06dIj169dPexpTYe391d5r3bBw7Vu2bNlVVbOjjrNusQ5JfgE4UFW7kswtZZJHU1Xbge0As7OzNTc33tA33LyT9+8elLH36vHGWKvm5+cZ9/u21ln73LSnccz1WjdMrvZFAx/4GeCNSS4DTgb+KvBBYGOSdVX1LHA2sK/13wdsBh5Lsg7YADyx7JlKkpZl0Xv4VfXuqjq7qmaAK4EvVNXVwBeBN7Vu1wA72/JtbZ22/Qs1yn0jSdKKWs778N8FvDPJHuAlwI2t/UbgJa39ncB1y5uiJGkSRrml87yqmgfm2/IjwGuO0OcvgF+awNwkSRPkX9pKUicMfEnqhIEvSZ0w8CWpEwa+JHXCwJekThj4ktQJA1+SOmHgS1InDHxJ6oSBL0mdMPAlqRMGviR1wsCXpE4Y+JLUCQNfkjph4EtSJwx8SeqEgS9JnTDwJakTBr4kdcLAl6ROGPiS1AkDX5I6YeBLUicMfEnqhIEvSZ0w8CWpEwa+JHXCwJekThj4ktQJA1+SOmHgS1InDHxJ6oSBL0mdMPAlqRMGviR1YtHAT3Jykq8k+dMk9yX5zdZ+bpIvJ9mT5FNJTmztJ7X1PW37zArXIEkawShX+N8HXldVfwv4KeCSJK8F3gd8oKpeDjwFXNv6Xws81do/0PpJkqZs0cCvgUNt9YXtVcDrgFtb+03AFW15a1unbb84SSY1YUnSeFJVi3dKTgB2AS8HPgz8R+CudhVPks3A56rqlUm+DlxSVY+1bQ8DF1XV44eNuQ3YBrBp06YLd+zYMVYBB548yP5nBsuvOmvDWGOsVYcOHWL9+vXTnsZUWHt/tfdaNyxc+5YtW3ZV1eyo46wbpVNV/RD4qSQbgT8AfnLUAxxlzO3AdoDZ2dmam5sba5wbbt7J+3cPyth79XhjrFXz8/OM+31b66x9btrTOOZ6rRsmV/uS3qVTVd8Gvgj8HWBjkud+YJwN7GvL+4DNAG37BuCJZc9UkrQso7xL54x2ZU+SvwL8HPAAg+B/U+t2DbCzLd/W1mnbv1Cj3DeSJK2oUW7pnAnc1O7jvwC4papuT3I/sCPJvwPuAW5s/W8EPpFkD/AkcOUKzFuStESLBn5V3Qu8+gjtjwCvOUL7XwC/NJHZSZImxr+0laROGPiS1AkDX5I6YeBLUicMfEnqhIEvSZ0w8CWpEwa+JHXCwJekThj4ktQJA1+SOmHgS1InDHxJ6oSBL0mdMPAlqRMGviR1wsCXpE4Y+JLUCQNfkjph4EtSJwx8SeqEgS9JnTDwJakTBr4kdcLAl6ROGPiS1AkDX5I6YeBLUicMfEnqhIEvSZ0w8CWpEwa+JHXCwJekThj4ktQJA1+SOmHgS1InDHxJ6sSigZ9kc5IvJrk/yX1JfqW1n5bkjiQPta+ntvYk+VCSPUnuTfLTK12EJGlxo1zhPwv8alWdD7wWeHuS84HrgDur6jzgzrYOcClwXnttA3534rOWJC3ZooFfVd+sqq+25e8CDwBnAVuBm1q3m4Ar2vJW4OM1cBewMcmZk564JGlpUlWjd05mgC8BrwT+vKo2tvYAT1XVxiS3A9dX1Z+0bXcC76qquw8baxuD3wDYtGnThTt27BirgANPHmT/M4PlV521Yawx1qpDhw6xfv36aU9jKqy9v9p7rRsWrn3Lli27qmp21HHWjdoxyXrgPwPvqKrvDDJ+oKoqyeg/OQb7bAe2A8zOztbc3NxSdn/eDTfv5P27B2XsvXq8Mdaq+fl5xv2+rXXWPjftaRxzvdYNk6t9pHfpJHkhg7C/uar+S2ve/9ytmvb1QGvfB2we2v3s1iZJmqJR3qUT4Ebggar6naFNtwHXtOVrgJ1D7W9p79Z5LXCwqr45wTlLksYwyi2dnwH+EbA7ydda268D1wO3JLkWeBT45bbts8BlwB7ge8BbJzlhSdJ4Fg389vA1C2y++Aj9C3j7MuclSZow/9JWkjph4EtSJwx8SeqEgS9JnTDwJakTBr4kdcLAl6ROGPiS1AkDX5I6YeBLUicMfEnqhIEvSZ0w8CWpEwa+JHXCwJekThj4ktQJA1+SOmHgS1InDHxJ6oSBL0mdMPAlqRMGviR1wsCXpE4Y+JLUCQNfkjph4EtSJwx8SeqEgS9JnTDwJakTBr4kdcLAl6ROGPiS1AkDX5I6YeBLUicMfEnqhIEvSZ0w8CWpE4sGfpLfS3IgydeH2k5LckeSh9rXU1t7knwoyZ4k9yb56ZWcvCRpdKNc4X8MuOSwtuuAO6vqPODOtg5wKXBee20Dfncy05QkLdeigV9VXwKePKx5K3BTW74JuGKo/eM1cBewMcmZE5qrJGkZUlWLd0pmgNur6pVt/dtVtbEtB3iqqjYmuR24vqr+pG27E3hXVd19hDG3MfgtgE2bNl24Y8eOsQo48ORB9j8zWH7VWRvGGmOtOnToEOvXr5/2NKbC2vurvde6YeHat2zZsquqZkcdZ91yJ1JVlWTxnxo/vt92YDvA7Oxszc3NjXX8G27eyft3D8rYe/V4Y6xV8/PzjPt9W+usfW7a0zjmeq0bJlf7uO/S2f/crZr29UBr3wdsHup3dmuTJE3ZuIF/G3BNW74G2DnU/pb2bp3XAger6pvLnKMkaQIWvaWT5JPAHHB6kseAfwtcD9yS5FrgUeCXW/fPApcBe4DvAW9dgTlLksawaOBX1VULbLr4CH0LePtyJyVJmjz/0laSOmHgS1InDHxJ6oSBL0mdMPAlqRMGviR1wsCXpE4s+7N0VpOZ6z7z/PLe698wxZlI0urjFb4kdcLAl6ROGPiS1AkDX5I6YeBLUicMfEnqhIEvSZ0w8CWpEwa+JHXCwJekThj4ktQJA1+SOmHgS1InDHxJ6oSBL0mdMPAlqRMGviR1wsCXpE4cV//F4TD/u0NJ+lFe4UtSJwx8SeqEgS9JnTDwJakTBr4kdcLAl6ROGPiS1AkDX5I6YeBLUicMfEnqxIp8tEKSS4APAicAH62q61fiOKPyYxYkaQWu8JOcAHwYuBQ4H7gqyfmTPo4kaWlW4gr/NcCeqnoEIMkOYCtw/woc65gb/m1hIf4WIelopnXXYSUC/yzgG0PrjwEXHd4pyTZgW1s9lOTBMY93OvD4qJ3zvjGPsgTH4hjNkmo/zlh7f47LukfMi4VqP2cpx5raxyNX1XZg+3LHSXJ3Vc1OYEprjrVbe096rRsmV/tKvEtnH7B5aP3s1iZJmqKVCPz/CZyX5NwkJwJXAretwHEkSUsw8Vs6VfVskn8O/DGDt2X+XlXdN+njDFn2baE1zNr71GvtvdYNE6o9VTWJcSRJq5x/aStJnTDwJakTqzrwk1yS5MEke5Jcd4TtJyX5VNv+5SQzQ9ve3dofTPLzx3TiEzBu7UlmkjyT5Gvt9ZFjPvllGKHuv5/kq0meTfKmw7Zdk+Sh9rrm2M16MpZZ+w+Hzvmae5PECLW/M8n9Se5NcmeSc4a2He/n/Wi1L+28V9WqfDF44Psw8DLgROBPgfMP6/PPgI+05SuBT7Xl81v/k4Bz2zgnTLumY1T7DPD1adewgnXPAH8T+DjwpqH204BH2tdT2/Kp067pWNTeth2adg0rXPsW4EVt+Z8O/Xvv4bwfsfZxzvtqvsJ//iMaquoHwHMf0TBsK3BTW74VuDhJWvuOqvp+Vf0ZsKeNt1Ysp/a1bNG6q2pvVd0L/OVh+/48cEdVPVlVTwF3AJcci0lPyHJqX+tGqf2LVfW9tnoXg7/vgT7O+0K1L9lqDvwjfUTDWQv1qapngYPAS0bcdzVbTu0A5ya5J8l/T/L3VnqyE7Sc89bDOT+ak5PcneSuJFdMdGYrb6m1Xwt8bsx9V5vl1A5LPO9T+2gFrZhvAi+tqieSXAj8YZILquo7056YVtQ5VbUvycuALyTZXVUPT3tSk5bkzcAs8LPTnsuxtkDtSzrvq/kKf5SPaHi+T5J1wAbgiRH3Xc3Grr3dxnoCoKp2Mbg/+IoVn/FkLOe89XDOF1RV+9rXR4B54NWTnNwKG6n2JK8H3gO8saq+v5R9V7Hl1L708z7thxZHeZixjsEDmHP5/w8zLjisz9v50QeXt7TlC/jRh7aPsLYe2i6n9jOeq5XBg6B9wGnTrmlSdQ/1/Rg//tD2zxg8uDu1La+JuidQ+6nASW35dOAhDnvwt5pfI/57fzWDi5fzDms/7s/7UWpf8nmfesGLfDMuA/53K/Y9re23GPyUAzgZ+DSDh7JfAV42tO972n4PApdOu5ZjVTvwi8B9wNeArwKXT7uWCdf9txnc53yawW9z9w3t+7b2/dgDvHXatRyr2oG/C+xuYbEbuHbataxA7Z8H9rd/118DbuvovB+x9nHOux+tIEmdWM338CVJE2TgS1InDHxJ6oSBL0mdMPAlqRMGviR1wsCXpE78P16Tveym3FMnAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "rf = RandomForestRegressor(random_state=seed, n_jobs=-1)\n",
    "rf.fit(x_train, y_train_log)\n",
    "\n",
    "feature_importances = pd.DataFrame(zip(x_train.columns, rf.feature_importances_), columns=['feature', 'importance'])\n",
    "feature_importances.hist(bins=100)\n",
    "\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ethical-firmware",
   "metadata": {
    "id": "ethical-firmware"
   },
   "source": [
    "**1.1.7) print the names of the 20 most important features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "amazing-marriage",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "amazing-marriage",
    "outputId": "53ab5987-aa41-43f6-816e-37580d1cff79"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>full_sq</td>\n",
       "      <td>0.242482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>life_sq</td>\n",
       "      <td>0.004718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>floor</td>\n",
       "      <td>0.006705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>max_floor</td>\n",
       "      <td>0.003285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>material</td>\n",
       "      <td>0.001351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>build_year</td>\n",
       "      <td>0.004023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>num_room</td>\n",
       "      <td>0.008017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>kitch_sq</td>\n",
       "      <td>0.003228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>state</td>\n",
       "      <td>0.002724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>area_m</td>\n",
       "      <td>0.001663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>raion_popul</td>\n",
       "      <td>0.000453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>green_zone_part</td>\n",
       "      <td>0.002172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>indust_part</td>\n",
       "      <td>0.001320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>children_preschool</td>\n",
       "      <td>0.000208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>preschool_quota</td>\n",
       "      <td>0.000665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>preschool_education_centers_raion</td>\n",
       "      <td>0.000487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>children_school</td>\n",
       "      <td>0.000209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>school_quota</td>\n",
       "      <td>0.000799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>school_education_centers_raion</td>\n",
       "      <td>0.001061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>school_education_centers_top_20_raion</td>\n",
       "      <td>0.000397</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  feature  importance\n",
       "0                                 full_sq    0.242482\n",
       "1                                 life_sq    0.004718\n",
       "2                                   floor    0.006705\n",
       "3                               max_floor    0.003285\n",
       "4                                material    0.001351\n",
       "5                              build_year    0.004023\n",
       "6                                num_room    0.008017\n",
       "7                                kitch_sq    0.003228\n",
       "8                                   state    0.002724\n",
       "9                                  area_m    0.001663\n",
       "10                            raion_popul    0.000453\n",
       "11                        green_zone_part    0.002172\n",
       "12                            indust_part    0.001320\n",
       "13                     children_preschool    0.000208\n",
       "14                        preschool_quota    0.000665\n",
       "15      preschool_education_centers_raion    0.000487\n",
       "16                        children_school    0.000209\n",
       "17                           school_quota    0.000799\n",
       "18         school_education_centers_raion    0.001061\n",
       "19  school_education_centers_top_20_raion    0.000397"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "top20features = feature_importances[:20]['feature'].to_list()\n",
    "feature_importances.head(20)\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "little-toddler",
   "metadata": {
    "id": "little-toddler"
   },
   "source": [
    "### Subproblem 1.2. Training models on the most important features (1 point)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vietnamese-patrick",
   "metadata": {
    "id": "vietnamese-patrick"
   },
   "source": [
    "In this subproblem, you will train several ML models on the found 20 most important features: train **Linear Regression**,  **Ridge regression**, **Random forest**,  **DecisionTree** and evaluate their performance using the Root Mean Squared Logarithmic Error (RMSLE) on both train/test sets. \n",
    "\n",
    "You will have to repeat the same actions in the next tasks too, so we recommend to implement\n",
    "a dedicated function for comparisons, which\n",
    "1. on input takes a training dataset `(X_train, y_train)` and a test sample `(X_test, y_test)`;\n",
    "2. it trains **all of the listed models** on the `(X_train, y_train)` sample;\n",
    "3. it computes and returns a table with the RMSLE score of each fitted model on the train and test datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stupid-principle",
   "metadata": {
    "id": "stupid-principle"
   },
   "source": [
    "**1.2.1) Implement the function described above**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "paperback-uniform",
   "metadata": {
    "id": "paperback-uniform"
   },
   "outputs": [],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "from sklearn.linear_model import LinearRegression, Ridge\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.metrics import make_scorer\n",
    "\n",
    "def custom_RMSLE(y_true_log: np.array, y_pred_log: np.array) -> float:\n",
    "    \"\"\"Calculation of Root Mean Squared Logarithmic Error (RMSLE) based on the logarithm of true and predicted values\n",
    "\n",
    "    Args:\n",
    "        y_true_log (np.array): ground truth of target values after log\n",
    "        y_pred_log (np.array): predicted values of log target\n",
    "\n",
    "    Returns:\n",
    "        float: Root Mean Squared Logarithmic Error\n",
    "    \"\"\"\n",
    "    MSLE = np.mean((np.log1p(np.exp(y_pred_log)) - np.log1p(np.exp(y_true_log))) ** 2)\n",
    "    return np.sqrt(MSLE)\n",
    "\n",
    "RMSLE = make_scorer(custom_RMSLE)\n",
    "\n",
    "def train_models(X_train, y_train, X_test, y_test) -> pd.DataFrame:\n",
    "    models_list = (LinearRegression(),\n",
    "                   Ridge(random_state=seed),\n",
    "                   DecisionTreeRegressor(random_state=seed),\n",
    "                   RandomForestRegressor(random_state=seed, n_jobs=-1)\n",
    "                   )\n",
    "    output = []\n",
    "    for model in models_list:\n",
    "        model.fit(X_train, y_train)\n",
    "\n",
    "        rmsle_train = RMSLE(model, X_train, y_train)\n",
    "        rmsle_test  = RMSLE(model, X_test,  y_test)\n",
    "        \n",
    "        output.append((type(model).__name__, rmsle_train, rmsle_test))\n",
    "    return pd.DataFrame(output, columns=['Model', 'Train error', 'Test error'])\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "demographic-honolulu",
   "metadata": {
    "id": "demographic-honolulu"
   },
   "source": [
    "**1.2.2) Apply the implemented function to our dataset (use only the 20 most important features)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "loved-bedroom",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "id": "loved-bedroom",
    "outputId": "c8bb94f8-18f2-4395-ec9c-79178786015c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model</th>\n",
       "      <th>Train error</th>\n",
       "      <th>Test error</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LinearRegression</td>\n",
       "      <td>0.523409</td>\n",
       "      <td>0.532404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ridge</td>\n",
       "      <td>0.523409</td>\n",
       "      <td>0.532403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DecisionTreeRegressor</td>\n",
       "      <td>0.036070</td>\n",
       "      <td>0.684096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestRegressor</td>\n",
       "      <td>0.183057</td>\n",
       "      <td>0.500074</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Model  Train error  Test error\n",
       "0       LinearRegression     0.523409    0.532404\n",
       "1                  Ridge     0.523409    0.532403\n",
       "2  DecisionTreeRegressor     0.036070    0.684096\n",
       "3  RandomForestRegressor     0.183057    0.500074"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### BEGIN Solution\n",
    "train_models(X_train = x_train[top20features], y_train = y_train_log,\n",
    "             X_test = x_test[top20features], y_test = y_test_log)\n",
    "### END Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6lSOGA4-7RSx",
   "metadata": {
    "id": "6lSOGA4-7RSx"
   },
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stock-jewel",
   "metadata": {
    "id": "stock-jewel"
   },
   "source": [
    "### Subproblem 1.3. Forward-backward methods (1 point)\n",
    "In the subproblems above, we have relied on the feature importances found by the random forest method. The main goal of this subproblem is to find feature importances by applying the forward-backward methods: the main idea is to add or remove features and compute how this removal influences the value of the loss function or some other criteria.\n",
    "\n",
    "The decision about adding or deleting a feature may be made based on:\n",
    "\n",
    "- AIC\n",
    "- BIC\n",
    "- validation error\n",
    "- Mallows $C_p$\n",
    "- sklearn's `estimator.score()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "southern-closer",
   "metadata": {
    "id": "southern-closer"
   },
   "source": [
    "In subtasks, 1.3.1 - 1.3.3, implement the following greedy feature selection algorithm with early stopping using the mean of the RMSLE scores achieved on the 3-fold cross-validation. Please bear in mind that **the lower** RMSLE (`mean_squared_log_error`) is, **the higher the model \"quality\" is** and look up `cross_val_score(...)` peculiarities in [scikit's manual](https://scikit-learn.org/stable/documentation.html) for more information.\n",
    "\n",
    "```python\n",
    "# Initialize with an empty list of features.\n",
    "list_of_best_features = []\n",
    "\n",
    "while round < n_rounds:\n",
    "    round = round + 1\n",
    "    \n",
    "    if no_more_features:\n",
    "        # end loop\n",
    "\n",
    "    # Iterate over currently *unused* features and use $k$-fold \n",
    "    # . `cross_val_score` to measure model \"quality\".\n",
    "    compute_quality_with_each_new_unused_feature(...)\n",
    "\n",
    "    # **Add** the feature that gives the highest \"quality\" of the model.\n",
    "    pick_and_add_the_best_feature(...)\n",
    "\n",
    "    if model_quality_has_increased_since_last_round:\n",
    "        round = 0\n",
    "\n",
    "return list_of_best_features\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suffering-italy",
   "metadata": {
    "id": "suffering-italy"
   },
   "source": [
    "**1.3.1) In the cell below, you have to implement a function that would iterate over a list of features and use $k$-fold `cross_val_score` to measure the model's \"quality\".** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "caroline-state",
   "metadata": {
    "id": "caroline-state"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "\n",
    "def selection_step(model, X, y, used_features=(), cv=3):\n",
    "    \"\"\"\n",
    "    Parameters\n",
    "    ==========\n",
    "        X: ndarray - training inputs\n",
    "        y: ndarray - training targets\n",
    "        used_features: - list of features\n",
    "        cv: int - number of folds\n",
    "\n",
    "    Returns\n",
    "    =======\n",
    "        scores - dictionary of scores\n",
    "    \"\"\"\n",
    "\n",
    "    ### BEGIN Solution\n",
    "    used_features = [i if isinstance(i, list) else [i] for i in used_features] # allows to process single feature or lists of features\n",
    "    scores = {cross_val_score(model, X[features], y, cv=cv, scoring=RMSLE, n_jobs=-1).mean() : features for features in used_features}\n",
    "    ### END Solution\n",
    "\n",
    "    return scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "laughing-patio",
   "metadata": {
    "id": "laughing-patio"
   },
   "source": [
    "**1.3.2) Implement a forward greedy feature selection function (the one described in the pseudocode in subtask 1 above) that would**\n",
    "\n",
    "**- compute the model's \"quality\" over the currently *unused* features**\n",
    "\n",
    "**- find the feature, which has given the best score and add it to the list of the currently *used* features**\n",
    "\n",
    "**- if the model's quality has increased since the last round, add this feature to the list of the best features and  print the best score and the current best features list**\n",
    "\n",
    "**Do not forget to implement the early stopping.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "650b17a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_steps(X, y, n_rounds, model):\n",
    "    \"\"\"\n",
    "    Parameters\n",
    "    ==========\n",
    "        X: ndarray - training inputs\n",
    "        y: ndarray - training targets\n",
    "        n_rounds: int - early stop when score doesn't increase n_rounds\n",
    "        model: sklearn model\n",
    "\n",
    "    Returns\n",
    "    =======\n",
    "        feat_best_list - list of features\n",
    "    \"\"\"\n",
    "    \n",
    "    feat_best_list = []\n",
    "\n",
    "    ### BEGIN Solution\n",
    "    rounds = 0\n",
    "    full_list = top20features.copy()\n",
    "    current_features = full_list.copy()\n",
    "    best_score = np.inf\n",
    "    while rounds < n_rounds:\n",
    "        rounds = rounds + 1\n",
    "        step = selection_step(model, X, y, current_features)\n",
    "        current_score = min(step)\n",
    "        best_features_round = step[current_score].copy()\n",
    "        full_list.remove(best_features_round[-1])\n",
    "        current_features = [best_features_round + [feature] for feature in full_list]\n",
    "        print(current_score, best_features_round)\n",
    "        \n",
    "        if best_score > current_score:\n",
    "            best_score = current_score\n",
    "            feat_best_list = best_features_round\n",
    "            rounds = 0        \n",
    "    ### END Solution\n",
    "    \n",
    "    return feat_best_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "geographic-holly",
   "metadata": {
    "id": "geographic-holly"
   },
   "source": [
    "**1.3.3) Use the function implemented above + DecisionTreeRegressor to get the best features according to this algorithm. (HINT: Use ```n_rounds = 2```)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "hazardous-retirement",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hazardous-retirement",
    "outputId": "81502f2d-f6fa-4497-d569-58605bfff430"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.523703552962209 ['full_sq']\n",
      "0.521719302350689 ['full_sq', 'state']\n",
      "0.5267535813750388 ['full_sq', 'state', 'school_education_centers_top_20_raion']\n",
      "0.5404714298715207 ['full_sq', 'state', 'school_education_centers_top_20_raion', 'num_room']\n",
      "Best features according to DecisionTreeRegressor : ['full_sq', 'state']\n"
     ]
    }
   ],
   "source": [
    "### BEGIN Solution\n",
    "dtree = DecisionTreeRegressor(random_state=seed)\n",
    "best_features = forward_steps(X=x_train, y=y_train_log, n_rounds=2, model=dtree)\n",
    "print(f'Best features according to {type(dtree).__name__} : {best_features}')\n",
    "### END Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "contrary-packet",
   "metadata": {
    "id": "contrary-packet"
   },
   "source": [
    "**1.3.4) Use Linear Regression, Ridge regression, Random forest and DecisionTree to get the RMSLE scores using these features. Remember the function you wrote earlier.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "graduate-tooth",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "id": "graduate-tooth",
    "outputId": "72812408-0c21-4229-ff2d-f29243966f3d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model</th>\n",
       "      <th>Train error</th>\n",
       "      <th>Test error</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LinearRegression</td>\n",
       "      <td>0.575545</td>\n",
       "      <td>0.575940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ridge</td>\n",
       "      <td>0.575545</td>\n",
       "      <td>0.575940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DecisionTreeRegressor</td>\n",
       "      <td>0.496464</td>\n",
       "      <td>0.530043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestRegressor</td>\n",
       "      <td>0.497104</td>\n",
       "      <td>0.527267</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Model  Train error  Test error\n",
       "0       LinearRegression     0.575545    0.575940\n",
       "1                  Ridge     0.575545    0.575940\n",
       "2  DecisionTreeRegressor     0.496464    0.530043\n",
       "3  RandomForestRegressor     0.497104    0.527267"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### BEGIN Solution\n",
    "train_models(x_train[best_features], y_train_log, x_test[best_features], y_test_log)\n",
    "### END Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "infinite-navigation",
   "metadata": {
    "id": "infinite-navigation"
   },
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "individual-rendering",
   "metadata": {
    "id": "individual-rendering"
   },
   "source": [
    "## Task 2. Multi-Class Classification Strategies (1 point)\n",
    "In this task, you will deal with the multiclass classification problem for the Glass Classification Data. Let's load the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "technical-trader",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "technical-trader",
    "outputId": "91043ffe-6127-441b-a6ae-58f111768d40"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RI</th>\n",
       "      <th>Na</th>\n",
       "      <th>Mg</th>\n",
       "      <th>Al</th>\n",
       "      <th>Si</th>\n",
       "      <th>K</th>\n",
       "      <th>Ca</th>\n",
       "      <th>Ba</th>\n",
       "      <th>Fe</th>\n",
       "      <th>Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>1.51646</td>\n",
       "      <td>13.41</td>\n",
       "      <td>3.55</td>\n",
       "      <td>1.25</td>\n",
       "      <td>72.81</td>\n",
       "      <td>0.68</td>\n",
       "      <td>8.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>1.52020</td>\n",
       "      <td>13.98</td>\n",
       "      <td>1.35</td>\n",
       "      <td>1.63</td>\n",
       "      <td>71.76</td>\n",
       "      <td>0.39</td>\n",
       "      <td>10.56</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>1.51829</td>\n",
       "      <td>13.24</td>\n",
       "      <td>3.90</td>\n",
       "      <td>1.41</td>\n",
       "      <td>72.33</td>\n",
       "      <td>0.55</td>\n",
       "      <td>8.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.10</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          RI     Na    Mg    Al     Si     K     Ca   Ba    Fe  Type\n",
       "82   1.51646  13.41  3.55  1.25  72.81  0.68   8.10  0.0  0.00     2\n",
       "129  1.52020  13.98  1.35  1.63  71.76  0.39  10.56  0.0  0.18     2\n",
       "116  1.51829  13.24  3.90  1.41  72.33  0.55   8.31  0.0  0.10     2"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('glass.csv')\n",
    "X, y = data.drop('Type', axis=1), data.Type\n",
    "data.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "flying-region",
   "metadata": {
    "id": "flying-region"
   },
   "source": [
    "The features of each glass object correspond to the fraction of the particular chemical element in the object. The target variable corresponds to the type of glass (6 classes).\n",
    "\n",
    "In this problem, you have to empirically compare the time complexity and performance of several multiclass labeling strategies for different algorithms. You must consider the following algorithms:\n",
    "* Single Decision Tree (depth 7)\n",
    "* Medium Random Forest (100 trees of depth 3)\n",
    "* KNearestNeighbors (5 neighbors)\n",
    "* Logistic Regression\n",
    "\n",
    "Note that all these algorithms by default support **multiclass labeling**. Nevertheless, we want you to compare this approach with **OneVSRest** and **OneVSOne** approaches applied to these algorithms. More precisely, for every pair (algorithm, approach) you are to perform a 5-fold cross-validation on the data and output the validation score and the computation time in the **table** form. Please note that you also have to choose the metric to optimize during CV (e.g. accuracy, balanced accuracy) on your own. \n",
    "\n",
    "After that, you are to answer the following questions:\n",
    "* Which metric did you choose to optimize during cross-validation and why? Explain\n",
    "* For which algorithms the usage of OneVSRest/OneVSOne approach provides significantly better performance without a significant increase in computation time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "authentic-slope",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 301
    },
    "id": "authentic-slope",
    "outputId": "5421c6dc-461c-4d7e-c472-aaf96785958a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cross Validation Scores\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DecisionTreeClassifier</th>\n",
       "      <th>RandomForestClassifier</th>\n",
       "      <th>KNeighborsClassifier</th>\n",
       "      <th>LogisticRegression</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Multiclas</th>\n",
       "      <td>0.554419</td>\n",
       "      <td>0.546522</td>\n",
       "      <td>0.577174</td>\n",
       "      <td>0.524475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OneVsRest</th>\n",
       "      <td>0.512538</td>\n",
       "      <td>0.563479</td>\n",
       "      <td>0.595097</td>\n",
       "      <td>0.497738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OneVsOne</th>\n",
       "      <td>0.555091</td>\n",
       "      <td>0.576689</td>\n",
       "      <td>0.591219</td>\n",
       "      <td>0.515765</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           DecisionTreeClassifier  RandomForestClassifier  \\\n",
       "Multiclas                0.554419                0.546522   \n",
       "OneVsRest                0.512538                0.563479   \n",
       "OneVsOne                 0.555091                0.576689   \n",
       "\n",
       "           KNeighborsClassifier  LogisticRegression  \n",
       "Multiclas              0.577174            0.524475  \n",
       "OneVsRest              0.595097            0.497738  \n",
       "OneVsOne               0.591219            0.515765  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Duration (CV=5)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DecisionTreeClassifier</th>\n",
       "      <th>RandomForestClassifier</th>\n",
       "      <th>KNeighborsClassifier</th>\n",
       "      <th>LogisticRegression</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Multiclas</th>\n",
       "      <td>0.026888</td>\n",
       "      <td>0.851484</td>\n",
       "      <td>0.074311</td>\n",
       "      <td>0.153918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OneVsRest</th>\n",
       "      <td>0.099292</td>\n",
       "      <td>1.990356</td>\n",
       "      <td>0.455355</td>\n",
       "      <td>0.274587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OneVsOne</th>\n",
       "      <td>0.145940</td>\n",
       "      <td>6.325500</td>\n",
       "      <td>1.547271</td>\n",
       "      <td>0.425850</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           DecisionTreeClassifier  RandomForestClassifier  \\\n",
       "Multiclas                0.026888                0.851484   \n",
       "OneVsRest                0.099292                1.990356   \n",
       "OneVsOne                 0.145940                6.325500   \n",
       "\n",
       "           KNeighborsClassifier  LogisticRegression  \n",
       "Multiclas              0.074311            0.153918  \n",
       "OneVsRest              0.455355            0.274587  \n",
       "OneVsOne               1.547271            0.425850  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier\n",
    "from sklearn.metrics import f1_score, accuracy_score\n",
    "from time import perf_counter\n",
    "\n",
    "# initialise multiclass, OVR, OVO classifiers\n",
    "multiclass_clfs = [DecisionTreeClassifier(max_depth=7, random_state=seed),\n",
    "                   RandomForestClassifier(n_estimators=100, max_depth=3, n_jobs=-1, random_state=seed),\n",
    "                   KNeighborsClassifier(n_jobs=-1, n_neighbors=5),\n",
    "                   LogisticRegression()]\n",
    "oneVsRest_clfs = [OneVsRestClassifier(clf, n_jobs=-1) for clf in multiclass_clfs]\n",
    "oneVsOne_clfs  = [OneVsOneClassifier(clf,  n_jobs=-1) for clf in multiclass_clfs]\n",
    "\n",
    "def measure_perfomance(clfs_list: list) -> tuple:\n",
    "    scores, timing = [], []\n",
    "    for clf in clfs_list:\n",
    "        start = perf_counter()\n",
    "        score = cross_val_score(clf, X, y, cv=5, scoring=make_scorer(f1_score, average='weighted')).mean()\n",
    "        end = perf_counter() - start\n",
    "        scores.append(score)\n",
    "        timing.append(end)\n",
    "    return scores, timing\n",
    "\n",
    "scores_MultL, timing_MultL = measure_perfomance(multiclass_clfs)\n",
    "scores_OVR, timing_OVR = measure_perfomance(oneVsRest_clfs)\n",
    "scores_OVO, timing_OVO = measure_perfomance(oneVsOne_clfs)\n",
    "\n",
    "\n",
    "# creating dataFrames\n",
    "model_names = [type(clf).__name__ for clf in multiclass_clfs]\n",
    "index_names = ['Multiclas', 'OneVsRest', 'OneVsOne']\n",
    "df_scores = pd.DataFrame((scores_MultL, scores_OVR, scores_OVO), columns=model_names, index=index_names)\n",
    "df_time = pd.DataFrame((timing_MultL, timing_OVR, timing_OVO), columns=model_names, index=index_names)\n",
    "\n",
    "print('Cross Validation Scores')\n",
    "display(df_scores)\n",
    "\n",
    "print('Training Duration (CV=5)')\n",
    "display(df_time)\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "married-preserve",
   "metadata": {
    "id": "married-preserve"
   },
   "source": [
    "**Which metric did you choose to optimize during cross validation and why? Explain**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dutch-alfred",
   "metadata": {
    "id": "dutch-alfred"
   },
   "source": [
    "**Your text answer (do not delete this comment)**:\n",
    "\n",
    "In this case, we are dealing with `unbalanced multiclass classification` most common class has 76 samples and the rare on only 9 samples. Also, based on the problem description, we may suspect that classes of glass represent its quality and consequently its market price. Therefore, it is essential to decrease the number of the `first type errors` (false positive) since it will lead to classifying less expensive glass as more expensive ones and businesses might be `sued for fraud` on another hand `second type errors` (false negative) might lead to the classification of high-quality glass as a cheap one and thus to `money losses`. Taking into account these two factors, we need to apply metrics that will minimize both precision and recall and will keep errors of the first and second type on a reasonable level. Such a metric is `F1 score`, which is a harmonic mean of precision and recall."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "elect-jerusalem",
   "metadata": {
    "id": "elect-jerusalem"
   },
   "source": [
    "**For which algorithms the usage of OneVSRest/OneVSOne approach provides significantly better performance without a significant increase in computation time?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "CH04Ord57RS4",
   "metadata": {
    "id": "CH04Ord57RS4"
   },
   "source": [
    "**Your text answer (do not delete this comment)**:\n",
    "\n",
    "For the chosen metrics F1 score, significantly better performance by OVR and OVO classifiers was achieved only in the case of RandomForest and KNN. Though, the computational time for OVR and OVO was also slightly larger 3-5 times in a series of runs for the Intel Core i5 CPU @ 1.70GHz.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "electric-matthew",
   "metadata": {
    "id": "electric-matthew"
   },
   "source": [
    "# Task 3. Stacking (2 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "third-liverpool",
   "metadata": {
    "id": "third-liverpool"
   },
   "source": [
    "Stacking is an ensembling approach, which combines several algorithms (base models) to get better results. This is achieved by **a)** training the base models (any ML models) on the initial train data (or parts of the train data, see the paragraph below), **b)** training the meta-model (also an ML model) on the predictions of those base models.\n",
    "\n",
    "One of the main problems of stacking is overfitting. To avoid it, when evaluating the performance of the meta-model on the train set, the training data is divided into $ n $ folds, $ (n-1) $ of which are used for training the base models. The $ n $ -th fold is used for the overall prediction (calculating the meta-factor) in the following manner: the predictions on the $ n $ -th fold are made by the base models (already trained on the rest of the dataset), then these predictions on the $ n $ -th fold are fed as inputs to the meta-model.\n",
    "\n",
    "When evaluating the performance of the meta-model on the test set, there can be used various approaches$.^{[1]}$ For example, in order, to obtain meta-factors for the test data, the base classifiers can be retrained on the entire training set, since the problem of overfitting does not arise here. In other words, if we want to calculate the factors for the test set, we can safely use the training set to train the base classifiers (once again, as it is said in the paragraph above, if we want to calculate factors for the training set, then it is necessary to ensure that the classifier does not predict for those objects on which it has been trained).  \n",
    "\n",
    "You can read more details about stacking [blockpost](https://blog.statsbot.co/ensemble-learning-d1dcd548e936), [kaggle ensemble guide](https://mlwave.com/kaggle-ensembling-guide/).\n",
    "\n",
    "P.S. Stacking and Blending are two similar approaches to combining classifiers (ensembling). The difference is that Stacking uses out-of-fold predictions for the train set, and Blending uses a validation set to train the next layer ([source](quora.com/What-are-examples-of-blending-and-stacking-in-Machine-Learning)). Because they are so similar you can use any of them. \n",
    "\n",
    "[1] http://www.machinelearning.ru/wiki/images/5/56/Guschin2015Stacking.pdf  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recognized-guest",
   "metadata": {
    "id": "recognized-guest"
   },
   "source": [
    "**Your task will be:**\n",
    "\n",
    "* Complete the ```meta_classfier``` function\n",
    "* Choose 6 different base models (base models can be any models that you know, and can differ between each other by different hyperparameters, models of ml, features e.t.c) and train them.\n",
    "* Report individual accuracy on the test set for each of the models.\n",
    "* Train the metaclassifier (in this case, once again, it is just an ML model) on the original dataset's features, report score on test.\n",
    "* Train the metaclassifier on the base models using $n$-fold cross-validation, as described above. Report the training score of the meta-classifier on the $ n $-th fold. Retrain the base models on the whole training set. Report the test score (accuracy) of the meta-classifier in this case.\n",
    "* Does stacking help to gain a better score?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incident-ceremony",
   "metadata": {
    "id": "incident-ceremony"
   },
   "source": [
    "For this task we will use the dataset fetch_covertype from [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_covtype.html#sklearn.datasets.fetch_covtype). Split it train-test - 60/40.\n",
    "More details about this dataset you can find [here](https://archive.ics.uci.edu/ml/datasets/Covertype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "banner-input",
   "metadata": {
    "id": "banner-input"
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "fc = datasets.fetch_covtype()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collected-flash",
   "metadata": {
    "id": "collected-flash"
   },
   "source": [
    "Write meta classifier function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "brilliant-greensboro",
   "metadata": {
    "id": "brilliant-greensboro"
   },
   "outputs": [],
   "source": [
    "def meta_classifier(base_clfs, final_classifier, X_train, X_test, y_train, cv=0):\n",
    "    \"\"\"\n",
    "    Meta classifier prediction using stacking. \n",
    "    Input:\n",
    "    :param base_clfs: list,  base classifiers which will be stacked together.\n",
    "    :param final_classifier: estimator, a classifier which will be used to combine the base estimators. \n",
    "    :param X_train: numpy array or pandas table, train set.\n",
    "    :param X_test: numpy array or pandas table, target for train set.\n",
    "    :param X_train: numpy array or pandas table, test set.\n",
    "    :param cv: number of cross-validation folds.\n",
    "    \n",
    "    Output:\n",
    "    :param y_pred: numpy array or pandas table, prediction of meta classifier using stacking on test set.\n",
    "    :param final_classifier(optional): estimator, trained final_calssifier.\n",
    "    \n",
    "    \n",
    "    More details https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html\n",
    "    \n",
    "    \"\"\"\n",
    "    ### BEGIN Solution (do not delete this comment)\n",
    "    if cv:\n",
    "        # this block used for evaluation of stacking on train set via n-1 folds training (cv > 0)\n",
    "        \n",
    "        # meta clf labels [n_samples X n_base_clfs] matrix\n",
    "        y_train_meta_cv = np.vstack([cross_val_predict(clf, X_train, y_train, cv=cv, n_jobs=-1) for clf in base_models]).T\n",
    "        \n",
    "        # final clf labels by cross validation [n_samples X 1] matrix\n",
    "        y_train_pred_cv = cross_val_predict(final_classifier, y_train_meta_cv, y_train)\n",
    "        final_classifier.fit(y_train_meta_cv, y_train)\n",
    "        return  y_train_pred_cv, final_classifier.fit(y_train_meta_cv, y_train)\n",
    "    \n",
    "    else:\n",
    "        # this block used for retraining on the entire train data and generating meta labels for test set. (cv == 0)\n",
    "        y_train_meta = np.vstack([clf.fit(X_train, y_train).predict(X_train) for clf in base_clfs]).T\n",
    "        y_test_meta  = np.vstack([clf.predict(X_test) for clf in base_clfs]).T\n",
    "        return y_test_meta, final_classifier.fit(y_train_meta, y_train)\n",
    "    \n",
    "    ### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vfZmdicwORxq",
   "metadata": {
    "id": "vfZmdicwORxq"
   },
   "source": [
    "### Subproblem 3.1 (1 point)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incredible-wichita",
   "metadata": {
    "id": "incredible-wichita"
   },
   "source": [
    "**3.1.1) Chose 6 different base models (base models can be any models that you know,and can differ with each other by different hyperparameters,  models of ml, features e.t.c) and train them.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "cleared-supply",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cleared-supply",
    "outputId": "6c3af77e-4cc3-41cf-85e1-c179b6332f02"
   },
   "outputs": [],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "from xgboost import XGBClassifier\n",
    "from sklearn.ensemble import ExtraTreesClassifier, RandomForestClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "seed = 1\n",
    "\n",
    "# train/test split 60/40\n",
    "X, y = fc.data, fc.target \n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4, random_state=seed)\n",
    "\n",
    "#scaling features for distance dependent models (KNN)\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)\n",
    "\n",
    "# base models initialization\n",
    "base_models = [\n",
    "    RandomForestClassifier(max_depth=4, n_jobs=-1, random_state=seed),\n",
    "    ExtraTreesClassifier(max_depth=8, random_state=seed),\n",
    "    XGBClassifier(max_depth=4, learning_rate=.01, random_state=seed, n_jobs=-1, eval_metric='mlogloss'),\n",
    "    DecisionTreeClassifier(max_depth=8, random_state=seed),\n",
    "    LogisticRegression(C=.1),\n",
    "    SGDClassifier(loss='modified_huber', random_state=seed)\n",
    "    ]\n",
    "\n",
    "# train base models\n",
    "for clf in base_models:\n",
    "    clf.fit(X_train, y_train)\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "streaming-crowd",
   "metadata": {
    "id": "streaming-crowd"
   },
   "source": [
    "**3.1.2) Report individual scores on test set. As a score use accuracy.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "standing-breakdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "standing-breakdown",
    "outputId": "34a2ebd2-6a1c-40c4-afec-d5cab639cd46"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Base models accuracy scores:\n",
      "1 RandomForestClassifier 0.672\n",
      "2 ExtraTreesClassifier 0.648\n",
      "3 XGBClassifier 0.722\n",
      "4 DecisionTreeClassifier 0.747\n",
      "5 LogisticRegression 0.724\n",
      "6 SGDClassifier 0.708\n"
     ]
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "\n",
    "# by default .score method return the mean accuracy\n",
    "print('Base models accuracy scores:')\n",
    "for i, clf in enumerate(base_models):\n",
    "    print(i + 1, type(clf).__name__, f'{clf.score(X_test, y_test):.3f}')\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8CsZdyGPOj6E",
   "metadata": {
    "id": "8CsZdyGPOj6E"
   },
   "source": [
    "### Subproblem 3.2 (1 point)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adolescent-street",
   "metadata": {
    "id": "adolescent-street"
   },
   "source": [
    "**3.2.1) Train metaclassifier (in this case, once again, it is just an ML model) on original dataset's features. And report score on test.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "lesbian-fields",
   "metadata": {
    "id": "lesbian-fields"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of meta CLF LogisticRegression : 0.725\n"
     ]
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "meta_clf = LogisticRegression()\n",
    "meta_clf.fit(X_train, y_train)\n",
    "test_accuracy = meta_clf.score(X_test, y_test)\n",
    "\n",
    "print(f'Accuracy of meta CLF {type(meta_clf).__name__} : {test_accuracy:.3f}')\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "horizontal-wallace",
   "metadata": {
    "id": "horizontal-wallace"
   },
   "source": [
    "**3.2.2) Train the metaclassifier on the base models using $n$-fold cross-validation, as described above. Report the training score of the meta-classifier on the $ n $-th fold. Retrain the base models on the whole training set. Report the test score (accuracy) of the meta-classifier in this case.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "renewable-concrete",
   "metadata": {
    "id": "renewable-concrete"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training score of meta clf in n-th fold: 0.712\n",
      "Test score usung full training set: 0.709\n"
     ]
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "y_train_meta_cv, final_classifier_cv = meta_classifier(base_models,\n",
    "                                                    meta_clf,\n",
    "                                                    X_train,\n",
    "                                                    X_test,\n",
    "                                                    y_train,\n",
    "                                                    cv=3)\n",
    "\n",
    "print(f'Training score of meta clf in n-th fold: {accuracy_score(y_train, y_train_meta_cv):.3f}')\n",
    "\n",
    "y_test_meta, final_classifier_cv = meta_classifier(base_models,\n",
    "                                                    meta_clf,\n",
    "                                                    X_train,\n",
    "                                                    X_test,\n",
    "                                                    y_train)\n",
    "\n",
    "y_test_pred = final_classifier_cv.predict(y_test_meta)\n",
    "print(f'Test score usung full training set: {accuracy_score(y_test, y_test_pred):.3f}')\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "printable-premium",
   "metadata": {
    "id": "printable-premium"
   },
   "source": [
    "**3.2.3) Report score(accuracy) on test.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "auburn-easter",
   "metadata": {
    "id": "auburn-easter"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.709\n"
     ]
    }
   ],
   "source": [
    "### BEGIN Solution (do not delete this comment)\n",
    "\n",
    "# same with previous task 3.2.1 as it was mentiond in telegram chat\n",
    "print(f'Test score: {accuracy_score(y_test, y_test_pred):.3f}')\n",
    "\n",
    "### END Solution (do not delete this comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "premier-scout",
   "metadata": {
    "id": "premier-scout"
   },
   "source": [
    "**3.2.4) Does stacking helped to gain a better score?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vSV5hJFR7RS-",
   "metadata": {
    "id": "vSV5hJFR7RS-"
   },
   "source": [
    "**Your text answer (do not delete this comment)**:\n",
    "\n",
    "In this case, stacking did not help to get a better score. LogisticRegression itself shows better score.\n",
    "The following improvements might be done:\n",
    "1. Optimization of the meta classifier and base models hyperparameters.\n",
    "2. Use more diversified alghoritms (now used 4 tree-based and 2 linear models). KNN were excluded due to slow inference.\n",
    "3. Optimize contributions of the models in the final classifier with respect to their scores.\n",
    "4. Determine an optimal features subspace for each model."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "ML_HW_2.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}