NLP Recommendation System.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP Recomendation System - IMDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook was based on Emma Grimaldi tutorial \n",
    "(https://towardsdatascience.com/how-to-build-from-scratch-a-content-based-movie-recommender-with-natural-language-processing-25ad400eb243)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an example of content-based movie recommender system. This recommender system has been created using the python Natural Language Processing library. In content based reccommendation systems the algorithm makes decisions just basing on our interests. The algorighm looks at what we like and suggest us similar content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Title</th>\n",
       "      <th>Year</th>\n",
       "      <th>Rated</th>\n",
       "      <th>Released</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Genre</th>\n",
       "      <th>Director</th>\n",
       "      <th>Writer</th>\n",
       "      <th>Actors</th>\n",
       "      <th>Plot</th>\n",
       "      <th>Language</th>\n",
       "      <th>Country</th>\n",
       "      <th>Awards</th>\n",
       "      <th>Poster</th>\n",
       "      <th>Ratings.Source</th>\n",
       "      <th>Ratings.Value</th>\n",
       "      <th>Metascore</th>\n",
       "      <th>imdbRating</th>\n",
       "      <th>imdbVotes</th>\n",
       "      <th>imdbID</th>\n",
       "      <th>Type</th>\n",
       "      <th>tomatoMeter</th>\n",
       "      <th>tomatoImage</th>\n",
       "      <th>tomatoRating</th>\n",
       "      <th>tomatoReviews</th>\n",
       "      <th>tomatoFresh</th>\n",
       "      <th>tomatoRotten</th>\n",
       "      <th>tomatoConsensus</th>\n",
       "      <th>tomatoUserMeter</th>\n",
       "      <th>tomatoUserRating</th>\n",
       "      <th>tomatoUserReviews</th>\n",
       "      <th>tomatoURL</th>\n",
       "      <th>DVD</th>\n",
       "      <th>BoxOffice</th>\n",
       "      <th>Production</th>\n",
       "      <th>Website</th>\n",
       "      <th>Response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>1994</td>\n",
       "      <td>R</td>\n",
       "      <td>14 Oct 1994</td>\n",
       "      <td>142 min</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>Frank Darabont</td>\n",
       "      <td>Stephen King (short story \"Rita Hayworth and S...</td>\n",
       "      <td>Tim Robbins, Morgan Freeman, Bob Gunton, Willi...</td>\n",
       "      <td>Two imprisoned men bond over a number of years...</td>\n",
       "      <td>English</td>\n",
       "      <td>USA</td>\n",
       "      <td>Nominated for 7 Oscars. Another 19 wins &amp; 30 n...</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>Internet Movie Database</td>\n",
       "      <td>9.3/10</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9.3</td>\n",
       "      <td>1,825,626</td>\n",
       "      <td>tt0111161</td>\n",
       "      <td>movie</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://www.rottentomatoes.com/m/shawshank_rede...</td>\n",
       "      <td>27 Jan 1998</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Columbia Pictures</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>The Godfather</td>\n",
       "      <td>1972</td>\n",
       "      <td>R</td>\n",
       "      <td>24 Mar 1972</td>\n",
       "      <td>175 min</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>Francis Ford Coppola</td>\n",
       "      <td>Mario Puzo (screenplay), Francis Ford Coppola ...</td>\n",
       "      <td>Marlon Brando, Al Pacino, James Caan, Richard ...</td>\n",
       "      <td>The aging patriarch of an organized crime dyna...</td>\n",
       "      <td>English, Italian, Latin</td>\n",
       "      <td>USA</td>\n",
       "      <td>Won 3 Oscars. Another 23 wins &amp; 27 nominations.</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>Internet Movie Database</td>\n",
       "      <td>9.2/10</td>\n",
       "      <td>100.0</td>\n",
       "      <td>9.2</td>\n",
       "      <td>1,243,444</td>\n",
       "      <td>tt0068646</td>\n",
       "      <td>movie</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://www.rottentomatoes.com/m/godfather/</td>\n",
       "      <td>09 Oct 2001</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Paramount Pictures</td>\n",
       "      <td>http://www.thegodfather.com</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>The Godfather: Part II</td>\n",
       "      <td>1974</td>\n",
       "      <td>R</td>\n",
       "      <td>20 Dec 1974</td>\n",
       "      <td>202 min</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>Francis Ford Coppola</td>\n",
       "      <td>Francis Ford Coppola (screenplay), Mario Puzo ...</td>\n",
       "      <td>Al Pacino, Robert Duvall, Diane Keaton, Robert...</td>\n",
       "      <td>The early life and career of Vito Corleone in ...</td>\n",
       "      <td>English, Italian, Spanish, Latin, Sicilian</td>\n",
       "      <td>USA</td>\n",
       "      <td>Won 6 Oscars. Another 10 wins &amp; 20 nominations.</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>Internet Movie Database</td>\n",
       "      <td>9.0/10</td>\n",
       "      <td>85.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>856,870</td>\n",
       "      <td>tt0071562</td>\n",
       "      <td>movie</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://www.rottentomatoes.com/m/godfather_part...</td>\n",
       "      <td>24 May 2005</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Paramount Pictures</td>\n",
       "      <td>http://www.thegodfather.com/</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>2008</td>\n",
       "      <td>PG-13</td>\n",
       "      <td>18 Jul 2008</td>\n",
       "      <td>152 min</td>\n",
       "      <td>Action, Crime, Drama</td>\n",
       "      <td>Christopher Nolan</td>\n",
       "      <td>Jonathan Nolan (screenplay), Christopher Nolan...</td>\n",
       "      <td>Christian Bale, Heath Ledger, Aaron Eckhart, M...</td>\n",
       "      <td>When the menace known as the Joker emerges fro...</td>\n",
       "      <td>English, Mandarin</td>\n",
       "      <td>USA, UK</td>\n",
       "      <td>Won 2 Oscars. Another 151 wins &amp; 153 nominations.</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>Internet Movie Database</td>\n",
       "      <td>9.0/10</td>\n",
       "      <td>82.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1,802,351</td>\n",
       "      <td>tt0468569</td>\n",
       "      <td>movie</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://www.rottentomatoes.com/m/the_dark_knight/</td>\n",
       "      <td>09 Dec 2008</td>\n",
       "      <td>$533,316,061</td>\n",
       "      <td>Warner Bros. Pictures/Legendary</td>\n",
       "      <td>http://thedarkknight.warnerbros.com/</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>12 Angry Men</td>\n",
       "      <td>1957</td>\n",
       "      <td>APPROVED</td>\n",
       "      <td>01 Apr 1957</td>\n",
       "      <td>96 min</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>Sidney Lumet</td>\n",
       "      <td>Reginald Rose (story), Reginald Rose (screenplay)</td>\n",
       "      <td>Martin Balsam, John Fiedler, Lee J. Cobb, E.G....</td>\n",
       "      <td>A jury holdout attempts to prevent a miscarria...</td>\n",
       "      <td>English</td>\n",
       "      <td>USA</td>\n",
       "      <td>Nominated for 3 Oscars. Another 16 wins &amp; 8 no...</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>Internet Movie Database</td>\n",
       "      <td>8.9/10</td>\n",
       "      <td>96.0</td>\n",
       "      <td>8.9</td>\n",
       "      <td>494,215</td>\n",
       "      <td>tt0050083</td>\n",
       "      <td>movie</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://www.rottentomatoes.com/m/1000013-12_ang...</td>\n",
       "      <td>06 Mar 2001</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Criterion Collection</td>\n",
       "      <td>http://www.criterion.com/films/27871-12-angry-men</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                     Title  Year     Rated     Released  Runtime  \\\n",
       "0           1  The Shawshank Redemption  1994         R  14 Oct 1994  142 min   \n",
       "1           2             The Godfather  1972         R  24 Mar 1972  175 min   \n",
       "2           3    The Godfather: Part II  1974         R  20 Dec 1974  202 min   \n",
       "3           4           The Dark Knight  2008     PG-13  18 Jul 2008  152 min   \n",
       "4           5              12 Angry Men  1957  APPROVED  01 Apr 1957   96 min   \n",
       "\n",
       "                  Genre              Director  \\\n",
       "0          Crime, Drama        Frank Darabont   \n",
       "1          Crime, Drama  Francis Ford Coppola   \n",
       "2          Crime, Drama  Francis Ford Coppola   \n",
       "3  Action, Crime, Drama     Christopher Nolan   \n",
       "4          Crime, Drama          Sidney Lumet   \n",
       "\n",
       "                                              Writer  \\\n",
       "0  Stephen King (short story \"Rita Hayworth and S...   \n",
       "1  Mario Puzo (screenplay), Francis Ford Coppola ...   \n",
       "2  Francis Ford Coppola (screenplay), Mario Puzo ...   \n",
       "3  Jonathan Nolan (screenplay), Christopher Nolan...   \n",
       "4  Reginald Rose (story), Reginald Rose (screenplay)   \n",
       "\n",
       "                                              Actors  \\\n",
       "0  Tim Robbins, Morgan Freeman, Bob Gunton, Willi...   \n",
       "1  Marlon Brando, Al Pacino, James Caan, Richard ...   \n",
       "2  Al Pacino, Robert Duvall, Diane Keaton, Robert...   \n",
       "3  Christian Bale, Heath Ledger, Aaron Eckhart, M...   \n",
       "4  Martin Balsam, John Fiedler, Lee J. Cobb, E.G....   \n",
       "\n",
       "                                                Plot  \\\n",
       "0  Two imprisoned men bond over a number of years...   \n",
       "1  The aging patriarch of an organized crime dyna...   \n",
       "2  The early life and career of Vito Corleone in ...   \n",
       "3  When the menace known as the Joker emerges fro...   \n",
       "4  A jury holdout attempts to prevent a miscarria...   \n",
       "\n",
       "                                     Language  Country  \\\n",
       "0                                     English      USA   \n",
       "1                     English, Italian, Latin      USA   \n",
       "2  English, Italian, Spanish, Latin, Sicilian      USA   \n",
       "3                           English, Mandarin  USA, UK   \n",
       "4                                     English      USA   \n",
       "\n",
       "                                              Awards  \\\n",
       "0  Nominated for 7 Oscars. Another 19 wins & 30 n...   \n",
       "1    Won 3 Oscars. Another 23 wins & 27 nominations.   \n",
       "2    Won 6 Oscars. Another 10 wins & 20 nominations.   \n",
       "3  Won 2 Oscars. Another 151 wins & 153 nominations.   \n",
       "4  Nominated for 3 Oscars. Another 16 wins & 8 no...   \n",
       "\n",
       "                                              Poster           Ratings.Source  \\\n",
       "0  https://images-na.ssl-images-amazon.com/images...  Internet Movie Database   \n",
       "1  https://images-na.ssl-images-amazon.com/images...  Internet Movie Database   \n",
       "2  https://images-na.ssl-images-amazon.com/images...  Internet Movie Database   \n",
       "3  https://images-na.ssl-images-amazon.com/images...  Internet Movie Database   \n",
       "4  https://images-na.ssl-images-amazon.com/images...  Internet Movie Database   \n",
       "\n",
       "  Ratings.Value  Metascore  imdbRating  imdbVotes     imdbID   Type  \\\n",
       "0        9.3/10       80.0         9.3  1,825,626  tt0111161  movie   \n",
       "1        9.2/10      100.0         9.2  1,243,444  tt0068646  movie   \n",
       "2        9.0/10       85.0         9.0    856,870  tt0071562  movie   \n",
       "3        9.0/10       82.0         9.0  1,802,351  tt0468569  movie   \n",
       "4        8.9/10       96.0         8.9    494,215  tt0050083  movie   \n",
       "\n",
       "   tomatoMeter  tomatoImage  tomatoRating  tomatoReviews  tomatoFresh  \\\n",
       "0          NaN          NaN           NaN            NaN          NaN   \n",
       "1          NaN          NaN           NaN            NaN          NaN   \n",
       "2          NaN          NaN           NaN            NaN          NaN   \n",
       "3          NaN          NaN           NaN            NaN          NaN   \n",
       "4          NaN          NaN           NaN            NaN          NaN   \n",
       "\n",
       "   tomatoRotten  tomatoConsensus  tomatoUserMeter  tomatoUserRating  \\\n",
       "0           NaN              NaN              NaN               NaN   \n",
       "1           NaN              NaN              NaN               NaN   \n",
       "2           NaN              NaN              NaN               NaN   \n",
       "3           NaN              NaN              NaN               NaN   \n",
       "4           NaN              NaN              NaN               NaN   \n",
       "\n",
       "   tomatoUserReviews                                          tomatoURL  \\\n",
       "0                NaN  http://www.rottentomatoes.com/m/shawshank_rede...   \n",
       "1                NaN         http://www.rottentomatoes.com/m/godfather/   \n",
       "2                NaN  http://www.rottentomatoes.com/m/godfather_part...   \n",
       "3                NaN   http://www.rottentomatoes.com/m/the_dark_knight/   \n",
       "4                NaN  http://www.rottentomatoes.com/m/1000013-12_ang...   \n",
       "\n",
       "           DVD     BoxOffice                       Production  \\\n",
       "0  27 Jan 1998           NaN                Columbia Pictures   \n",
       "1  09 Oct 2001           NaN               Paramount Pictures   \n",
       "2  24 May 2005           NaN               Paramount Pictures   \n",
       "3  09 Dec 2008  $533,316,061  Warner Bros. Pictures/Legendary   \n",
       "4  06 Mar 2001           NaN             Criterion Collection   \n",
       "\n",
       "                                             Website  Response  \n",
       "0                                                NaN      True  \n",
       "1                        http://www.thegodfather.com      True  \n",
       "2                       http://www.thegodfather.com/      True  \n",
       "3               http://thedarkknight.warnerbros.com/      True  \n",
       "4  http://www.criterion.com/films/27871-12-angry-men      True  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from rake_nltk import Rake\n",
    "import numpy as np\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "pd.set_option('display.max_columns', 50)\n",
    "df = pd.read_csv('https://query.data.world/s/uikepcpffyo2nhig52xxeevdialfl7')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Title</th>\n",
       "      <th>Genre</th>\n",
       "      <th>Director</th>\n",
       "      <th>Actors</th>\n",
       "      <th>Plot</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>Frank Darabont</td>\n",
       "      <td>Tim Robbins, Morgan Freeman, Bob Gunton, Willi...</td>\n",
       "      <td>Two imprisoned men bond over a number of years...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Godfather</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>Francis Ford Coppola</td>\n",
       "      <td>Marlon Brando, Al Pacino, James Caan, Richard ...</td>\n",
       "      <td>The aging patriarch of an organized crime dyna...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The Godfather: Part II</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>Francis Ford Coppola</td>\n",
       "      <td>Al Pacino, Robert Duvall, Diane Keaton, Robert...</td>\n",
       "      <td>The early life and career of Vito Corleone in ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>Action, Crime, Drama</td>\n",
       "      <td>Christopher Nolan</td>\n",
       "      <td>Christian Bale, Heath Ledger, Aaron Eckhart, M...</td>\n",
       "      <td>When the menace known as the Joker emerges fro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12 Angry Men</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>Sidney Lumet</td>\n",
       "      <td>Martin Balsam, John Fiedler, Lee J. Cobb, E.G....</td>\n",
       "      <td>A jury holdout attempts to prevent a miscarria...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      Title                 Genre              Director  \\\n",
       "0  The Shawshank Redemption          Crime, Drama        Frank Darabont   \n",
       "1             The Godfather          Crime, Drama  Francis Ford Coppola   \n",
       "2    The Godfather: Part II          Crime, Drama  Francis Ford Coppola   \n",
       "3           The Dark Knight  Action, Crime, Drama     Christopher Nolan   \n",
       "4              12 Angry Men          Crime, Drama          Sidney Lumet   \n",
       "\n",
       "                                              Actors  \\\n",
       "0  Tim Robbins, Morgan Freeman, Bob Gunton, Willi...   \n",
       "1  Marlon Brando, Al Pacino, James Caan, Richard ...   \n",
       "2  Al Pacino, Robert Duvall, Diane Keaton, Robert...   \n",
       "3  Christian Bale, Heath Ledger, Aaron Eckhart, M...   \n",
       "4  Martin Balsam, John Fiedler, Lee J. Cobb, E.G....   \n",
       "\n",
       "                                                Plot  \n",
       "0  Two imprisoned men bond over a number of years...  \n",
       "1  The aging patriarch of an organized crime dyna...  \n",
       "2  The early life and career of Vito Corleone in ...  \n",
       "3  When the menace known as the Joker emerges fro...  \n",
       "4  A jury holdout attempts to prevent a miscarria...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df[['Title','Genre','Director','Actors','Plot']]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# discarding the commas between the actors' full names and getting only the first three names\n",
    "df['Actors'] = df['Actors'].map(lambda x: x.split(',')[:3])\n",
    "\n",
    "# putting the genres in a list of words\n",
    "df['Genre'] = df['Genre'].map(lambda x: x.lower().split(','))\n",
    "\n",
    "df['Director'] = df['Director'].map(lambda x: x.split(' '))\n",
    "\n",
    "# merging together first and last name for each actor and director, so it's considered as one word \n",
    "# and there is no mix up between people sharing a first name\n",
    "for index, row in df.iterrows():\n",
    "    row['Actors'] = [x.lower().replace(' ','') for x in row['Actors']]\n",
    "    row['Director'] = ''.join(row['Director']).lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initializing the new column\n",
    "df['Key_words'] = \"\"\n",
    "\n",
    "for index, row in df.iterrows():\n",
    "    plot = row['Plot']\n",
    "    \n",
    "    # instantiating Rake, by default is uses english stopwords from NLTK\n",
    "    # and discard all puntuation characters\n",
    "    r = Rake()\n",
    "\n",
    "    # extracting the words by passing the text\n",
    "    r.extract_keywords_from_text(plot)\n",
    "\n",
    "    # getting the dictionary whith key words and their scores\n",
    "    key_words_dict_scores = r.get_word_degrees()\n",
    "    \n",
    "    # assigning the key words to the new column\n",
    "    row['Key_words'] = list(key_words_dict_scores.keys())\n",
    "\n",
    "# dropping the Plot column\n",
    "df.drop(columns = ['Plot'], inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.set_index('Title', inplace = True)\n",
    "df['bag_of_words'] = ''\n",
    "columns = df.columns\n",
    "for index, row in df.iterrows():\n",
    "    words = ''\n",
    "    for col in columns:\n",
    "        if col != 'Director':\n",
    "            words = words + ' '.join(row[col])+ ' '\n",
    "        else:\n",
    "            words = words + row[col]+ ' '\n",
    "    row['bag_of_words'] = words\n",
    "    \n",
    "df.drop(columns = [col for col in df.columns if col!= 'bag_of_words'], inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bag_of_words</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Title</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>The Shawshank Redemption</th>\n",
       "      <td>crime  drama frankdarabont timrobbins morganfr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The Godfather</th>\n",
       "      <td>crime  drama francisfordcoppola marlonbrando a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The Godfather: Part II</th>\n",
       "      <td>crime  drama francisfordcoppola alpacino rober...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The Dark Knight</th>\n",
       "      <td>action  crime  drama christophernolan christia...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12 Angry Men</th>\n",
       "      <td>crime  drama sidneylumet martinbalsam johnfied...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                               bag_of_words\n",
       "Title                                                                      \n",
       "The Shawshank Redemption  crime  drama frankdarabont timrobbins morganfr...\n",
       "The Godfather             crime  drama francisfordcoppola marlonbrando a...\n",
       "The Godfather: Part II    crime  drama francisfordcoppola alpacino rober...\n",
       "The Dark Knight           action  crime  drama christophernolan christia...\n",
       "12 Angry Men              crime  drama sidneylumet martinbalsam johnfied..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    The Shawshank Redemption\n",
       "1               The Godfather\n",
       "2      The Godfather: Part II\n",
       "3             The Dark Knight\n",
       "4                12 Angry Men\n",
       "Name: Title, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiating and generating the count matrix\n",
    "count = CountVectorizer()\n",
    "count_matrix = count.fit_transform(df['bag_of_words'])\n",
    "\n",
    "# creating a Series for the movie titles so they are associated to an ordered numerical\n",
    "# list I will use later to match the indexes\n",
    "indices = pd.Series(df.index)\n",
    "indices[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 0.15789474, 0.13764944, ..., 0.05263158, 0.05263158,\n",
       "        0.05564149],\n",
       "       [0.15789474, 1.        , 0.36706517, ..., 0.05263158, 0.05263158,\n",
       "        0.05564149],\n",
       "       [0.13764944, 0.36706517, 1.        , ..., 0.04588315, 0.04588315,\n",
       "        0.04850713],\n",
       "       ...,\n",
       "       [0.05263158, 0.05263158, 0.04588315, ..., 1.        , 0.05263158,\n",
       "        0.05564149],\n",
       "       [0.05263158, 0.05263158, 0.04588315, ..., 0.05263158, 1.        ,\n",
       "        0.05564149],\n",
       "       [0.05564149, 0.05564149, 0.04850713, ..., 0.05564149, 0.05564149,\n",
       "        1.        ]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generating the cosine similarity matrix\n",
    "cosine_sim = cosine_similarity(count_matrix, count_matrix)\n",
    "cosine_sim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function that takes in movie title as input and returns the top 10 recommended movies\n",
    "def recommendations(title):\n",
    "    \n",
    "    recommended_movies = []\n",
    "    \n",
    "    # gettin the index of the movie that matches the title\n",
    "    try:\n",
    "        idx = indices[indices == title].index[0]\n",
    "    except:\n",
    "        return print(\"This movie in not registred in our database\")\n",
    "    # creating a Series with the similarity scores in descending order\n",
    "    score_series = pd.Series(cosine_sim[idx]).sort_values(ascending = False)\n",
    "    #print(score_series)\n",
    "    \n",
    "    # getting the indexes of the 10 most similar movies\n",
    "    top_10_indexes = list(score_series.iloc[1:11].index)\n",
    "    \n",
    "    # populating the list with the titles of the best 10 matching movies\n",
    "    for i in top_10_indexes:\n",
    "        recommended_movies.append(list(df.index)[i])\n",
    "        \n",
    "    return recommended_movies, score_series[1:11]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Reccomendations</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The Godfather: Part II</td>\n",
       "      <td>0.367065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>Scarface</td>\n",
       "      <td>0.229416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>Fargo</td>\n",
       "      <td>0.220755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>226</th>\n",
       "      <td>Rope</td>\n",
       "      <td>0.220755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>On the Waterfront</td>\n",
       "      <td>0.216295</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Reccomendations     Score\n",
       "2    The Godfather: Part II  0.367065\n",
       "83                 Scarface  0.229416\n",
       "128                   Fargo  0.220755\n",
       "226                    Rope  0.220755\n",
       "100       On the Waterfront  0.216295"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "un, du = recommendations('The Godfather')\n",
    "d = {'Reccomendations': un, 'Score': du}\n",
    "df2 = pd.DataFrame(data=d)\n",
    "df2.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}