Smart Recommender (5-12) - TuriCreate + Generic Combo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Smart Recommender using Turi Create\n",
    "https://apple.github.io/turicreate/docs/userguide/\n",
    "https://apple.github.io/turicreate/docs/api/generated/turicreate.recommender.item_similarity_recommender.create.html#turicreate.recommender.item_similarity_recommender.create\n",
    "\n",
    "The goal of this notebook is to return 5 relevant, personalized club recommendations for each user. \n",
    "\n",
    "This recommendation system uses the Turi Create Item Similarity Algorithm to perform collaborative filtering on a simulated list of userID's (simulated favorites, interests). The simulated data points can be replaced with real user data once collected (the datapoints we need are **UserID (integer), Favorites (a list of club name strings),and Interests (a list of tag numbers**). \n",
    "\n",
    "Once recommendations from item similarity are returned, the goal is then to ensure that recommendations are attuned to user interests (and not just random due to item similarity)\n",
    "- The script filters out club recommendations that do not have tags that match their interest tags. This will get rid of a decent percentage of the item similarity recommendations (at least in ths simulated data, since it is completely random)\n",
    "- Then, all missing recommendations (since we want 5 per user) will be filled with recommendations using our generic recommender based on favorited clubs\n",
    "- If the user has no favorited clubs then missing recommendations will be filled with clubs from users' specific interests\n",
    "- If a user has no interests, then the recommender will recommend all item similarity recommendations (kind of random LOL).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Stuff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: dnspython in /home/tejashah88/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages (2.0.0)\r\n"
     ]
    }
   ],
   "source": [
    "# import packages\n",
    "# Native imports\n",
    "import os\n",
    "!pip install dnspython\n",
    "import json\n",
    "\n",
    "# MongoDB Python driver import\n",
    "import pymongo\n",
    "\n",
    "# 3rd-party data science imports\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "# pd.set_option('display.max_rows', None)\n",
    "\n",
    "\n",
    "# NLP Pre-processing tools\n",
    "import re, string, unicodedata\n",
    "import nltk\n",
    "#import contractions      # https://github.com/kootenpv/contractions\n",
    "#import inflect           # https://github.com/jaraco/inflect\n",
    "from nltk import word_tokenize, sent_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import LancasterStemmer, WordNetLemmatizer\n",
    "#import flashtext         # https://github.com/vi3k6i5/flashtext\n",
    "\n",
    "# Main NLP tool kit\n",
    "import gensim           # https://radimrehurek.com/gensim/auto_examples/index.html\n",
    "\n",
    "# Model training\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "import bs4 as bs\n",
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize # tokenizes sentences\n",
    "import re\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk.tag import pos_tag\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.corpus import wordnet\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[\"figure.figsize\"] = (12, 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: python-dotenv in /home/tejashah88/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages (0.15.0)\r\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!pip install python-dotenv\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to MongoDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pymongo[srv] in /home/tejashah88/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages (3.11.2)\n",
      "Requirement already satisfied: dnspython<2.0.0,>=1.16.0 in /home/tejashah88/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages (from pymongo[srv]) (1.16.0)\n"
     ]
    },
    {
     "ename": "ConfigurationError",
     "evalue": "The DNS operation timed out after 21.15655255317688 seconds",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTimeout\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/pymongo/srv_resolver.py\u001b[0m in \u001b[0;36m_resolve_uri\u001b[0;34m(self, encapsulate_errors)\u001b[0m\n\u001b[1;32m     72\u001b[0m             results = resolver.query('_mongodb._tcp.' + self.__fqdn, 'SRV',\n\u001b[0;32m---> 73\u001b[0;31m                                      lifetime=self.__connect_timeout)\n\u001b[0m\u001b[1;32m     74\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/dns/resolver.py\u001b[0m in \u001b[0;36mquery\u001b[0;34m(qname, rdtype, rdclass, tcp, source, raise_on_no_answer, source_port, lifetime)\u001b[0m\n\u001b[1;32m   1222\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1223\u001b[0;31m         \u001b[0;31m# Is it a port literal?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1224\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mservice\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/dns/resolver.py\u001b[0m in \u001b[0;36mresolve\u001b[0;34m(qname, rdtype, rdclass, tcp, source, raise_on_no_answer, source_port, lifetime, search)\u001b[0m\n\u001b[1;32m   1206\u001b[0m                         \u001b[0;32mif\u001b[0m \u001b[0mv6\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrrset\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1207\u001b[0;31m                             \u001b[0;32mfor\u001b[0m \u001b[0mrdata\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mv6\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrrset\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1208\u001b[0m                                 \u001b[0mv6addrs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maddress\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/dns/resolver.py\u001b[0m in \u001b[0;36mresolve\u001b[0;34m(self, qname, rdtype, rdclass, tcp, source, raise_on_no_answer, source_port, lifetime, search)\u001b[0m\n\u001b[1;32m   1042\u001b[0m         \u001b[0mequivalent\u001b[0m \u001b[0mto\u001b[0m \u001b[0mspecifying\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0;34m\"use EDNS0\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1043\u001b[0;31m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1044\u001b[0m         \u001b[0;34m*\u001b[0m\u001b[0mednsflags\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0man\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mint\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mEDNS\u001b[0m \u001b[0mflag\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/dns/resolver.py\u001b[0m in \u001b[0;36m_compute_timeout\u001b[0;34m(self, start, lifetime)\u001b[0m\n\u001b[1;32m    949\u001b[0m                         \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 950\u001b[0;31m                         \u001b[0;32mcontinue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    951\u001b[0m                     \u001b[0;32mexcept\u001b[0m \u001b[0mEOFError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mex\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTimeout\u001b[0m: The DNS operation timed out after 21.15655255317688 seconds",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mConfigurationError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-e66ff4e75036>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msystem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'pip install pymongo[srv]'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mconn_string\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetenv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'MONGO_URI_PROD'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpymongo\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mMongoClient\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconn_string\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, host, port, document_class, tz_aware, connect, type_registry, **kwargs)\u001b[0m\n\u001b[1;32m    639\u001b[0m                 res = uri_parser.parse_uri(\n\u001b[1;32m    640\u001b[0m                     \u001b[0mentity\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mport\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidate\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwarn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnormalize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 641\u001b[0;31m                     connect_timeout=timeout)\n\u001b[0m\u001b[1;32m    642\u001b[0m                 \u001b[0mseeds\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"nodelist\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    643\u001b[0m                 \u001b[0musername\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"username\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0musername\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/pymongo/uri_parser.py\u001b[0m in \u001b[0;36mparse_uri\u001b[0;34m(uri, default_port, validate, warn, normalize, connect_timeout)\u001b[0m\n\u001b[1;32m    498\u001b[0m         \u001b[0mconnect_timeout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconnect_timeout\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0moptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"connectTimeoutMS\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    499\u001b[0m         \u001b[0mdns_resolver\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_SrvResolver\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfqdn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconnect_timeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mconnect_timeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 500\u001b[0;31m         \u001b[0mnodes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdns_resolver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_hosts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    501\u001b[0m         \u001b[0mdns_options\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdns_resolver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_options\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    502\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mdns_options\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/pymongo/srv_resolver.py\u001b[0m in \u001b[0;36mget_hosts\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget_hosts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 102\u001b[0;31m         \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnodes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_srv_response_and_hosts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    103\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mnodes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/pymongo/srv_resolver.py\u001b[0m in \u001b[0;36m_get_srv_response_and_hosts\u001b[0;34m(self, encapsulate_errors)\u001b[0m\n\u001b[1;32m     81\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     82\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_get_srv_response_and_hosts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencapsulate_errors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 83\u001b[0;31m         \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_resolve_uri\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mencapsulate_errors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     84\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     85\u001b[0m         \u001b[0;31m# Construct address tuples\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/extra-software/anaconda3/envs/sc-notebook/lib/python3.6/site-packages/pymongo/srv_resolver.py\u001b[0m in \u001b[0;36m_resolve_uri\u001b[0;34m(self, encapsulate_errors)\u001b[0m\n\u001b[1;32m     77\u001b[0m                 \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     78\u001b[0m             \u001b[0;31m# Else, raise all errors as ConfigurationError.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 79\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mConfigurationError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     80\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresults\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     81\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mConfigurationError\u001b[0m: The DNS operation timed out after 21.15655255317688 seconds"
     ]
    }
   ],
   "source": [
    "!pip install pymongo[srv]\n",
    "conn_string = os.getenv('MONGO_URI_PROD')\n",
    "client = pymongo.MongoClient(conn_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ALL_CLUB_TAGS = []\n",
    "\n",
    "# Fetch list of tags\n",
    "for tag in client['production-db']['tag'].find():\n",
    "    ALL_CLUB_TAGS += [{\n",
    "        'id': tag['_id'],\n",
    "        'name': tag['name']\n",
    "    }]\n",
    "\n",
    "all_club_tags_df = pd.DataFrame(ALL_CLUB_TAGS)\n",
    "all_club_tags_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "club_info_db = []\n",
    "\n",
    "for user in client['production-db']['new_base_user'].find({'role': 'officer'}):\n",
    "    club_name = user['club']['name'].strip()\n",
    "    club_link_name = user['club']['link_name'].strip()\n",
    "    club_description = user['club']['about_us'].strip()\n",
    "    club_tags = user['club']['tags']\n",
    "    \n",
    "    club_info_db += [{\n",
    "        'name': club_name,\n",
    "        'link_name': club_link_name,\n",
    "        'description': club_description,\n",
    "        'tags': club_tags,\n",
    "    }]\n",
    "    \n",
    "club_db_df = pd.DataFrame(club_info_db)\n",
    "club_db_df = club_db_df.dropna(axis=0, how='any', thresh=None, subset=None, inplace=False)\n",
    "club_db_df = club_db_df.reset_index(drop = True)\n",
    "club_db_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simulate User Favorites and Interest Lists \n",
    "Note: we only do this since we don't have data yet! Once we have a list of users with their favorites and interests, we can use that instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Find number of clubs in each tag\n",
    "tag1 = set([1])\n",
    "istag1 = tag1.issubset\n",
    "club_db_df[club_db_df.tags.map(istag1)]\n",
    "\n",
    "for i in all_club_tags_df.index:\n",
    "    tagnum = set([i])\n",
    "    istag = tagnum.issubset\n",
    "    num_clubs_in_tag = len(club_db_df[club_db_df.tags.map(istag)])\n",
    "    print ('tag', i, 'has', num_clubs_in_tag, 'clubs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simulation Function\n",
    "import random\n",
    "'''\n",
    "    - Takes in # of Students/Users to sample for\n",
    "    - Assumes that users will only favorite clubs that have tags in interest list \n",
    "        (if there are even enough clubs in the tag, else it just randomly picks favorite clubs)\n",
    "    - Simulates between 1-3 interests and 1-3 favorites from those interests\n",
    "    \n",
    "'''\n",
    "def simulate(num_students):\n",
    "    users = list(range(num_students))\n",
    "    sample_users = pd.DataFrame(users).rename(columns={0:'user_id'})\n",
    "    interests, favorites = [], []\n",
    "    for i in range(len(sample_users)):\n",
    "        a = random.sample(set(range(0, 24)), random.randint(1,3))\n",
    "        interests += [a]\n",
    "        intersections = [set.intersection(set(a), set(i)) for i in club_db_df['tags']]\n",
    "        club_db_df['intersections'] = [list(i) for i in intersections]\n",
    "        club_db_df['len_intersections'] = [len(i) for i in club_db_df['intersections']]\n",
    "        clubs_within_interests = club_db_df[club_db_df['len_intersections']>=1]\n",
    "        favorites += [random.sample(set(clubs_within_interests['name']), random.randint(1,3)) if len(clubs_within_interests)>=3 \n",
    "                      else random.sample(set(club_db_df['name']), random.randint(1,10))]\n",
    "\n",
    "    sample_users['interests'] = interests\n",
    "    sample_users['favorites'] = favorites\n",
    "    return sample_users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_users = simulate(100)\n",
    "sample_users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create Exploded DF to prepare for TuriCreate modeling (one favorite per line)\n",
    "sample_users_exploded = sample_users.explode('favorites')\n",
    "sample_users_exploded.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implement TuriCreate Item Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install turicreate\n",
    "import turicreate as tc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 471,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create SFrame for Turi Create\n",
    "sf_users = tc.SFrame(sample_users_exploded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 472,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">user_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">interests</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">favorites</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[15.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Her Campus at Berkeley</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[22.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Web Development at<br>Berkeley ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[6.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">DataStory</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[6.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Berkeley Phi Beta Lambda</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[6.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Berkeley Group</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[22.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Software and Hardware<br>Recruiting List ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[22.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Political Computer<br>Science ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[22.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Blockchain at Berkeley</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[13.0, 22.0, 7.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Cal RoboBears</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">[13.0, 22.0, 7.0]</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Latinx Student<br>Association at Berkeley ...</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[195 rows x 3 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tuser_id\tint\n",
       "\tinterests\tarray\n",
       "\tfavorites\tstr\n",
       "\n",
       "Rows: 195\n",
       "\n",
       "Data:\n",
       "+---------+-------------------+-------------------------------+\n",
       "| user_id |     interests     |           favorites           |\n",
       "+---------+-------------------+-------------------------------+\n",
       "|    0    |       [15.0]      |     Her Campus at Berkeley    |\n",
       "|    1    |       [22.0]      |  Web Development at Berkeley  |\n",
       "|    2    |       [6.0]       |           DataStory           |\n",
       "|    2    |       [6.0]       |    Berkeley Phi Beta Lambda   |\n",
       "|    2    |       [6.0]       |       The Berkeley Group      |\n",
       "|    3    |       [22.0]      | Software and Hardware Recr... |\n",
       "|    3    |       [22.0]      |   Political Computer Science  |\n",
       "|    3    |       [22.0]      |     Blockchain at Berkeley    |\n",
       "|    4    | [13.0, 22.0, 7.0] |         Cal RoboBears         |\n",
       "|    4    | [13.0, 22.0, 7.0] | Latinx Student Association... |\n",
       "+---------+-------------------+-------------------------------+\n",
       "[195 rows x 3 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 472,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sf_users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 473,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Warning: Ignoring columns interests;</pre>"
      ],
      "text/plain": [
       "Warning: Ignoring columns interests;"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre> To use one of these as a rating column, specify the column name to be used as target</pre>"
      ],
      "text/plain": [
       " To use one of these as a rating column, specify the column name to be used as target"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>    and use a method that allows the use of a target.</pre>"
      ],
      "text/plain": [
       "    and use a method that allows the use of a target."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Preparing data set.</pre>"
      ],
      "text/plain": [
       "Preparing data set."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>    Data has 195 observations with 100 users and 138 items.</pre>"
      ],
      "text/plain": [
       "    Data has 195 observations with 100 users and 138 items."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>    Data prepared in: 0.003988s</pre>"
      ],
      "text/plain": [
       "    Data prepared in: 0.003988s"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Training model from provided data.</pre>"
      ],
      "text/plain": [
       "Training model from provided data."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Create tc model\n",
    "m = tc.item_similarity_recommender.create(sf_users, user_id = 'user_id', item_id = 'favorites', similarity_type = 'cosine', only_top_k = 5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 474,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Gathering per-item and per-user statistics.</pre>"
      ],
      "text/plain": [
       "Gathering per-item and per-user statistics."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+--------------------------------+------------+</pre>"
      ],
      "text/plain": [
       "+--------------------------------+------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Elapsed Time (Item Statistics) | % Complete |</pre>"
      ],
      "text/plain": [
       "| Elapsed Time (Item Statistics) | % Complete |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+--------------------------------+------------+</pre>"
      ],
      "text/plain": [
       "+--------------------------------+------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 10.739ms                       | 100        |</pre>"
      ],
      "text/plain": [
       "| 10.739ms                       | 100        |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+--------------------------------+------------+</pre>"
      ],
      "text/plain": [
       "+--------------------------------+------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Setting up lookup tables.</pre>"
      ],
      "text/plain": [
       "Setting up lookup tables."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Create Trained Recommendation DataFrame\n",
    "recs = m.recommend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create and clean joined df (with user favorites and model recommendations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 475,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Processing data in one pass using dense lookup tables.</pre>"
      ],
      "text/plain": [
       "Processing data in one pass using dense lookup tables."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-------------------------------------+------------------+-----------------+</pre>"
      ],
      "text/plain": [
       "+-------------------------------------+------------------+-----------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Elapsed Time (Constructing Lookups) | Total % Complete | Items Processed |</pre>"
      ],
      "text/plain": [
       "| Elapsed Time (Constructing Lookups) | Total % Complete | Items Processed |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-------------------------------------+------------------+-----------------+</pre>"
      ],
      "text/plain": [
       "+-------------------------------------+------------------+-----------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 11.59ms                             | 0.75             | 1               |</pre>"
      ],
      "text/plain": [
       "| 11.59ms                             | 0.75             | 1               |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 13.529ms                            | 100              | 138             |</pre>"
      ],
      "text/plain": [
       "| 13.529ms                            | 100              | 138             |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>interests</th>\n",
       "      <th>favorites</th>\n",
       "      <th>recommended</th>\n",
       "      <th>score</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>[15.0]</td>\n",
       "      <td>Her Campus at Berkeley</td>\n",
       "      <td>Cal Seismic Design Team</td>\n",
       "      <td>0.707107</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>[15.0]</td>\n",
       "      <td>Her Campus at Berkeley</td>\n",
       "      <td>Berkeley Innovation</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>[15.0]</td>\n",
       "      <td>Her Campus at Berkeley</td>\n",
       "      <td>UC Rally Committee</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>[15.0]</td>\n",
       "      <td>Her Campus at Berkeley</td>\n",
       "      <td>Latinx Student Association at Berkeley</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>[15.0]</td>\n",
       "      <td>Her Campus at Berkeley</td>\n",
       "      <td>Cal RoboBears</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1945</th>\n",
       "      <td>99</td>\n",
       "      <td>[0.0, 15.0, 13.0]</td>\n",
       "      <td>United Nations Association of Berkeley</td>\n",
       "      <td>Political Computer Science</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1946</th>\n",
       "      <td>99</td>\n",
       "      <td>[0.0, 15.0, 13.0]</td>\n",
       "      <td>United Nations Association of Berkeley</td>\n",
       "      <td>Software and Hardware Recruiting List</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1947</th>\n",
       "      <td>99</td>\n",
       "      <td>[0.0, 15.0, 13.0]</td>\n",
       "      <td>United Nations Association of Berkeley</td>\n",
       "      <td>The Berkeley Group</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1948</th>\n",
       "      <td>99</td>\n",
       "      <td>[0.0, 15.0, 13.0]</td>\n",
       "      <td>United Nations Association of Berkeley</td>\n",
       "      <td>Berkeley Phi Beta Lambda</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1949</th>\n",
       "      <td>99</td>\n",
       "      <td>[0.0, 15.0, 13.0]</td>\n",
       "      <td>United Nations Association of Berkeley</td>\n",
       "      <td>DataStory</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1950 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id          interests                               favorites  \\\n",
       "0           0             [15.0]                  Her Campus at Berkeley   \n",
       "1           0             [15.0]                  Her Campus at Berkeley   \n",
       "2           0             [15.0]                  Her Campus at Berkeley   \n",
       "3           0             [15.0]                  Her Campus at Berkeley   \n",
       "4           0             [15.0]                  Her Campus at Berkeley   \n",
       "...       ...                ...                                     ...   \n",
       "1945       99  [0.0, 15.0, 13.0]  United Nations Association of Berkeley   \n",
       "1946       99  [0.0, 15.0, 13.0]  United Nations Association of Berkeley   \n",
       "1947       99  [0.0, 15.0, 13.0]  United Nations Association of Berkeley   \n",
       "1948       99  [0.0, 15.0, 13.0]  United Nations Association of Berkeley   \n",
       "1949       99  [0.0, 15.0, 13.0]  United Nations Association of Berkeley   \n",
       "\n",
       "                                 recommended     score  rank  \n",
       "0                    Cal Seismic Design Team  0.707107     1  \n",
       "1                        Berkeley Innovation  0.500000     2  \n",
       "2                         UC Rally Committee  0.000000     3  \n",
       "3     Latinx Student Association at Berkeley  0.000000     4  \n",
       "4                              Cal RoboBears  0.000000     5  \n",
       "...                                      ...       ...   ...  \n",
       "1945              Political Computer Science  0.000000     6  \n",
       "1946   Software and Hardware Recruiting List  0.000000     7  \n",
       "1947                      The Berkeley Group  0.000000     8  \n",
       "1948                Berkeley Phi Beta Lambda  0.000000     9  \n",
       "1949                               DataStory  0.000000    10  \n",
       "\n",
       "[1950 rows x 6 columns]"
      ]
     },
     "execution_count": 475,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-------------------------------------+------------------+-----------------+</pre>"
      ],
      "text/plain": [
       "+-------------------------------------+------------------+-----------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Finalizing lookup tables.</pre>"
      ],
      "text/plain": [
       "Finalizing lookup tables."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Generating candidate set for working with new users.</pre>"
      ],
      "text/plain": [
       "Generating candidate set for working with new users."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Finished training in 0.014618s</pre>"
      ],
      "text/plain": [
       "Finished training in 0.014618s"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "joined_recs = sf_users.join(recs, on= 'user_id', how='inner')\n",
    "joined_recs_df = pd.DataFrame(joined_recs).rename(columns = {'favorites.1': 'recommended'})\n",
    "joined_recs_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 476,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>favorites</th>\n",
       "      <th>recommended</th>\n",
       "      <th>score</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>Cal Seismic Design Team</td>\n",
       "      <td>0.707107</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>Berkeley Innovation</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>UC Rally Committee</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>Latinx Student Association at Berkeley</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>Cal RoboBears</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                  favorites                             recommended  \\\n",
       "0        0  (Her Campus at Berkeley,)                 Cal Seismic Design Team   \n",
       "1        0  (Her Campus at Berkeley,)                     Berkeley Innovation   \n",
       "2        0  (Her Campus at Berkeley,)                      UC Rally Committee   \n",
       "3        0  (Her Campus at Berkeley,)  Latinx Student Association at Berkeley   \n",
       "4        0  (Her Campus at Berkeley,)                           Cal RoboBears   \n",
       "\n",
       "      score  rank  \n",
       "0  0.707107     1  \n",
       "1  0.500000     2  \n",
       "2  0.000000     3  \n",
       "3  0.000000     4  \n",
       "4  0.000000     5  "
      ]
     },
     "execution_count": 476,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Taking top 5 ranked recommendations and collapsing dataframe so it is easier to read\n",
    "old_fav = sample_users[['user_id', 'favorites']]\n",
    "fixed = joined_recs_df.join(old_fav, on = 'user_id', lsuffix='_left', rsuffix='_right', how='left')\n",
    "fixed = fixed[['user_id_left','favorites_right','recommended','score','rank']].rename(columns={'user_id_left':'user_id', 'favorites_right': 'favorites'})\n",
    "fixed['favorites'] = [tuple(i) for i in fixed['favorites']]\n",
    "fixed = fixed.drop_duplicates()\n",
    "fixed = fixed[(fixed['rank'] <= 5)]\n",
    "final = fixed.groupby('user_id')\n",
    "recs2 = []\n",
    "for name, group in final:\n",
    "    recs2 += [group['recommended'].tolist()]\n",
    "fixed.head()                                     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final Useful Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 477,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>favorites</th>\n",
       "      <th>recs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>[Cal Seismic Design Team, Berkeley Innovation,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>(Web Development at Berkeley,)</td>\n",
       "      <td>[Berkeley Innovation, Berkeley Legends, SENDfo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>(DataStory, Berkeley Phi Beta Lambda, The Berk...</td>\n",
       "      <td>[Cal Dragon Boat, Cal Triathlon, UC Rally Comm...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>(Software and Hardware Recruiting List, Politi...</td>\n",
       "      <td>[Extended Reality at Berkeley, REACH! Asian an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>(Cal RoboBears, Latinx Student Association at ...</td>\n",
       "      <td>[Launchpad, Cal Triathlon, UC Rally Committee,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                                          favorites  \\\n",
       "0        0                          (Her Campus at Berkeley,)   \n",
       "1        1                     (Web Development at Berkeley,)   \n",
       "2        2  (DataStory, Berkeley Phi Beta Lambda, The Berk...   \n",
       "3        3  (Software and Hardware Recruiting List, Politi...   \n",
       "4        4  (Cal RoboBears, Latinx Student Association at ...   \n",
       "\n",
       "                                                recs  \n",
       "0  [Cal Seismic Design Team, Berkeley Innovation,...  \n",
       "1  [Berkeley Innovation, Berkeley Legends, SENDfo...  \n",
       "2  [Cal Dragon Boat, Cal Triathlon, UC Rally Comm...  \n",
       "3  [Extended Reality at Berkeley, REACH! Asian an...  \n",
       "4  [Launchpad, Cal Triathlon, UC Rally Committee,...  "
      ]
     },
     "execution_count": 477,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_cleaned = fixed[['user_id', 'favorites']].drop_duplicates().reset_index().drop(columns={'index'})\n",
    "final_cleaned['recs'] = recs2\n",
    "final_cleaned.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Recommendation Test For Feasibility\n",
    "Notes: This model looks at what other users' favorites are and creates recommendations based on other pairings. Similarity scores for now don't seem high, and there are other parameters we should look at and tweak in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 478,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Favorites:  ('Partners in Health Engage at Berkeley',)  Recommendations:  ['Pinnacle Consulting', \"Active for Alzheimer's\", 'Latinx Student Association at Berkeley', 'Cal RoboBears', 'Blockchain at Berkeley']\n"
     ]
    }
   ],
   "source": [
    "#Run to compare a random club's favorites with recommendations\n",
    "\n",
    "club = random.randint(0,len(sample_users))\n",
    "print('Favorites: ', final_cleaned['favorites'][club],' Recommendations: ', final_cleaned['recs'][club])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test whether recommendations are in a each user's interests\n",
    "We only want to keep recommended clubs from item similarity that are in each user's interests! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 479,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>favorites</th>\n",
       "      <th>recs</th>\n",
       "      <th>interests</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>[Cal Seismic Design Team, Berkeley Innovation,...</td>\n",
       "      <td>[15]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>(Web Development at Berkeley,)</td>\n",
       "      <td>[Berkeley Innovation, Berkeley Legends, SENDfo...</td>\n",
       "      <td>[22]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>(DataStory, Berkeley Phi Beta Lambda, The Berk...</td>\n",
       "      <td>[Cal Dragon Boat, Cal Triathlon, UC Rally Comm...</td>\n",
       "      <td>[6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>(Software and Hardware Recruiting List, Politi...</td>\n",
       "      <td>[Extended Reality at Berkeley, REACH! Asian an...</td>\n",
       "      <td>[22]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>(Cal RoboBears, Latinx Student Association at ...</td>\n",
       "      <td>[Launchpad, Cal Triathlon, UC Rally Committee,...</td>\n",
       "      <td>[13, 22, 7]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>95</td>\n",
       "      <td>(Azaad Premiere Dance,)</td>\n",
       "      <td>[Cal UPHC, Latinx Student Association at Berke...</td>\n",
       "      <td>[7]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>96</td>\n",
       "      <td>(Perennial: The Undergraduate Environmental Jo...</td>\n",
       "      <td>[Student Premed Advising Network [SPAN], Latin...</td>\n",
       "      <td>[18, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>97</td>\n",
       "      <td>(Peer Health Exchange,)</td>\n",
       "      <td>[United Nations Association of Berkeley, Stude...</td>\n",
       "      <td>[14, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>98</td>\n",
       "      <td>(ATP at Berkeley, Phoenix Consulting Group, SE...</td>\n",
       "      <td>[MCBcDNA, Consult Your Community (CYC), Berkel...</td>\n",
       "      <td>[15, 4]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>99</td>\n",
       "      <td>(United Nations Association of Berkeley,)</td>\n",
       "      <td>[Peer Health Exchange, Student Premed Advising...</td>\n",
       "      <td>[0, 15, 13]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    user_id                                          favorites  \\\n",
       "0         0                          (Her Campus at Berkeley,)   \n",
       "1         1                     (Web Development at Berkeley,)   \n",
       "2         2  (DataStory, Berkeley Phi Beta Lambda, The Berk...   \n",
       "3         3  (Software and Hardware Recruiting List, Politi...   \n",
       "4         4  (Cal RoboBears, Latinx Student Association at ...   \n",
       "..      ...                                                ...   \n",
       "95       95                            (Azaad Premiere Dance,)   \n",
       "96       96  (Perennial: The Undergraduate Environmental Jo...   \n",
       "97       97                            (Peer Health Exchange,)   \n",
       "98       98  (ATP at Berkeley, Phoenix Consulting Group, SE...   \n",
       "99       99          (United Nations Association of Berkeley,)   \n",
       "\n",
       "                                                 recs    interests  \n",
       "0   [Cal Seismic Design Team, Berkeley Innovation,...         [15]  \n",
       "1   [Berkeley Innovation, Berkeley Legends, SENDfo...         [22]  \n",
       "2   [Cal Dragon Boat, Cal Triathlon, UC Rally Comm...          [6]  \n",
       "3   [Extended Reality at Berkeley, REACH! Asian an...         [22]  \n",
       "4   [Launchpad, Cal Triathlon, UC Rally Committee,...  [13, 22, 7]  \n",
       "..                                                ...          ...  \n",
       "95  [Cal UPHC, Latinx Student Association at Berke...          [7]  \n",
       "96  [Student Premed Advising Network [SPAN], Latin...      [18, 0]  \n",
       "97  [United Nations Association of Berkeley, Stude...      [14, 0]  \n",
       "98  [MCBcDNA, Consult Your Community (CYC), Berkel...      [15, 4]  \n",
       "99  [Peer Health Exchange, Student Premed Advising...  [0, 15, 13]  \n",
       "\n",
       "[100 rows x 4 columns]"
      ]
     },
     "execution_count": 479,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relevant_recs = final_cleaned\n",
    "relevant_recs['interests'] = sample_users['interests']\n",
    "relevant_recs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 480,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 480,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def relevance(rec_list, interest_list):\n",
    "    '''\n",
    "    This function takes in a list of recommendations and an interest list and returns recommendations that are in interest lists\n",
    "    If a user has no interests, then will just return all recommendations as relevant.\n",
    "    '''\n",
    "    relevant_clubs = []\n",
    "    if len(interest_list) == 0:\n",
    "        relevant_clubs = [i for i in rec_list]\n",
    "        return relevant_clubs \n",
    "    else:\n",
    "        for club in rec_list:\n",
    "            club_tags = list(club_db_df[club_db_df['name'] == club]['tags'])[0]\n",
    "            for tag in club_tags:\n",
    "                if tag in interest_list:\n",
    "                    relevant_clubs += [club]\n",
    "                    break\n",
    "    return relevant_clubs        \n",
    "        \n",
    "        \n",
    "relevance(relevant_recs['recs'][0], relevant_recs['interests'][0])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 481,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>favorites</th>\n",
       "      <th>recs</th>\n",
       "      <th>interests</th>\n",
       "      <th>relevant_clubs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>[Cal Seismic Design Team, Berkeley Innovation,...</td>\n",
       "      <td>[15]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>(Web Development at Berkeley,)</td>\n",
       "      <td>[Berkeley Innovation, Berkeley Legends, SENDfo...</td>\n",
       "      <td>[22]</td>\n",
       "      <td>[Berkeley Legends]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>(DataStory, Berkeley Phi Beta Lambda, The Berk...</td>\n",
       "      <td>[Cal Dragon Boat, Cal Triathlon, UC Rally Comm...</td>\n",
       "      <td>[6]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>(Software and Hardware Recruiting List, Politi...</td>\n",
       "      <td>[Extended Reality at Berkeley, REACH! Asian an...</td>\n",
       "      <td>[22]</td>\n",
       "      <td>[Extended Reality at Berkeley, FemTech]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>(Cal RoboBears, Latinx Student Association at ...</td>\n",
       "      <td>[Launchpad, Cal Triathlon, UC Rally Committee,...</td>\n",
       "      <td>[13, 22, 7]</td>\n",
       "      <td>[Launchpad, Blockchain at Berkeley, Political ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                                          favorites  \\\n",
       "0        0                          (Her Campus at Berkeley,)   \n",
       "1        1                     (Web Development at Berkeley,)   \n",
       "2        2  (DataStory, Berkeley Phi Beta Lambda, The Berk...   \n",
       "3        3  (Software and Hardware Recruiting List, Politi...   \n",
       "4        4  (Cal RoboBears, Latinx Student Association at ...   \n",
       "\n",
       "                                                recs    interests  \\\n",
       "0  [Cal Seismic Design Team, Berkeley Innovation,...         [15]   \n",
       "1  [Berkeley Innovation, Berkeley Legends, SENDfo...         [22]   \n",
       "2  [Cal Dragon Boat, Cal Triathlon, UC Rally Comm...          [6]   \n",
       "3  [Extended Reality at Berkeley, REACH! Asian an...         [22]   \n",
       "4  [Launchpad, Cal Triathlon, UC Rally Committee,...  [13, 22, 7]   \n",
       "\n",
       "                                      relevant_clubs  \n",
       "0                                                 []  \n",
       "1                                 [Berkeley Legends]  \n",
       "2                                                 []  \n",
       "3            [Extended Reality at Berkeley, FemTech]  \n",
       "4  [Launchpad, Blockchain at Berkeley, Political ...  "
      ]
     },
     "execution_count": 481,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply to the rest of the rows\n",
    "relevant_recs['relevant_clubs'] = relevant_recs.apply(lambda row : relevance(row['recs'],\n",
    "                                  row['interests']), axis = 1)\n",
    "relevant_recs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fill blanks with Generic Club Recommendations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GENERIC CLUB ALGORITHM (taken from other notebook):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 482,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>link_name</th>\n",
       "      <th>tags</th>\n",
       "      <th>intersections</th>\n",
       "      <th>len_intersections</th>\n",
       "      <th>clean_description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Berkeley Alt. Protein Project</td>\n",
       "      <td>the-berkeley-alt.-protein-project</td>\n",
       "      <td>[10, 11, 18]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[alt, protein, project, strives, build, health...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EthiCAL Apparel</td>\n",
       "      <td>ethical-apparel</td>\n",
       "      <td>[2, 8, 20]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[p, ethical, apparel, strong, student, run, so...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UC Rally Committee</td>\n",
       "      <td>uc-rally-committee</td>\n",
       "      <td>[20, 21]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[rally, committee, members, official, guardian...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Formula Electric at Berkeley</td>\n",
       "      <td>formula-electric-at-berkeley</td>\n",
       "      <td>[2, 8, 9]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[join, us, building, cal, first, ever, formula...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Codebase</td>\n",
       "      <td>codebase</td>\n",
       "      <td>[5, 19, 22]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[codebase, community, developers, empowers, st...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>281</th>\n",
       "      <td>UNICEF at Berkeley</td>\n",
       "      <td>unicef-at-berkeley</td>\n",
       "      <td>[0, 19, 20]</td>\n",
       "      <td>[0]</td>\n",
       "      <td>1</td>\n",
       "      <td>[p, leading, unicef, campus, initiatives, aspi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282</th>\n",
       "      <td>Volunteers Around the World</td>\n",
       "      <td>volunteers-around-the-world</td>\n",
       "      <td>[0, 4, 11]</td>\n",
       "      <td>[0]</td>\n",
       "      <td>1</td>\n",
       "      <td>[p, student, organization, dedicated, promotin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283</th>\n",
       "      <td>Womxn Ignite @ Berkeley</td>\n",
       "      <td>womxn-ignite-berkeley</td>\n",
       "      <td>[0, 14, 20]</td>\n",
       "      <td>[0]</td>\n",
       "      <td>1</td>\n",
       "      <td>[p, hello, bears, womxn, ignite, collegiate, c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284</th>\n",
       "      <td>Delta Consulting</td>\n",
       "      <td>delta-consulting</td>\n",
       "      <td>[2, 6, 22]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[h, overview, h, p, delta, consulting, strong,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285</th>\n",
       "      <td>Biofuels Technology Club</td>\n",
       "      <td>biofuels-technology-club</td>\n",
       "      <td>[10, 17, 18]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[p, biofuels, technology, club, btc, seeks, cr...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>286 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name                          link_name  \\\n",
       "0    The Berkeley Alt. Protein Project  the-berkeley-alt.-protein-project   \n",
       "1                      EthiCAL Apparel                    ethical-apparel   \n",
       "2                   UC Rally Committee                 uc-rally-committee   \n",
       "3         Formula Electric at Berkeley       formula-electric-at-berkeley   \n",
       "4                             Codebase                           codebase   \n",
       "..                                 ...                                ...   \n",
       "281                 UNICEF at Berkeley                 unicef-at-berkeley   \n",
       "282        Volunteers Around the World        volunteers-around-the-world   \n",
       "283            Womxn Ignite @ Berkeley              womxn-ignite-berkeley   \n",
       "284                   Delta Consulting                   delta-consulting   \n",
       "285           Biofuels Technology Club           biofuels-technology-club   \n",
       "\n",
       "             tags intersections  len_intersections  \\\n",
       "0    [10, 11, 18]            []                  0   \n",
       "1      [2, 8, 20]            []                  0   \n",
       "2        [20, 21]            []                  0   \n",
       "3       [2, 8, 9]            []                  0   \n",
       "4     [5, 19, 22]            []                  0   \n",
       "..            ...           ...                ...   \n",
       "281   [0, 19, 20]           [0]                  1   \n",
       "282    [0, 4, 11]           [0]                  1   \n",
       "283   [0, 14, 20]           [0]                  1   \n",
       "284    [2, 6, 22]            []                  0   \n",
       "285  [10, 17, 18]            []                  0   \n",
       "\n",
       "                                     clean_description  \n",
       "0    [alt, protein, project, strives, build, health...  \n",
       "1    [p, ethical, apparel, strong, student, run, so...  \n",
       "2    [rally, committee, members, official, guardian...  \n",
       "3    [join, us, building, cal, first, ever, formula...  \n",
       "4    [codebase, community, developers, empowers, st...  \n",
       "..                                                 ...  \n",
       "281  [p, leading, unicef, campus, initiatives, aspi...  \n",
       "282  [p, student, organization, dedicated, promotin...  \n",
       "283  [p, hello, bears, womxn, ignite, collegiate, c...  \n",
       "284  [h, overview, h, p, delta, consulting, strong,...  \n",
       "285  [p, biofuels, technology, club, btc, seeks, cr...  \n",
       "\n",
       "[286 rows x 6 columns]"
      ]
     },
     "execution_count": 482,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#STEP 1: CLEAN TABLE\n",
    "def clean_table(table):\n",
    "    \"\"\"\n",
    "    Description:\n",
    "    Cleans club dataframe descriptions into another column containing a lists of significant words in each description.\n",
    "    \n",
    "    Input:\n",
    "    table - club dataframe\n",
    "    \n",
    "    Output:\n",
    "    cleaned_table - table with a new column: \"cleaned descriptions\"\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    def clean_description(description):\n",
    "        \"\"\"\n",
    "        Description:\n",
    "        Clean single description into lists of significant words.\n",
    "\n",
    "        Input:\n",
    "        description - string of club description \n",
    "\n",
    "        Output:\n",
    "        new_description - list of significant words in description\n",
    "        \"\"\"\n",
    "        \n",
    "        try: \n",
    "            # Remove punctuation\n",
    "            new_description = re.sub(\"[^a-zA-Z]\", \" \", description)\n",
    "\n",
    "            # Tokenize into words (all lower case)\n",
    "            new_description = new_description.lower().split()\n",
    "\n",
    "            # Remove stopwords\n",
    "            eng_stopwords = set(stopwords.words(\"english\"))\n",
    "            new_description = [w for w in new_description if not w in eng_stopwords]\n",
    "\n",
    "            # Remove \"uc\" and \"berkeley\"\n",
    "            uc_berkeley = ['uc', 'berkeley', 'also', 'providing', 'various', 'well', 'provide', 'one']\n",
    "            new_description = [w for w in new_description if not w in uc_berkeley]\n",
    "        except TypeError:\n",
    "            return [\"\"]\n",
    "\n",
    "        return new_description\n",
    "    \n",
    "    clean_descriptions = []\n",
    "    \n",
    "    for i in np.arange(len(table)):\n",
    "        clean_descriptions += [clean_description(table['description'][i])]\n",
    "        \n",
    "    cleaned_table = table.drop(['description'], axis=1)\n",
    "    cleaned_table['clean_description'] = clean_descriptions\n",
    "    \n",
    "    return cleaned_table\n",
    "\n",
    "cleaned_table = clean_table(club_db_df)\n",
    "cleaned_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 483,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>link_name</th>\n",
       "      <th>tags</th>\n",
       "      <th>intersections</th>\n",
       "      <th>len_intersections</th>\n",
       "      <th>clean_description</th>\n",
       "      <th>vector_sum</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Berkeley Alt. Protein Project</td>\n",
       "      <td>the-berkeley-alt.-protein-project</td>\n",
       "      <td>[10, 11, 18]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[alt, protein, project, strives, build, health...</td>\n",
       "      <td>[0.0083650965, -0.014156526, -0.0027822545, 0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EthiCAL Apparel</td>\n",
       "      <td>ethical-apparel</td>\n",
       "      <td>[2, 8, 20]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[p, ethical, apparel, strong, student, run, so...</td>\n",
       "      <td>[0.018845102, -0.14857782, -0.011552009, -0.04...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UC Rally Committee</td>\n",
       "      <td>uc-rally-committee</td>\n",
       "      <td>[20, 21]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[rally, committee, members, official, guardian...</td>\n",
       "      <td>[0.008026369, -0.09144393, 0.026339643, -0.037...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Formula Electric at Berkeley</td>\n",
       "      <td>formula-electric-at-berkeley</td>\n",
       "      <td>[2, 8, 9]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[join, us, building, cal, first, ever, formula...</td>\n",
       "      <td>[0.0011783288, -0.052926827, 0.03044209, -0.03...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Codebase</td>\n",
       "      <td>codebase</td>\n",
       "      <td>[5, 19, 22]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[codebase, community, developers, empowers, st...</td>\n",
       "      <td>[0.009571702, -0.08639321, 0.029330207, -0.025...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>281</th>\n",
       "      <td>UNICEF at Berkeley</td>\n",
       "      <td>unicef-at-berkeley</td>\n",
       "      <td>[0, 19, 20]</td>\n",
       "      <td>[0]</td>\n",
       "      <td>1</td>\n",
       "      <td>[p, leading, unicef, campus, initiatives, aspi...</td>\n",
       "      <td>[0.008294601, -0.11319724, -0.0039510364, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282</th>\n",
       "      <td>Volunteers Around the World</td>\n",
       "      <td>volunteers-around-the-world</td>\n",
       "      <td>[0, 4, 11]</td>\n",
       "      <td>[0]</td>\n",
       "      <td>1</td>\n",
       "      <td>[p, student, organization, dedicated, promotin...</td>\n",
       "      <td>[0.02544077, -0.09716282, 0.037796564, -0.0106...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283</th>\n",
       "      <td>Womxn Ignite @ Berkeley</td>\n",
       "      <td>womxn-ignite-berkeley</td>\n",
       "      <td>[0, 14, 20]</td>\n",
       "      <td>[0]</td>\n",
       "      <td>1</td>\n",
       "      <td>[p, hello, bears, womxn, ignite, collegiate, c...</td>\n",
       "      <td>[-0.0014062113, -0.09386333, -0.01878618, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284</th>\n",
       "      <td>Delta Consulting</td>\n",
       "      <td>delta-consulting</td>\n",
       "      <td>[2, 6, 22]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[h, overview, h, p, delta, consulting, strong,...</td>\n",
       "      <td>[0.02652006, -0.18142174, 0.04048846, -0.07147...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285</th>\n",
       "      <td>Biofuels Technology Club</td>\n",
       "      <td>biofuels-technology-club</td>\n",
       "      <td>[10, 17, 18]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>[p, biofuels, technology, club, btc, seeks, cr...</td>\n",
       "      <td>[0.055413112, -0.120385155, -0.007249103, -0.0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>286 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name                          link_name  \\\n",
       "0    The Berkeley Alt. Protein Project  the-berkeley-alt.-protein-project   \n",
       "1                      EthiCAL Apparel                    ethical-apparel   \n",
       "2                   UC Rally Committee                 uc-rally-committee   \n",
       "3         Formula Electric at Berkeley       formula-electric-at-berkeley   \n",
       "4                             Codebase                           codebase   \n",
       "..                                 ...                                ...   \n",
       "281                 UNICEF at Berkeley                 unicef-at-berkeley   \n",
       "282        Volunteers Around the World        volunteers-around-the-world   \n",
       "283            Womxn Ignite @ Berkeley              womxn-ignite-berkeley   \n",
       "284                   Delta Consulting                   delta-consulting   \n",
       "285           Biofuels Technology Club           biofuels-technology-club   \n",
       "\n",
       "             tags intersections  len_intersections  \\\n",
       "0    [10, 11, 18]            []                  0   \n",
       "1      [2, 8, 20]            []                  0   \n",
       "2        [20, 21]            []                  0   \n",
       "3       [2, 8, 9]            []                  0   \n",
       "4     [5, 19, 22]            []                  0   \n",
       "..            ...           ...                ...   \n",
       "281   [0, 19, 20]           [0]                  1   \n",
       "282    [0, 4, 11]           [0]                  1   \n",
       "283   [0, 14, 20]           [0]                  1   \n",
       "284    [2, 6, 22]            []                  0   \n",
       "285  [10, 17, 18]            []                  0   \n",
       "\n",
       "                                     clean_description  \\\n",
       "0    [alt, protein, project, strives, build, health...   \n",
       "1    [p, ethical, apparel, strong, student, run, so...   \n",
       "2    [rally, committee, members, official, guardian...   \n",
       "3    [join, us, building, cal, first, ever, formula...   \n",
       "4    [codebase, community, developers, empowers, st...   \n",
       "..                                                 ...   \n",
       "281  [p, leading, unicef, campus, initiatives, aspi...   \n",
       "282  [p, student, organization, dedicated, promotin...   \n",
       "283  [p, hello, bears, womxn, ignite, collegiate, c...   \n",
       "284  [h, overview, h, p, delta, consulting, strong,...   \n",
       "285  [p, biofuels, technology, club, btc, seeks, cr...   \n",
       "\n",
       "                                            vector_sum  \n",
       "0    [0.0083650965, -0.014156526, -0.0027822545, 0....  \n",
       "1    [0.018845102, -0.14857782, -0.011552009, -0.04...  \n",
       "2    [0.008026369, -0.09144393, 0.026339643, -0.037...  \n",
       "3    [0.0011783288, -0.052926827, 0.03044209, -0.03...  \n",
       "4    [0.009571702, -0.08639321, 0.029330207, -0.025...  \n",
       "..                                                 ...  \n",
       "281  [0.008294601, -0.11319724, -0.0039510364, -0.0...  \n",
       "282  [0.02544077, -0.09716282, 0.037796564, -0.0106...  \n",
       "283  [-0.0014062113, -0.09386333, -0.01878618, -0.0...  \n",
       "284  [0.02652006, -0.18142174, 0.04048846, -0.07147...  \n",
       "285  [0.055413112, -0.120385155, -0.007249103, -0.0...  \n",
       "\n",
       "[286 rows x 7 columns]"
      ]
     },
     "execution_count": 483,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#STEP 2: VECTORIZE TABLE\n",
    "def vectorize_table(table, yield_model = False):\n",
    "    MIN_WORD_COUNT = 20\n",
    "    VECTOR_SIZE = 100\n",
    "    CONTEXT_WINDOW_SIZE = 10\n",
    "    \n",
    "    \"\"\"\n",
    "    Description:\n",
    "    Uses cleaned table to create another column containing vectors using gensim's word2vec.\n",
    "    \n",
    "    Input:\n",
    "    table - cleaned table\n",
    "    \n",
    "    Output:\n",
    "    vectorized_table - table with a new_column: \"vector sum\"\n",
    "    \n",
    "    Run word2vec model\n",
    "    \"\"\"\n",
    "    \n",
    "    list_vectors = []\n",
    "    \n",
    "    model = gensim.models.Word2Vec(\n",
    "        table['clean_description'],\n",
    "        min_count=MIN_WORD_COUNT,\n",
    "        size=VECTOR_SIZE,\n",
    "        window=CONTEXT_WINDOW_SIZE,\n",
    "        compute_loss=True,\n",
    "        sample=1e-3 / 2,\n",
    "        seed=42,\n",
    "        workers=1\n",
    "    )\n",
    "    \n",
    "    for i in range(len(table)):\n",
    "        ith_description = table['clean_description'][i]    \n",
    "        \n",
    "        ith_vector_list = []\n",
    "        for ith_description_word in ith_description:\n",
    "            if ith_description_word in model.wv:\n",
    "                ith_vector_list += [model.wv[ith_description_word]]\n",
    "        \n",
    "        if len(ith_vector_list) == 0:\n",
    "            description_sum_vector = [1e-6] * VECTOR_SIZE\n",
    "        else:\n",
    "            description_sum_vector = sum(np.array(ith_vector_list))\n",
    "            \n",
    "        list_vectors += [description_sum_vector]\n",
    "        \n",
    "    vectorized_table = table.copy()\n",
    "    vectorized_table['vector_sum'] = list_vectors\n",
    "    \n",
    "    if yield_model:\n",
    "        return vectorized_table, model\n",
    "    else:\n",
    "        return vectorized_table\n",
    "    \n",
    "vector_table, w2v_model = vectorize_table(cleaned_table, yield_model=True)\n",
    "vector_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 484,
   "metadata": {},
   "outputs": [],
   "source": [
    "#STEP 3: CREATE DISTANCE TABLE\n",
    "from scipy import spatial\n",
    "\n",
    "def create_distance_table(table):\n",
    "    \"\"\"\n",
    "    Description:\n",
    "    Uses a vectorized table to create a pivot table containing distances between each club.\n",
    "    \n",
    "    Input:\n",
    "    table - table with vectorized descriptions\n",
    "    \n",
    "    Output:\n",
    "    distance_table - table containing all distances between each club\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    dict = table[['link_name', 'vector_sum']].set_index('link_name')['vector_sum'].to_dict()\n",
    "    distance_list = []\n",
    "    \n",
    "    for club_1 in dict:\n",
    "        vector_1 = dict[club_1]\n",
    "        distance_dictionary = {}\n",
    "        \n",
    "        for club_2 in dict:\n",
    "            vector_2 = dict[club_2]\n",
    "            \n",
    "            cosine_sim = 1 - spatial.distance.cosine(vector_1, vector_2)\n",
    "            distance_dictionary[club_2] = cosine_sim\n",
    "            \n",
    "        distance_list += [distance_dictionary]\n",
    "        \n",
    "    distance_table = pd.DataFrame(data=distance_list)\n",
    "    distance_table.index = table['link_name']\n",
    "    \n",
    "    return distance_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 485,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>the-berkeley-alt.-protein-project</th>\n",
       "      <th>ethical-apparel</th>\n",
       "      <th>uc-rally-committee</th>\n",
       "      <th>formula-electric-at-berkeley</th>\n",
       "      <th>codebase</th>\n",
       "      <th>berkeley-phi-beta-lambda</th>\n",
       "      <th>cal-seismic-design-team-</th>\n",
       "      <th>sendforc-at-uc-berkeley</th>\n",
       "      <th>beam--berkeley-engineers-and-mentors</th>\n",
       "      <th>berkeley-finance-club</th>\n",
       "      <th>...</th>\n",
       "      <th>nextgen-consulting</th>\n",
       "      <th>bare-magazine</th>\n",
       "      <th>phi-delta-epsilon</th>\n",
       "      <th>underwater-robotics-at-berkeley</th>\n",
       "      <th>reach-asian-and-pacific-islander-recruitment-and-retention-center</th>\n",
       "      <th>unicef-at-berkeley</th>\n",
       "      <th>volunteers-around-the-world</th>\n",
       "      <th>womxn-ignite-berkeley</th>\n",
       "      <th>delta-consulting</th>\n",
       "      <th>biofuels-technology-club</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>link_name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>the-berkeley-alt.-protein-project</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.936440</td>\n",
       "      <td>0.934839</td>\n",
       "      <td>0.913621</td>\n",
       "      <td>0.946679</td>\n",
       "      <td>0.943201</td>\n",
       "      <td>0.946135</td>\n",
       "      <td>0.952112</td>\n",
       "      <td>0.059047</td>\n",
       "      <td>0.931035</td>\n",
       "      <td>...</td>\n",
       "      <td>0.946223</td>\n",
       "      <td>0.059047</td>\n",
       "      <td>0.936881</td>\n",
       "      <td>0.932828</td>\n",
       "      <td>0.937526</td>\n",
       "      <td>0.941223</td>\n",
       "      <td>0.936032</td>\n",
       "      <td>0.927831</td>\n",
       "      <td>0.937292</td>\n",
       "      <td>0.935098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ethical-apparel</th>\n",
       "      <td>0.936440</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.972559</td>\n",
       "      <td>0.933958</td>\n",
       "      <td>0.975164</td>\n",
       "      <td>0.977925</td>\n",
       "      <td>0.972106</td>\n",
       "      <td>0.969986</td>\n",
       "      <td>0.020248</td>\n",
       "      <td>0.962386</td>\n",
       "      <td>...</td>\n",
       "      <td>0.993335</td>\n",
       "      <td>0.020248</td>\n",
       "      <td>0.993228</td>\n",
       "      <td>0.986603</td>\n",
       "      <td>0.992584</td>\n",
       "      <td>0.982321</td>\n",
       "      <td>0.984404</td>\n",
       "      <td>0.992591</td>\n",
       "      <td>0.992389</td>\n",
       "      <td>0.978835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uc-rally-committee</th>\n",
       "      <td>0.934839</td>\n",
       "      <td>0.972559</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.947197</td>\n",
       "      <td>0.973878</td>\n",
       "      <td>0.982986</td>\n",
       "      <td>0.971159</td>\n",
       "      <td>0.972477</td>\n",
       "      <td>0.034249</td>\n",
       "      <td>0.967880</td>\n",
       "      <td>...</td>\n",
       "      <td>0.978608</td>\n",
       "      <td>0.034249</td>\n",
       "      <td>0.976043</td>\n",
       "      <td>0.971811</td>\n",
       "      <td>0.968250</td>\n",
       "      <td>0.978412</td>\n",
       "      <td>0.981915</td>\n",
       "      <td>0.968939</td>\n",
       "      <td>0.975256</td>\n",
       "      <td>0.967887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>formula-electric-at-berkeley</th>\n",
       "      <td>0.913621</td>\n",
       "      <td>0.933958</td>\n",
       "      <td>0.947197</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.940542</td>\n",
       "      <td>0.962001</td>\n",
       "      <td>0.949755</td>\n",
       "      <td>0.941188</td>\n",
       "      <td>0.050537</td>\n",
       "      <td>0.927266</td>\n",
       "      <td>...</td>\n",
       "      <td>0.944117</td>\n",
       "      <td>0.050537</td>\n",
       "      <td>0.937625</td>\n",
       "      <td>0.954711</td>\n",
       "      <td>0.935886</td>\n",
       "      <td>0.939227</td>\n",
       "      <td>0.937696</td>\n",
       "      <td>0.929891</td>\n",
       "      <td>0.953451</td>\n",
       "      <td>0.929127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>codebase</th>\n",
       "      <td>0.946679</td>\n",
       "      <td>0.975164</td>\n",
       "      <td>0.973878</td>\n",
       "      <td>0.940542</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.983713</td>\n",
       "      <td>0.977173</td>\n",
       "      <td>0.973726</td>\n",
       "      <td>0.059115</td>\n",
       "      <td>0.970006</td>\n",
       "      <td>...</td>\n",
       "      <td>0.981242</td>\n",
       "      <td>0.059115</td>\n",
       "      <td>0.978218</td>\n",
       "      <td>0.972722</td>\n",
       "      <td>0.975969</td>\n",
       "      <td>0.971463</td>\n",
       "      <td>0.979377</td>\n",
       "      <td>0.970345</td>\n",
       "      <td>0.980653</td>\n",
       "      <td>0.959684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unicef-at-berkeley</th>\n",
       "      <td>0.941223</td>\n",
       "      <td>0.982321</td>\n",
       "      <td>0.978412</td>\n",
       "      <td>0.939227</td>\n",
       "      <td>0.971463</td>\n",
       "      <td>0.975574</td>\n",
       "      <td>0.965858</td>\n",
       "      <td>0.970782</td>\n",
       "      <td>-0.019417</td>\n",
       "      <td>0.962485</td>\n",
       "      <td>...</td>\n",
       "      <td>0.987128</td>\n",
       "      <td>-0.019417</td>\n",
       "      <td>0.981799</td>\n",
       "      <td>0.975986</td>\n",
       "      <td>0.981096</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.982877</td>\n",
       "      <td>0.983824</td>\n",
       "      <td>0.979850</td>\n",
       "      <td>0.989599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>volunteers-around-the-world</th>\n",
       "      <td>0.936032</td>\n",
       "      <td>0.984404</td>\n",
       "      <td>0.981915</td>\n",
       "      <td>0.937696</td>\n",
       "      <td>0.979377</td>\n",
       "      <td>0.983528</td>\n",
       "      <td>0.971591</td>\n",
       "      <td>0.967102</td>\n",
       "      <td>0.027456</td>\n",
       "      <td>0.967990</td>\n",
       "      <td>...</td>\n",
       "      <td>0.989707</td>\n",
       "      <td>0.027456</td>\n",
       "      <td>0.986948</td>\n",
       "      <td>0.980247</td>\n",
       "      <td>0.982695</td>\n",
       "      <td>0.982877</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.982263</td>\n",
       "      <td>0.986885</td>\n",
       "      <td>0.976712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>womxn-ignite-berkeley</th>\n",
       "      <td>0.927831</td>\n",
       "      <td>0.992591</td>\n",
       "      <td>0.968939</td>\n",
       "      <td>0.929891</td>\n",
       "      <td>0.970345</td>\n",
       "      <td>0.971397</td>\n",
       "      <td>0.964334</td>\n",
       "      <td>0.963412</td>\n",
       "      <td>0.016586</td>\n",
       "      <td>0.952025</td>\n",
       "      <td>...</td>\n",
       "      <td>0.991496</td>\n",
       "      <td>0.016586</td>\n",
       "      <td>0.994340</td>\n",
       "      <td>0.989475</td>\n",
       "      <td>0.989725</td>\n",
       "      <td>0.983824</td>\n",
       "      <td>0.982263</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.986532</td>\n",
       "      <td>0.977335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>delta-consulting</th>\n",
       "      <td>0.937292</td>\n",
       "      <td>0.992389</td>\n",
       "      <td>0.975256</td>\n",
       "      <td>0.953451</td>\n",
       "      <td>0.980653</td>\n",
       "      <td>0.983970</td>\n",
       "      <td>0.977770</td>\n",
       "      <td>0.967687</td>\n",
       "      <td>0.032679</td>\n",
       "      <td>0.969046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.992949</td>\n",
       "      <td>0.032679</td>\n",
       "      <td>0.987620</td>\n",
       "      <td>0.989393</td>\n",
       "      <td>0.993019</td>\n",
       "      <td>0.979850</td>\n",
       "      <td>0.986885</td>\n",
       "      <td>0.986532</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.973482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>biofuels-technology-club</th>\n",
       "      <td>0.935098</td>\n",
       "      <td>0.978835</td>\n",
       "      <td>0.967887</td>\n",
       "      <td>0.929127</td>\n",
       "      <td>0.959684</td>\n",
       "      <td>0.963963</td>\n",
       "      <td>0.959060</td>\n",
       "      <td>0.961365</td>\n",
       "      <td>-0.008870</td>\n",
       "      <td>0.947665</td>\n",
       "      <td>...</td>\n",
       "      <td>0.981205</td>\n",
       "      <td>-0.008870</td>\n",
       "      <td>0.975230</td>\n",
       "      <td>0.969630</td>\n",
       "      <td>0.971816</td>\n",
       "      <td>0.989599</td>\n",
       "      <td>0.976712</td>\n",
       "      <td>0.977335</td>\n",
       "      <td>0.973482</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>286 rows × 286 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   the-berkeley-alt.-protein-project  \\\n",
       "link_name                                                              \n",
       "the-berkeley-alt.-protein-project                           1.000000   \n",
       "ethical-apparel                                             0.936440   \n",
       "uc-rally-committee                                          0.934839   \n",
       "formula-electric-at-berkeley                                0.913621   \n",
       "codebase                                                    0.946679   \n",
       "...                                                              ...   \n",
       "unicef-at-berkeley                                          0.941223   \n",
       "volunteers-around-the-world                                 0.936032   \n",
       "womxn-ignite-berkeley                                       0.927831   \n",
       "delta-consulting                                            0.937292   \n",
       "biofuels-technology-club                                    0.935098   \n",
       "\n",
       "                                   ethical-apparel  uc-rally-committee  \\\n",
       "link_name                                                                \n",
       "the-berkeley-alt.-protein-project         0.936440            0.934839   \n",
       "ethical-apparel                           1.000000            0.972559   \n",
       "uc-rally-committee                        0.972559            1.000000   \n",
       "formula-electric-at-berkeley              0.933958            0.947197   \n",
       "codebase                                  0.975164            0.973878   \n",
       "...                                            ...                 ...   \n",
       "unicef-at-berkeley                        0.982321            0.978412   \n",
       "volunteers-around-the-world               0.984404            0.981915   \n",
       "womxn-ignite-berkeley                     0.992591            0.968939   \n",
       "delta-consulting                          0.992389            0.975256   \n",
       "biofuels-technology-club                  0.978835            0.967887   \n",
       "\n",
       "                                   formula-electric-at-berkeley  codebase  \\\n",
       "link_name                                                                   \n",
       "the-berkeley-alt.-protein-project                      0.913621  0.946679   \n",
       "ethical-apparel                                        0.933958  0.975164   \n",
       "uc-rally-committee                                     0.947197  0.973878   \n",
       "formula-electric-at-berkeley                           1.000000  0.940542   \n",
       "codebase                                               0.940542  1.000000   \n",
       "...                                                         ...       ...   \n",
       "unicef-at-berkeley                                     0.939227  0.971463   \n",
       "volunteers-around-the-world                            0.937696  0.979377   \n",
       "womxn-ignite-berkeley                                  0.929891  0.970345   \n",
       "delta-consulting                                       0.953451  0.980653   \n",
       "biofuels-technology-club                               0.929127  0.959684   \n",
       "\n",
       "                                   berkeley-phi-beta-lambda  \\\n",
       "link_name                                                     \n",
       "the-berkeley-alt.-protein-project                  0.943201   \n",
       "ethical-apparel                                    0.977925   \n",
       "uc-rally-committee                                 0.982986   \n",
       "formula-electric-at-berkeley                       0.962001   \n",
       "codebase                                           0.983713   \n",
       "...                                                     ...   \n",
       "unicef-at-berkeley                                 0.975574   \n",
       "volunteers-around-the-world                        0.983528   \n",
       "womxn-ignite-berkeley                              0.971397   \n",
       "delta-consulting                                   0.983970   \n",
       "biofuels-technology-club                           0.963963   \n",
       "\n",
       "                                   cal-seismic-design-team-  \\\n",
       "link_name                                                     \n",
       "the-berkeley-alt.-protein-project                  0.946135   \n",
       "ethical-apparel                                    0.972106   \n",
       "uc-rally-committee                                 0.971159   \n",
       "formula-electric-at-berkeley                       0.949755   \n",
       "codebase                                           0.977173   \n",
       "...                                                     ...   \n",
       "unicef-at-berkeley                                 0.965858   \n",
       "volunteers-around-the-world                        0.971591   \n",
       "womxn-ignite-berkeley                              0.964334   \n",
       "delta-consulting                                   0.977770   \n",
       "biofuels-technology-club                           0.959060   \n",
       "\n",
       "                                   sendforc-at-uc-berkeley  \\\n",
       "link_name                                                    \n",
       "the-berkeley-alt.-protein-project                 0.952112   \n",
       "ethical-apparel                                   0.969986   \n",
       "uc-rally-committee                                0.972477   \n",
       "formula-electric-at-berkeley                      0.941188   \n",
       "codebase                                          0.973726   \n",
       "...                                                    ...   \n",
       "unicef-at-berkeley                                0.970782   \n",
       "volunteers-around-the-world                       0.967102   \n",
       "womxn-ignite-berkeley                             0.963412   \n",
       "delta-consulting                                  0.967687   \n",
       "biofuels-technology-club                          0.961365   \n",
       "\n",
       "                                   beam--berkeley-engineers-and-mentors  \\\n",
       "link_name                                                                 \n",
       "the-berkeley-alt.-protein-project                              0.059047   \n",
       "ethical-apparel                                                0.020248   \n",
       "uc-rally-committee                                             0.034249   \n",
       "formula-electric-at-berkeley                                   0.050537   \n",
       "codebase                                                       0.059115   \n",
       "...                                                                 ...   \n",
       "unicef-at-berkeley                                            -0.019417   \n",
       "volunteers-around-the-world                                    0.027456   \n",
       "womxn-ignite-berkeley                                          0.016586   \n",
       "delta-consulting                                               0.032679   \n",
       "biofuels-technology-club                                      -0.008870   \n",
       "\n",
       "                                   berkeley-finance-club  ...  \\\n",
       "link_name                                                 ...   \n",
       "the-berkeley-alt.-protein-project               0.931035  ...   \n",
       "ethical-apparel                                 0.962386  ...   \n",
       "uc-rally-committee                              0.967880  ...   \n",
       "formula-electric-at-berkeley                    0.927266  ...   \n",
       "codebase                                        0.970006  ...   \n",
       "...                                                  ...  ...   \n",
       "unicef-at-berkeley                              0.962485  ...   \n",
       "volunteers-around-the-world                     0.967990  ...   \n",
       "womxn-ignite-berkeley                           0.952025  ...   \n",
       "delta-consulting                                0.969046  ...   \n",
       "biofuels-technology-club                        0.947665  ...   \n",
       "\n",
       "                                   nextgen-consulting  bare-magazine  \\\n",
       "link_name                                                              \n",
       "the-berkeley-alt.-protein-project            0.946223       0.059047   \n",
       "ethical-apparel                              0.993335       0.020248   \n",
       "uc-rally-committee                           0.978608       0.034249   \n",
       "formula-electric-at-berkeley                 0.944117       0.050537   \n",
       "codebase                                     0.981242       0.059115   \n",
       "...                                               ...            ...   \n",
       "unicef-at-berkeley                           0.987128      -0.019417   \n",
       "volunteers-around-the-world                  0.989707       0.027456   \n",
       "womxn-ignite-berkeley                        0.991496       0.016586   \n",
       "delta-consulting                             0.992949       0.032679   \n",
       "biofuels-technology-club                     0.981205      -0.008870   \n",
       "\n",
       "                                   phi-delta-epsilon  \\\n",
       "link_name                                              \n",
       "the-berkeley-alt.-protein-project           0.936881   \n",
       "ethical-apparel                             0.993228   \n",
       "uc-rally-committee                          0.976043   \n",
       "formula-electric-at-berkeley                0.937625   \n",
       "codebase                                    0.978218   \n",
       "...                                              ...   \n",
       "unicef-at-berkeley                          0.981799   \n",
       "volunteers-around-the-world                 0.986948   \n",
       "womxn-ignite-berkeley                       0.994340   \n",
       "delta-consulting                            0.987620   \n",
       "biofuels-technology-club                    0.975230   \n",
       "\n",
       "                                   underwater-robotics-at-berkeley  \\\n",
       "link_name                                                            \n",
       "the-berkeley-alt.-protein-project                         0.932828   \n",
       "ethical-apparel                                           0.986603   \n",
       "uc-rally-committee                                        0.971811   \n",
       "formula-electric-at-berkeley                              0.954711   \n",
       "codebase                                                  0.972722   \n",
       "...                                                            ...   \n",
       "unicef-at-berkeley                                        0.975986   \n",
       "volunteers-around-the-world                               0.980247   \n",
       "womxn-ignite-berkeley                                     0.989475   \n",
       "delta-consulting                                          0.989393   \n",
       "biofuels-technology-club                                  0.969630   \n",
       "\n",
       "                                   reach-asian-and-pacific-islander-recruitment-and-retention-center  \\\n",
       "link_name                                                                                              \n",
       "the-berkeley-alt.-protein-project                                           0.937526                   \n",
       "ethical-apparel                                                             0.992584                   \n",
       "uc-rally-committee                                                          0.968250                   \n",
       "formula-electric-at-berkeley                                                0.935886                   \n",
       "codebase                                                                    0.975969                   \n",
       "...                                                                              ...                   \n",
       "unicef-at-berkeley                                                          0.981096                   \n",
       "volunteers-around-the-world                                                 0.982695                   \n",
       "womxn-ignite-berkeley                                                       0.989725                   \n",
       "delta-consulting                                                            0.993019                   \n",
       "biofuels-technology-club                                                    0.971816                   \n",
       "\n",
       "                                   unicef-at-berkeley  \\\n",
       "link_name                                               \n",
       "the-berkeley-alt.-protein-project            0.941223   \n",
       "ethical-apparel                              0.982321   \n",
       "uc-rally-committee                           0.978412   \n",
       "formula-electric-at-berkeley                 0.939227   \n",
       "codebase                                     0.971463   \n",
       "...                                               ...   \n",
       "unicef-at-berkeley                           1.000000   \n",
       "volunteers-around-the-world                  0.982877   \n",
       "womxn-ignite-berkeley                        0.983824   \n",
       "delta-consulting                             0.979850   \n",
       "biofuels-technology-club                     0.989599   \n",
       "\n",
       "                                   volunteers-around-the-world  \\\n",
       "link_name                                                        \n",
       "the-berkeley-alt.-protein-project                     0.936032   \n",
       "ethical-apparel                                       0.984404   \n",
       "uc-rally-committee                                    0.981915   \n",
       "formula-electric-at-berkeley                          0.937696   \n",
       "codebase                                              0.979377   \n",
       "...                                                        ...   \n",
       "unicef-at-berkeley                                    0.982877   \n",
       "volunteers-around-the-world                           1.000000   \n",
       "womxn-ignite-berkeley                                 0.982263   \n",
       "delta-consulting                                      0.986885   \n",
       "biofuels-technology-club                              0.976712   \n",
       "\n",
       "                                   womxn-ignite-berkeley  delta-consulting  \\\n",
       "link_name                                                                    \n",
       "the-berkeley-alt.-protein-project               0.927831          0.937292   \n",
       "ethical-apparel                                 0.992591          0.992389   \n",
       "uc-rally-committee                              0.968939          0.975256   \n",
       "formula-electric-at-berkeley                    0.929891          0.953451   \n",
       "codebase                                        0.970345          0.980653   \n",
       "...                                                  ...               ...   \n",
       "unicef-at-berkeley                              0.983824          0.979850   \n",
       "volunteers-around-the-world                     0.982263          0.986885   \n",
       "womxn-ignite-berkeley                           1.000000          0.986532   \n",
       "delta-consulting                                0.986532          1.000000   \n",
       "biofuels-technology-club                        0.977335          0.973482   \n",
       "\n",
       "                                   biofuels-technology-club  \n",
       "link_name                                                    \n",
       "the-berkeley-alt.-protein-project                  0.935098  \n",
       "ethical-apparel                                    0.978835  \n",
       "uc-rally-committee                                 0.967887  \n",
       "formula-electric-at-berkeley                       0.929127  \n",
       "codebase                                           0.959684  \n",
       "...                                                     ...  \n",
       "unicef-at-berkeley                                 0.989599  \n",
       "volunteers-around-the-world                        0.976712  \n",
       "womxn-ignite-berkeley                              0.977335  \n",
       "delta-consulting                                   0.973482  \n",
       "biofuels-technology-club                           1.000000  \n",
       "\n",
       "[286 rows x 286 columns]"
      ]
     },
     "execution_count": 485,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distance_table = create_distance_table(vector_table)\n",
    "distance_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 486,
   "metadata": {},
   "outputs": [],
   "source": [
    "#STEP 4: RECOMMEND\n",
    "def recommend(table, club_db_df, club_name, k):\n",
    "    \"\"\"\n",
    "    Description:\n",
    "    Recommends club based off of k-nearest neighbors, prioritizing matching tags.\n",
    "    \n",
    "    Input:\n",
    "    table - table containing all distances between each club\n",
    "    club_db_df - table containing club tags\n",
    "    club_name - string of club name we want to create recommendations for\n",
    "    k - represents how many neighbors\n",
    "    \n",
    "    Output:\n",
    "    recommendations - k recommendations based off of closest distances\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    def filter_by_tag(club_link_name, k):\n",
    "        \"\"\"\n",
    "        Description:\n",
    "        Return boolean list that contains max amount of matching tags that satisfies matching k amount of clubs.\n",
    "\n",
    "        Output:\n",
    "        filtered_clubs - list of booleans based off of clubs filtered by tags\n",
    "\n",
    "        \"\"\"\n",
    "        \n",
    "        def count_tags(a, b, num_tags):\n",
    "            \"\"\"\n",
    "            Description:\n",
    "            Returns if club A and B contain at least k tags in common.\n",
    "\n",
    "            Input:\n",
    "            a - list of club tags for club A\n",
    "            b - list of club tags for club B\n",
    "            num_tags - minimum number of tags required to match\n",
    "\n",
    "            Output:\n",
    "            matching_tags - # of matching tags\n",
    "\n",
    "            \"\"\"\n",
    "            matching_tags = len(set(a).intersection(set(b))) >= num_tags\n",
    "            return matching_tags\n",
    "\n",
    "        club_tags = club_db_df[club_db_df['link_name'] == club_link_name]['tags'].iloc[0]\n",
    "        filtered_clubs = []\n",
    "\n",
    "        num_of_tags = len(club_tags)\n",
    "        while (np.count_nonzero(filtered_clubs) - 1 < k):\n",
    "            if num_of_tags == 0:\n",
    "                print(\"Matched {} tags\".format(num_of_tags))\n",
    "                return [True] * len(club_db_df)\n",
    "\n",
    "            filtered_clubs = []\n",
    "            for other_club_tags in club_db_df['tags']:\n",
    "                if count_tags(club_tags, other_club_tags, num_of_tags):\n",
    "                    filtered_clubs += [True]\n",
    "                else:\n",
    "                    filtered_clubs += [False]\n",
    "            num_of_tags -= 1\n",
    "\n",
    "        else:\n",
    "            # print(\"Matched {} tags\".format(num_of_tags + 1))\n",
    "            # print(\"{} matches\".format(np.count_nonzero(filtered_clubs)))\n",
    "            return filtered_clubs\n",
    "    \n",
    "    filtered_clubs = filter_by_tag(club_name, k)\n",
    "        \n",
    "    filtered_distances = table[filtered_clubs]\n",
    "    club_distances = filtered_distances[club_name] \n",
    "    sorted_club_distances = club_distances.sort_values(ascending = True, na_position = 'last')\n",
    "    recommendations = sorted_club_distances.keys()[1: k + 1]\n",
    "    \n",
    "    return list(recommendations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 487,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['plextech', 'datastory', 'data-science-society']"
      ]
     },
     "execution_count": 487,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommendations = recommend(distance_table, club_db_df, 'big-data-at-berkeley', 3)\n",
    "recommendations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### APPLYING IT TO OUR COLLAB FILTERING"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 521,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['latinx-student-association-at-berkeley',\n",
       " 'berkeley-chinese-students-and-scholars-association',\n",
       " 'ra-on',\n",
       " 'ballet-folklorico-reflejos-de-mexico',\n",
       " 'team-hbv-']"
      ]
     },
     "execution_count": 521,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def fill_with_generic(favorites_list, interests_list, relevant_clubs_list):\n",
    "    '''\n",
    "    For each row, we want to fill relevant_clubs with randomly sampled top recs for each club in favorites list\n",
    "    edge case: if there is no favorites list, fill with clubs within interested tags\n",
    "    Users with no interests will not need fill because all clubs are relevant according to relevance function\n",
    "    '''\n",
    "    new_recs = []\n",
    "    #set final recs to relevant_clubs link_names\n",
    "    final_recs = [list(club_db_df[club_db_df['name'] == i]['link_name'])[0] for i in relevant_clubs_list]\n",
    "    \n",
    "    #If there are less than 5 relevant clubs, then fill with generic recs\n",
    "    if len(relevant_clubs_list) < 5:\n",
    "        \n",
    "             \n",
    "        if len(list(favorites_list)) != 0:\n",
    "            #for each favorited club, recommend 5 clubs (to cover case with 1 favorite)\n",
    "            for fav_club in favorites_list:\n",
    "                linkname = list(club_db_df[club_db_df['name'] == fav_club]['link_name'])[0]\n",
    "                rec = recommend(distance_table, club_db_df, linkname, 5)\n",
    "                new_recs += rec\n",
    "\n",
    "            #make sure there are no duplicate clubs\n",
    "            unique_new = list(set(new_recs))\n",
    "            new_not_sim = list(np.setdiff1d(unique_new,final_recs))\n",
    "\n",
    "\n",
    "            #for each empty space in relevant_clubs\n",
    "            num_sampled = 5-len(relevant_clubs_list)\n",
    "            final_recs += random.sample(new_not_sim, num_sampled)\n",
    "\n",
    "        #if there is no favorites list, suggest random clubs within interests\n",
    "        else:\n",
    "            for i in interests_list:\n",
    "                clubs_w_interests = []\n",
    "\n",
    "                #for each club\n",
    "                for club in range(len(club_db_df)):\n",
    "\n",
    "                    #if interest tag is equal to the club tags\n",
    "                    if i in club_db_df['tags'][club]:\n",
    "                        clubs_w_interests += [club_db_df['link_name'][club]]\n",
    "                        \n",
    "                #make sure no duplicates    \n",
    "                unique_new = list(set(clubs_w_interests))\n",
    "                new_not_sim = list(np.setdiff1d(unique_new,final_recs))\n",
    "\n",
    "            num_sampled = 5-len(relevant_clubs_list)\n",
    "            final_recs += random.sample(new_not_sim, num_sampled)\n",
    "            \n",
    "    return final_recs\n",
    "    \n",
    "        \n",
    "fill_with_generic(relevant_recs['favorites'][41], relevant_recs['interests'][41], relevant_recs['relevant_clubs'][41])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 522,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>favorites</th>\n",
       "      <th>recs</th>\n",
       "      <th>interests</th>\n",
       "      <th>relevant_clubs</th>\n",
       "      <th>filled_relevant_clubs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>[Cal Seismic Design Team, Berkeley Innovation,...</td>\n",
       "      <td>[15]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[caliber-magazine, caravan-magazine, the-golde...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>(Web Development at Berkeley,)</td>\n",
       "      <td>[Berkeley Innovation, Berkeley Legends, SENDfo...</td>\n",
       "      <td>[22]</td>\n",
       "      <td>[Berkeley Legends]</td>\n",
       "      <td>[berkeley-legends, upsilon-pi-epsilon-(upe), d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>(DataStory, Berkeley Phi Beta Lambda, The Berk...</td>\n",
       "      <td>[Cal Dragon Boat, Cal Triathlon, UC Rally Comm...</td>\n",
       "      <td>[6]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[minecraft-at-berkeley, the-berkeley-group, mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>(Software and Hardware Recruiting List, Politi...</td>\n",
       "      <td>[Extended Reality at Berkeley, REACH! Asian an...</td>\n",
       "      <td>[22]</td>\n",
       "      <td>[Extended Reality at Berkeley, FemTech]</td>\n",
       "      <td>[extended-reality-at-berkeley, femtech, mdb, c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>(Cal RoboBears, Latinx Student Association at ...</td>\n",
       "      <td>[Launchpad, Cal Triathlon, UC Rally Committee,...</td>\n",
       "      <td>[13, 22, 7]</td>\n",
       "      <td>[Launchpad, Blockchain at Berkeley, Political ...</td>\n",
       "      <td>[launchpad, blockchain-at-berkeley, political-...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                                          favorites  \\\n",
       "0        0                          (Her Campus at Berkeley,)   \n",
       "1        1                     (Web Development at Berkeley,)   \n",
       "2        2  (DataStory, Berkeley Phi Beta Lambda, The Berk...   \n",
       "3        3  (Software and Hardware Recruiting List, Politi...   \n",
       "4        4  (Cal RoboBears, Latinx Student Association at ...   \n",
       "\n",
       "                                                recs    interests  \\\n",
       "0  [Cal Seismic Design Team, Berkeley Innovation,...         [15]   \n",
       "1  [Berkeley Innovation, Berkeley Legends, SENDfo...         [22]   \n",
       "2  [Cal Dragon Boat, Cal Triathlon, UC Rally Comm...          [6]   \n",
       "3  [Extended Reality at Berkeley, REACH! Asian an...         [22]   \n",
       "4  [Launchpad, Cal Triathlon, UC Rally Committee,...  [13, 22, 7]   \n",
       "\n",
       "                                      relevant_clubs  \\\n",
       "0                                                 []   \n",
       "1                                 [Berkeley Legends]   \n",
       "2                                                 []   \n",
       "3            [Extended Reality at Berkeley, FemTech]   \n",
       "4  [Launchpad, Blockchain at Berkeley, Political ...   \n",
       "\n",
       "                               filled_relevant_clubs  \n",
       "0  [caliber-magazine, caravan-magazine, the-golde...  \n",
       "1  [berkeley-legends, upsilon-pi-epsilon-(upe), d...  \n",
       "2  [minecraft-at-berkeley, the-berkeley-group, mi...  \n",
       "3  [extended-reality-at-berkeley, femtech, mdb, c...  \n",
       "4  [launchpad, blockchain-at-berkeley, political-...  "
      ]
     },
     "execution_count": 522,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relevant_recs['filled_relevant_clubs'] = relevant_recs.apply(lambda row : fill_with_generic(row['favorites'], row['interests'], row['relevant_clubs']), axis = 1)\n",
    "relevant_recs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feasibility Test: do recommendations look related to simulated favorites?\n",
    "All recommendations will be within a user's interests!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 523,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Favorites:  (\"American Medical Women's Association (AMWA) at Berkeley\", 'Ballet Folklorico Reflejos de Mexico')  Recommendations:  ['latinx-student-association-at-berkeley', 'mcbcdna', 'ra-on', 'peer-health-exchange', 'berkeley-chinese-students-and-scholars-association']\n"
     ]
    }
   ],
   "source": [
    "#club2 = random.randint(0,len(sample_users))\n",
    "#print (club2)\n",
    "print('Favorites: ', relevant_recs['favorites'][club2],' Recommendations: ', relevant_recs['filled_relevant_clubs'][club2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 525,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Advocacy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>ASUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Business</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>CalGreek</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Community Service</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>Computer Science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>Consulting</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>Cultural</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>Design</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>Engineering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10</td>\n",
       "      <td>Environmental</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>11</td>\n",
       "      <td>Health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>12</td>\n",
       "      <td>Media</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>13</td>\n",
       "      <td>Performing Arts</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>14</td>\n",
       "      <td>Political</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>15</td>\n",
       "      <td>Pre-professional</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>16</td>\n",
       "      <td>Religious &amp; Spiritual</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>17</td>\n",
       "      <td>Research</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>18</td>\n",
       "      <td>Sciences</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>19</td>\n",
       "      <td>Social</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>20</td>\n",
       "      <td>Social Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>21</td>\n",
       "      <td>Sports &amp; Rec.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>22</td>\n",
       "      <td>Technology</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>23</td>\n",
       "      <td>Publication</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id                   name\n",
       "0    0               Advocacy\n",
       "1    1                   ASUC\n",
       "2    2               Business\n",
       "3    3               CalGreek\n",
       "4    4      Community Service\n",
       "5    5       Computer Science\n",
       "6    6             Consulting\n",
       "7    7               Cultural\n",
       "8    8                 Design\n",
       "9    9            Engineering\n",
       "10  10          Environmental\n",
       "11  11                 Health\n",
       "12  12                  Media\n",
       "13  13        Performing Arts\n",
       "14  14              Political\n",
       "15  15       Pre-professional\n",
       "16  16  Religious & Spiritual\n",
       "17  17               Research\n",
       "18  18               Sciences\n",
       "19  19                 Social\n",
       "20  20            Social Good\n",
       "21  21          Sports & Rec.\n",
       "22  22             Technology\n",
       "23  23            Publication"
      ]
     },
     "execution_count": 525,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_club_tags_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 526,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>favorites</th>\n",
       "      <th>interests</th>\n",
       "      <th>personalized recs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>(Her Campus at Berkeley,)</td>\n",
       "      <td>[15]</td>\n",
       "      <td>[caliber-magazine, caravan-magazine, the-golde...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>(Web Development at Berkeley,)</td>\n",
       "      <td>[22]</td>\n",
       "      <td>[berkeley-legends, upsilon-pi-epsilon-(upe), d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>(DataStory, Berkeley Phi Beta Lambda, The Berk...</td>\n",
       "      <td>[6]</td>\n",
       "      <td>[minecraft-at-berkeley, the-berkeley-group, mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>(Software and Hardware Recruiting List, Politi...</td>\n",
       "      <td>[22]</td>\n",
       "      <td>[extended-reality-at-berkeley, femtech, mdb, c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>(Cal RoboBears, Latinx Student Association at ...</td>\n",
       "      <td>[13, 22, 7]</td>\n",
       "      <td>[launchpad, blockchain-at-berkeley, political-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>95</td>\n",
       "      <td>(Azaad Premiere Dance,)</td>\n",
       "      <td>[7]</td>\n",
       "      <td>[latinx-student-association-at-berkeley, ra-on...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>96</td>\n",
       "      <td>(Perennial: The Undergraduate Environmental Jo...</td>\n",
       "      <td>[18, 0]</td>\n",
       "      <td>[student-premed-advising-network-[span], pre-p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>97</td>\n",
       "      <td>(Peer Health Exchange,)</td>\n",
       "      <td>[14, 0]</td>\n",
       "      <td>[united-nations-association-of-berkeley, latin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>98</td>\n",
       "      <td>(ATP at Berkeley, Phoenix Consulting Group, SE...</td>\n",
       "      <td>[15, 4]</td>\n",
       "      <td>[mcbcdna, atp-at-berkeley, uc-berkeley-nsslha,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>99</td>\n",
       "      <td>(United Nations Association of Berkeley,)</td>\n",
       "      <td>[0, 15, 13]</td>\n",
       "      <td>[peer-health-exchange, student-premed-advising...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    user_id                                          favorites    interests  \\\n",
       "0         0                          (Her Campus at Berkeley,)         [15]   \n",
       "1         1                     (Web Development at Berkeley,)         [22]   \n",
       "2         2  (DataStory, Berkeley Phi Beta Lambda, The Berk...          [6]   \n",
       "3         3  (Software and Hardware Recruiting List, Politi...         [22]   \n",
       "4         4  (Cal RoboBears, Latinx Student Association at ...  [13, 22, 7]   \n",
       "..      ...                                                ...          ...   \n",
       "95       95                            (Azaad Premiere Dance,)          [7]   \n",
       "96       96  (Perennial: The Undergraduate Environmental Jo...      [18, 0]   \n",
       "97       97                            (Peer Health Exchange,)      [14, 0]   \n",
       "98       98  (ATP at Berkeley, Phoenix Consulting Group, SE...      [15, 4]   \n",
       "99       99          (United Nations Association of Berkeley,)  [0, 15, 13]   \n",
       "\n",
       "                                    personalized recs  \n",
       "0   [caliber-magazine, caravan-magazine, the-golde...  \n",
       "1   [berkeley-legends, upsilon-pi-epsilon-(upe), d...  \n",
       "2   [minecraft-at-berkeley, the-berkeley-group, mi...  \n",
       "3   [extended-reality-at-berkeley, femtech, mdb, c...  \n",
       "4   [launchpad, blockchain-at-berkeley, political-...  \n",
       "..                                                ...  \n",
       "95  [latinx-student-association-at-berkeley, ra-on...  \n",
       "96  [student-premed-advising-network-[span], pre-p...  \n",
       "97  [united-nations-association-of-berkeley, latin...  \n",
       "98  [mcbcdna, atp-at-berkeley, uc-berkeley-nsslha,...  \n",
       "99  [peer-health-exchange, student-premed-advising...  \n",
       "\n",
       "[100 rows x 4 columns]"
      ]
     },
     "execution_count": 526,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_personalized_recs = relevant_recs.drop(['recs', 'relevant_clubs'], axis=1).rename({'filled_relevant_clubs':'personalized recs'}, axis=1)\n",
    "final_personalized_recs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}