line_of_thought.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AIML Engineer Take-Home Project\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise Info and Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objective\n",
    "You are tasked with building a recommendation system using a provided public dataset,\n",
    "focusing on both classical recommendation models and modern LLM-based approaches.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Project Requirements\n",
    "Package your application as a service using a suitable framework, e.g. FastAPI or Flask. We\n",
    "are intentionally not being prescriptive here, but do your best to demonstrate your\n",
    "understanding of best practices when building your solution.\n",
    "1. Dataset\n",
    "  - Use the [Anime Recommendations Database from Kaggle](https://www.kaggle.com/datasets/CooperUnion/anime-recommendations-database). This dataset contains\n",
    "information on various anime, including user ratings, genres, and other relevant\n",
    "attributes.\n",
    "2. Data Endpoint\n",
    "  - Your service should have an API endpoint to query the dataset\n",
    "3. Classical Recommendation System\n",
    "  - Build a recommendation system that utilizes the dataset to suggest top k anime\n",
    "for a user based on their viewing history and preferences.\n",
    "  - Ensure that the recommendation logic excludes recently viewed items (e.g.,\n",
    "anime watched within the last 7 days).\n",
    "4. Contextual LLM-Based Personalization\n",
    "  - Implement a feature where users can get personalized anime recommendations\n",
    "based on a natural language description of their current mood or preferences\n",
    "(e.g., \"I want something uplifting and adventurous\").\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Submission\n",
    "Submit a single .zip file that includes:\n",
    "- all source code\n",
    "- A System Design doc\n",
    "\n",
    "  - Provide a document (e.g. `SYSTEM_DESIGN.md`) that explains your choices and\n",
    "the architecture.\n",
    "  - Discuss how you would extend the current system to make it more accurate in\n",
    "response to more “vague” user input\n",
    "  - Include recommendations of how to transition this to an LLM deployed in-house\n",
    "- A Presentation Slide Deck to present your project for 30 mins in during a panel interview\n",
    "  - Can include information from the System Design doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandas in ./recapp/lib/python3.9/site-packages (2.2.2)\n",
      "Requirement already satisfied: numpy>=1.22.4 in ./recapp/lib/python3.9/site-packages (from pandas) (2.0.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in ./recapp/lib/python3.9/site-packages (from pandas) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in ./recapp/lib/python3.9/site-packages (from pandas) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in ./recapp/lib/python3.9/site-packages (from pandas) (2024.1)\n",
      "Requirement already satisfied: six>=1.5 in ./recapp/lib/python3.9/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
      "Requirement already satisfied: numpy in ./recapp/lib/python3.9/site-packages (2.0.1)\n",
      "Requirement already satisfied: matplotlib in ./recapp/lib/python3.9/site-packages (3.9.2)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in ./recapp/lib/python3.9/site-packages (from matplotlib) (1.2.1)\n",
      "Requirement already satisfied: cycler>=0.10 in ./recapp/lib/python3.9/site-packages (from matplotlib) (0.12.1)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in ./recapp/lib/python3.9/site-packages (from matplotlib) (4.53.1)\n",
      "Requirement already satisfied: kiwisolver>=1.3.1 in ./recapp/lib/python3.9/site-packages (from matplotlib) (1.4.5)\n",
      "Requirement already satisfied: numpy>=1.23 in ./recapp/lib/python3.9/site-packages (from matplotlib) (2.0.1)\n",
      "Requirement already satisfied: packaging>=20.0 in ./recapp/lib/python3.9/site-packages (from matplotlib) (24.1)\n",
      "Requirement already satisfied: pillow>=8 in ./recapp/lib/python3.9/site-packages (from matplotlib) (10.4.0)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in ./recapp/lib/python3.9/site-packages (from matplotlib) (3.1.2)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in ./recapp/lib/python3.9/site-packages (from matplotlib) (2.9.0.post0)\n",
      "Requirement already satisfied: importlib-resources>=3.2.0 in ./recapp/lib/python3.9/site-packages (from matplotlib) (6.4.4)\n",
      "Requirement already satisfied: zipp>=3.1.0 in ./recapp/lib/python3.9/site-packages (from importlib-resources>=3.2.0->matplotlib) (3.20.0)\n",
      "Requirement already satisfied: six>=1.5 in ./recapp/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\n",
      "Collecting scikit-learn\n",
      "  Downloading scikit_learn-1.5.1-cp39-cp39-macosx_12_0_arm64.whl.metadata (12 kB)\n",
      "Requirement already satisfied: numpy>=1.19.5 in ./recapp/lib/python3.9/site-packages (from scikit-learn) (2.0.1)\n",
      "Collecting scipy>=1.6.0 (from scikit-learn)\n",
      "  Downloading scipy-1.13.1-cp39-cp39-macosx_12_0_arm64.whl.metadata (60 kB)\n",
      "Collecting joblib>=1.2.0 (from scikit-learn)\n",
      "  Downloading joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)\n",
      "Collecting threadpoolctl>=3.1.0 (from scikit-learn)\n",
      "  Downloading threadpoolctl-3.5.0-py3-none-any.whl.metadata (13 kB)\n",
      "Downloading scikit_learn-1.5.1-cp39-cp39-macosx_12_0_arm64.whl (11.0 MB)\n",
      "\u001b[2K   \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.0/11.0 MB\u001b[0m \u001b[31m19.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m20.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading joblib-1.4.2-py3-none-any.whl (301 kB)\n",
      "Downloading scipy-1.13.1-cp39-cp39-macosx_12_0_arm64.whl (30.3 MB)\n",
      "\u001b[2K   \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m30.3/30.3 MB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m:02\u001b[0m\n",
      "\u001b[?25hDownloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB)\n",
      "Installing collected packages: threadpoolctl, scipy, joblib, scikit-learn\n",
      "Successfully installed joblib-1.4.2 scikit-learn-1.5.1 scipy-1.13.1 threadpoolctl-3.5.0\n"
     ]
    }
   ],
   "source": [
    "!pip install pandas\n",
    "!pip install numpy\n",
    "!pip install matplotlib\n",
    "# !pip install seaborn\n",
    "!pip install scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First ideas\n",
    "\n",
    "My first idea is to use `NMF`, but let's first find out the current state of the art (SOTA) for recommender systems.\n",
    "\n",
    "#### Current state of the art\n",
    "\n",
    "There is not a single best algorithm for recommender systems. Solutions for the Netflix Prize include[^1][^2]:\n",
    "- Decomposition models (SVD, NMF, SVD++, etc.)\n",
    "- RBM\n",
    "- Decision Tree-based methods (Gradient Boosted Decision Trees, etc.)\n",
    "- Neural Networks\n",
    "- SVM\n",
    "\n",
    "Also, it is common to use a blend of models, like BellKor Solution to the Netflix Grand Prize[^3], which makes sense as many models tend to improve with ensemble methods.\n",
    "\n",
    "#### Way to go\n",
    "\n",
    "Considering this and the escope of this project, I will test simple models.\n",
    "\n",
    "\n",
    "[^1]: Stephen Gower. [Netflix Prize and SVD](http://buzzard.ups.edu/courses/2014spring/420projects/math420-UPS-spring-2014-gower-netflix-SVD.pdf). April 18th 2014 \n",
    "\n",
    "[^2]: [Netflix Recommendations: Beyond the 5 stars (Part 2)](https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-2-d9b96aa399f5)\n",
    "\n",
    "[^3]: [The BellKor Solution to the Netflix Grand Prize](https://www2.seas.gwu.edu/~simhaweb/champalg/cf/papers/KorenBellKor2009.pdf)\n",
    "\n",
    "Considering this and the escope of this project, I will use the NMF algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature engineering\n",
    "\n",
    "Some ideas of features to be used in the model:\n",
    "- User's movie ratings\n",
    "- User's movie watching history\n",
    "- User's movie type history\n",
    "- User's movie episode size\n",
    "- Movie's average rating\n",
    "- Movie's popularity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "# from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "# from sklearn.pipeline import Pipeline\n",
    "# from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ratings:\n",
      "   user_id  anime_id  rating\n",
      "0        1        20      -1\n",
      "1        1        24      -1\n",
      "2        1        79      -1\n",
      "3        1       226      -1\n",
      "4        1       241      -1\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 7813737 entries, 0 to 7813736\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Dtype\n",
      "---  ------    -----\n",
      " 0   user_id   int64\n",
      " 1   anime_id  int64\n",
      " 2   rating    int64\n",
      "dtypes: int64(3)\n",
      "memory usage: 178.8 MB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>anime_id</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>7.813737e+06</td>\n",
       "      <td>7.813737e+06</td>\n",
       "      <td>7.813737e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.672796e+04</td>\n",
       "      <td>8.909072e+03</td>\n",
       "      <td>6.144030e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.099795e+04</td>\n",
       "      <td>8.883950e+03</td>\n",
       "      <td>3.727800e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>-1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.897400e+04</td>\n",
       "      <td>1.240000e+03</td>\n",
       "      <td>6.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.679100e+04</td>\n",
       "      <td>6.213000e+03</td>\n",
       "      <td>7.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>5.475700e+04</td>\n",
       "      <td>1.409300e+04</td>\n",
       "      <td>9.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>7.351600e+04</td>\n",
       "      <td>3.451900e+04</td>\n",
       "      <td>1.000000e+01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            user_id      anime_id        rating\n",
       "count  7.813737e+06  7.813737e+06  7.813737e+06\n",
       "mean   3.672796e+04  8.909072e+03  6.144030e+00\n",
       "std    2.099795e+04  8.883950e+03  3.727800e+00\n",
       "min    1.000000e+00  1.000000e+00 -1.000000e+00\n",
       "25%    1.897400e+04  1.240000e+03  6.000000e+00\n",
       "50%    3.679100e+04  6.213000e+03  7.000000e+00\n",
       "75%    5.475700e+04  1.409300e+04  9.000000e+00\n",
       "max    7.351600e+04  3.451900e+04  1.000000e+01"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import ratings and anime data\n",
    "ratings = pd.read_csv('data/rating.csv')\n",
    "anime = pd.read_csv('data/anime.csv')\n",
    "\n",
    "# Explore ratings dataframe\n",
    "print(\"Ratings:\")\n",
    "print(ratings.head())\n",
    "ratings.info()\n",
    "ratings.describe()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Anime DataFrame Info:\n",
      "   anime_id                              name  \\\n",
      "0     32281                    Kimi no Na wa.   \n",
      "1      5114  Fullmetal Alchemist: Brotherhood   \n",
      "2     28977                          Gintama°   \n",
      "3      9253                       Steins;Gate   \n",
      "4      9969                     Gintama&#039;   \n",
      "\n",
      "                                               genre   type episodes  rating  \\\n",
      "0               Drama, Romance, School, Supernatural  Movie        1    9.37   \n",
      "1  Action, Adventure, Drama, Fantasy, Magic, Mili...     TV       64    9.26   \n",
      "2  Action, Comedy, Historical, Parody, Samurai, S...     TV       51    9.25   \n",
      "3                                   Sci-Fi, Thriller     TV       24    9.17   \n",
      "4  Action, Comedy, Historical, Parody, Samurai, S...     TV       51    9.16   \n",
      "\n",
      "   members  \n",
      "0   200630  \n",
      "1   793665  \n",
      "2   114262  \n",
      "3   673572  \n",
      "4   151266  \n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 12294 entries, 0 to 12293\n",
      "Data columns (total 7 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   anime_id  12294 non-null  int64  \n",
      " 1   name      12294 non-null  object \n",
      " 2   genre     12232 non-null  object \n",
      " 3   type      12269 non-null  object \n",
      " 4   episodes  12294 non-null  object \n",
      " 5   rating    12064 non-null  float64\n",
      " 6   members   12294 non-null  int64  \n",
      "dtypes: float64(1), int64(2), object(4)\n",
      "memory usage: 672.5+ KB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anime_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>members</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>12294.000000</td>\n",
       "      <td>12064.000000</td>\n",
       "      <td>1.229400e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>14058.221653</td>\n",
       "      <td>6.473902</td>\n",
       "      <td>1.807134e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>11455.294701</td>\n",
       "      <td>1.026746</td>\n",
       "      <td>5.482068e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.670000</td>\n",
       "      <td>5.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>3484.250000</td>\n",
       "      <td>5.880000</td>\n",
       "      <td>2.250000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>10260.500000</td>\n",
       "      <td>6.570000</td>\n",
       "      <td>1.550000e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>24794.500000</td>\n",
       "      <td>7.180000</td>\n",
       "      <td>9.437000e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>34527.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>1.013917e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           anime_id        rating       members\n",
       "count  12294.000000  12064.000000  1.229400e+04\n",
       "mean   14058.221653      6.473902  1.807134e+04\n",
       "std    11455.294701      1.026746  5.482068e+04\n",
       "min        1.000000      1.670000  5.000000e+00\n",
       "25%     3484.250000      5.880000  2.250000e+02\n",
       "50%    10260.500000      6.570000  1.550000e+03\n",
       "75%    24794.500000      7.180000  9.437000e+03\n",
       "max    34527.000000     10.000000  1.013917e+06"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Explore anime dataframe\n",
    "print(\"Anime DataFrame Info:\")\n",
    "print(anime.head())\n",
    "anime.info()\n",
    "anime.describe()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['1' '64' '51' '24' '10' '148' '110' '13' '201' '25' '22' '75' '4' '26'\n",
      " '12' '27' '43' '74' '37' '2' '11' '99' 'Unknown' '39' '101' '47' '50'\n",
      " '62' '33' '112' '23' '3' '94' '6' '8' '14' '7' '40' '15' '203' '77' '291'\n",
      " '120' '102' '96' '38' '79' '175' '103' '70' '153' '45' '5' '21' '63' '52'\n",
      " '28' '145' '36' '69' '60' '178' '114' '35' '61' '34' '109' '20' '9' '49'\n",
      " '366' '97' '48' '78' '358' '155' '104' '113' '54' '167' '161' '42' '142'\n",
      " '31' '373' '220' '46' '195' '17' '1787' '73' '147' '127' '16' '19' '98'\n",
      " '150' '76' '53' '124' '29' '115' '224' '44' '58' '93' '154' '92' '67'\n",
      " '172' '86' '30' '276' '59' '72' '330' '41' '105' '128' '137' '56' '55'\n",
      " '65' '243' '193' '18' '191' '180' '91' '192' '66' '182' '32' '164' '100'\n",
      " '296' '694' '95' '68' '117' '151' '130' '87' '170' '119' '84' '108' '156'\n",
      " '140' '331' '305' '300' '510' '200' '88' '1471' '526' '143' '726' '136'\n",
      " '1818' '237' '1428' '365' '163' '283' '71' '260' '199' '225' '312' '240'\n",
      " '1306' '1565' '773' '1274' '90' '475' '263' '83' '85' '1006' '80' '162'\n",
      " '132' '141' '125']\n",
      "['Movie' 'TV' 'OVA' 'Special' 'Music' 'ONA' nan]\n"
     ]
    }
   ],
   "source": [
    "print(anime['episodes'].unique())\n",
    "print(anime['type'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Null counts in Ratings DataFrame: \n",
      "user_id     0\n",
      "anime_id    0\n",
      "rating      0\n",
      "dtype: int64 out of 7813737\n",
      "\n",
      "Null counts in Anime DataFrame: \n",
      "anime_id      0\n",
      "name          0\n",
      "genre        62\n",
      "type         25\n",
      "episodes      0\n",
      "rating      230\n",
      "members       0\n",
      "dtype: int64 out of 12294\n"
     ]
    }
   ],
   "source": [
    "# Count nulls in ratings_df\n",
    "print(f\"Null counts in Ratings DataFrame: \\n{ratings.isnull().sum()} out of {len(ratings)}\")\n",
    "\n",
    "# Count nulls in anime\n",
    "print(f\"\\nNull counts in Anime DataFrame: \\n{anime.isnull().sum()} out of {len(anime)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated anime DataFrame info:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anime_id</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>members</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>12294.000000</td>\n",
       "      <td>12294.000000</td>\n",
       "      <td>12294.000000</td>\n",
       "      <td>1.229400e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>14058.221653</td>\n",
       "      <td>12.012445</td>\n",
       "      <td>6.334078</td>\n",
       "      <td>1.807134e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>11455.294701</td>\n",
       "      <td>46.264788</td>\n",
       "      <td>1.435287</td>\n",
       "      <td>5.482068e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>5.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>3484.250000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5.820000</td>\n",
       "      <td>2.250000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>10260.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>6.550000</td>\n",
       "      <td>1.550000e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>24794.500000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>7.170000</td>\n",
       "      <td>9.437000e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>34527.000000</td>\n",
       "      <td>1818.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>1.013917e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           anime_id      episodes        rating       members\n",
       "count  12294.000000  12294.000000  12294.000000  1.229400e+04\n",
       "mean   14058.221653     12.012445      6.334078  1.807134e+04\n",
       "std    11455.294701     46.264788      1.435287  5.482068e+04\n",
       "min        1.000000     -1.000000     -1.000000  5.000000e+00\n",
       "25%     3484.250000      1.000000      5.820000  2.250000e+02\n",
       "50%    10260.500000      2.000000      6.550000  1.550000e+03\n",
       "75%    24794.500000     12.000000      7.170000  9.437000e+03\n",
       "max    34527.000000   1818.000000     10.000000  1.013917e+06"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Replace \"Unknown\" in episodes with -1 and convert to int\n",
    "anime['episodes'] = anime['episodes'].replace('Unknown', '-1')\n",
    "anime['episodes'] = anime['episodes'].astype(int)\n",
    "\n",
    "# Fill NaN genres and types with \"Unknown\"\n",
    "anime['genre'] = anime['genre'].fillna('Unknown')\n",
    "anime['type'] = anime['type'].fillna('Unknown')\n",
    "\n",
    "# Fill NaN ratings with -1\n",
    "anime['rating'] = anime['rating'].fillna(-1)\n",
    "\n",
    "# Verify changes\n",
    "print(\"Updated anime DataFrame info:\")\n",
    "anime.describe()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: ylabel='Frequency'>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjoAAAGdCAYAAAAbudkLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAApS0lEQVR4nO3dfVRV153/8c9FBSUBfCCCKIgxaoeo0KBQp5rGyATRZazOg01NRWNt02LrDDFVp2t0+jDFxgm1097ErDZIsjITrWuMmVUT24hmzAOJgqI1TIxYVFRAjQEEKyDs3x9Z3l9u8PFy4Vz2fb/Wumt59tlsvpvDw8dz9znHZYwxAgAAsFCI0wUAAAB0FYIOAACwFkEHAABYi6ADAACsRdABAADWIugAAABrEXQAAIC1CDoAAMBavZ0uwGnt7e06c+aMIiIi5HK5nC4HAADcAmOMLl68qLi4OIWEXP+8TdAHnTNnzig+Pt7pMgAAgA+qqqo0bNiw6+4P+qATEREh6dMvVGRkpMPVAACAW9HQ0KD4+HjP3/HrCfqgc/XtqsjISIIOAAA9zM2WnbAYGQAAWIugAwAArEXQAQAA1iLoAAAAaxF0AACAtQg6AADAWgQdAABgLYIOAACwFkEHAABYi6ADAACsRdABAADWIugAAABrEXQAAIC1gv7p5V0pceV2p0u4bcfXznS6BAAA/MaaMzqXLl3S8OHDtXz5cqdLAQAAAcKaoPNv//Zv+tKXvuR0GQAAIIBYEXSOHj2qDz/8UFlZWU6XAgAAAojjQWfPnj2aNWuW4uLi5HK5tG3btg593G63EhMT1bdvX6Wnp2vv3r1e+5cvX668vLxuqhgAAPQUjgedpqYmJScny+12X3P/5s2blZubqzVr1mj//v1KTk5WZmamzp49K0l69dVXNXr0aI0ePbo7ywYAAD2A41ddZWVl3fAtp/z8fC1ZskSLFi2SJG3YsEHbt29XQUGBVq5cqffee0+bNm3Sli1b1NjYqNbWVkVGRmr16tXXHK+5uVnNzc2e7YaGBv9OCAAABAzHz+jcSEtLi0pLS5WRkeFpCwkJUUZGhoqLiyVJeXl5qqqq0vHjx/Xv//7vWrJkyXVDztX+UVFRnld8fHyXzwMAADgjoIPO+fPn1dbWppiYGK/2mJgY1dTU+DTmqlWrVF9f73lVVVX5o1QAABCAHH/ryp8WLlx40z5hYWEKCwvr+mIAAIDjAvqMTnR0tHr16qXa2lqv9traWsXGxnZqbLfbraSkJE2cOLFT4wAAgMAV0EEnNDRUqampKioq8rS1t7erqKhIkyZN6tTYOTk5Ki8v1759+zpbJgAACFCOv3XV2NioiooKz3ZlZaXKyso0cOBAJSQkKDc3V9nZ2ZowYYLS0tK0fv16NTU1ea7CAgAAuB7Hg05JSYmmTp3q2c7NzZUkZWdnq7CwUPPmzdO5c+e0evVq1dTUKCUlRTt27OiwQBkAAODzXMYY43QRTmpoaFBUVJTq6+sVGRnp17F5ejkAAF3jVv9+B/Qana7EYmQAAOwXtEGHxcgAANgvaIMOAACwH0EHAABYK2iDDmt0AACwX9AGHdboAABgv6ANOgAAwH4EHQAAYC2CDgAAsBZBBwAAWCtogw5XXQEAYL+gDTpcdQUAgP2CNugAAAD7EXQAAIC1CDoAAMBaBB0AAGAtgg4AALBW0AYdLi8HAMB+QRt0uLwcAAD7BW3QAQAA9iPoAAAAaxF0AACAtQg6AADAWgQdAABgLYIOAACwVtAGHe6jAwCA/YI26HAfHQAA7Be0QQcAANiPoAMAAKxF0AEAANYi6AAAAGsRdAAAgLUIOgAAwFoEHQAAYC2CDgAAsFbQBh3ujAwAgP2CNuhwZ2QAAOwXtEEHAADYj6ADAACsRdABAADWIugAAABrEXQAAIC1CDoAAMBaBB0AAGAtgg4AALAWQQcAAFiLoAMAAKxF0AEAANYi6AAAAGsFbdDh6eUAANgvaIMOTy8HAMB+QRt0AACA/Qg6AADAWgQdAABgLYIOAACwFkEHAABYi6ADAACsRdABAADWIugAAABrEXQAAIC1CDoAAMBaBB0AAGAtgg4AALAWQQcAAFiLoAMAAKxF0AEAANYi6AAAAGsRdAAAgLV6fNCpq6vThAkTlJKSorFjx+o3v/mN0yUBAIAA0dvpAjorIiJCe/bsUXh4uJqamjR27FjNnTtXgwYNcro0AADgsB5/RqdXr14KDw+XJDU3N8sYI2OMw1UBAIBA4HjQ2bNnj2bNmqW4uDi5XC5t27atQx+3263ExET17dtX6enp2rt3r9f+uro6JScna9iwYXryyScVHR3dTdUDAIBA5njQaWpqUnJystxu9zX3b968Wbm5uVqzZo3279+v5ORkZWZm6uzZs54+/fv318GDB1VZWan/+q//Um1tbXeVDwAAApjjQScrK0s//elPNWfOnGvuz8/P15IlS7Ro0SIlJSVpw4YNCg8PV0FBQYe+MTExSk5O1ltvvXXdz9fc3KyGhgavFwAAsJPjQedGWlpaVFpaqoyMDE9bSEiIMjIyVFxcLEmqra3VxYsXJUn19fXas2ePxowZc90x8/LyFBUV5XnFx8d37SQAAIBjAjronD9/Xm1tbYqJifFqj4mJUU1NjSTpxIkTmjJlipKTkzVlyhR973vf07hx46475qpVq1RfX+95VVVVdekcAACAc3r85eVpaWkqKyu75f5hYWEKCwvruoIAAEDACOgzOtHR0erVq1eHxcW1tbWKjY11qCoAANBTBHTQCQ0NVWpqqoqKijxt7e3tKioq0qRJkzo1ttvtVlJSkiZOnNjZMgEAQIBy/K2rxsZGVVRUeLYrKytVVlamgQMHKiEhQbm5ucrOztaECROUlpam9evXq6mpSYsWLerU583JyVFOTo4aGhoUFRXV2WkAAIAA5HjQKSkp0dSpUz3bubm5kqTs7GwVFhZq3rx5OnfunFavXq2amhqlpKRox44dHRYoAwAAfJ7LBPnzEq6e0amvr1dkZKRfx05cud2v43WH42tnOl0CAAA3dat/vwN6jU5XYo0OAAD2C9qgk5OTo/Lycu3bt8/pUgAAQBcJ2qADAADsR9ABAADWIugAAABrBW3QYTEyAAD2C9qgw2JkAADsF7RBBwAA2I+gAwAArEXQAQAA1graoMNiZAAA7Be0QYfFyAAA2C9ogw4AALAfQQcAAFiLoAMAAKxF0AEAANYi6AAAAGsFbdDh8nIAAOwXtEGHy8sBALBf0AYdAABgP4IOAACwFkEHAABYi6ADAACsRdABAADWIugAAABrBW3Q4T46AADYL2iDDvfRAQDAfkEbdAAAgP0IOgAAwFoEHQAAYC2CDgAAsBZBBwAAWIugAwAArEXQAQAA1iLoAAAAawVt0OHOyAAA2C9ogw53RgYAwH5BG3QAAID9CDoAAMBaPgWdP//5z/6uAwAAwO98Cjr33HOPpk6dqpdeekmXL1/2d00AAAB+4VPQ2b9/v8aPH6/c3FzFxsbq29/+tvbu3evv2gAAADrFp6CTkpKiX/7ylzpz5owKCgpUXV2tyZMna+zYscrPz9e5c+f8XScAAMBt69Ri5N69e2vu3LnasmWLfv7zn6uiokLLly9XfHy8FixYoOrqan/VCQAAcNs6FXRKSkr03e9+V0OGDFF+fr6WL1+uY8eO6Y033tCZM2c0e/Zsf9UJAABw23r78kH5+fnauHGjjhw5ohkzZujFF1/UjBkzFBLyaW4aMWKECgsLlZiY6M9aAQAAbotPQefZZ5/VY489poULF2rIkCHX7DN48GA9//zznSoOAACgM3wKOkePHr1pn9DQUGVnZ/syPAAAgF/4tEZn48aN2rJlS4f2LVu26IUXXuh0UQAAAP7gU9DJy8tTdHR0h/bBgwfrZz/7WaeLAgAA8Aefgs7Jkyc1YsSIDu3Dhw/XyZMnO11Ud3C73UpKStLEiROdLgUAAHQRn4LO4MGDdejQoQ7tBw8e1KBBgzpdVHfIyclReXm59u3b53QpAACgi/gUdB555BF9//vf1+7du9XW1qa2tjbt2rVLy5Yt09e+9jV/1wgAAOATn666+slPfqLjx49r2rRp6t370yHa29u1YMEC1ugAAICA4VPQCQ0N1ebNm/WTn/xEBw8eVL9+/TRu3DgNHz7c3/UBAAD4zKegc9Xo0aM1evRof9UCAADgVz4Fnba2NhUWFqqoqEhnz55Ve3u71/5du3b5pTgAAIDO8CnoLFu2TIWFhZo5c6bGjh0rl8vl77oAAAA6zaegs2nTJv3ud7/TjBkz/F0PAACA3/h0eXloaKjuuecef9cCAADgVz4FnSeeeEK//OUvZYzxdz0AAAB+49NbV2+//bZ2796t119/Xffee6/69OnjtX/r1q1+KQ4AAKAzfAo6/fv315w5c/xdCwAAgF/5FHQ2btzo7zoAAAD8zqc1OpJ05coV7dy5U88995wuXrwoSTpz5owaGxv9VhwAAEBn+HRG58SJE5o+fbpOnjyp5uZm/c3f/I0iIiL085//XM3NzdqwYYO/6wQAALhtPp3RWbZsmSZMmKBPPvlE/fr187TPmTNHRUVFfisOAACgM3w6o/PWW2/p3XffVWhoqFd7YmKiTp8+7ZfCAAAAOsunMzrt7e1qa2vr0H7q1ClFRER0uigAAAB/8CnoPPTQQ1q/fr1n2+VyqbGxUWvWrOn2x0JUVVXpgQceUFJSksaPH68tW7Z06+cHAACBy6e3rp5++mllZmYqKSlJly9f1te//nUdPXpU0dHRevnll/1d4w317t1b69evV0pKimpqapSamqoZM2bojjvu6NY6AABA4PEp6AwbNkwHDx7Upk2bdOjQITU2Nmrx4sWaP3++1+Lk7jBkyBANGTJEkhQbG6vo6GhduHCBoAMAAHy/j07v3r316KOP6qmnntIzzzyjb37zmz6FnD179mjWrFmKi4uTy+XStm3bOvRxu91KTExU3759lZ6err17915zrNLSUrW1tSk+Pv626wAAAPbx6YzOiy++eMP9CxYsuOWxmpqalJycrMcee0xz587tsH/z5s3Kzc3Vhg0blJ6ervXr1yszM1NHjhzR4MGDPf0uXLigBQsW6De/+c2tTwQAAFjNZXx4BPmAAQO8tltbW3Xp0iWFhoYqPDxcFy5c8K0Yl0uvvPKKvvrVr3ra0tPTNXHiRP3617+W9OkVX/Hx8fre976nlStXSpLnpoVLlizRN77xjRt+jubmZjU3N3u2GxoaFB8fr/r6ekVGRvpU9/Ukrtzu1/G6w/G1M50uAQCAm2poaFBUVNRN/3779NbVJ5984vVqbGzUkSNHNHnyZL8uRm5paVFpaakyMjL+f8EhIcrIyFBxcbEkyRijhQsX6sEHH7xpyJGkvLw8RUVFeV68zQUAgL18XqPzeaNGjdLatWu1bNkyfw2p8+fPq62tTTExMV7tMTExqqmpkSS988472rx5s7Zt26aUlBSlpKToT3/603XHXLVqlerr6z2vqqoqv9ULAAACi09rdK47WO/eOnPmjD+HvKnJkyervb39lvuHhYUpLCysCysCAACBwqeg8z//8z9e28YYVVdX69e//rW+/OUv+6UwSYqOjlavXr1UW1vr1V5bW6vY2NhOje12u+V2u695h2cAAGAHn4LOZxcLS58uIr7rrrv04IMP6umnn/ZHXZKk0NBQpaamqqioyPM529vbVVRUpKVLl3Zq7JycHOXk5HgWMwEAAPv4FHRu562im2lsbFRFRYVnu7KyUmVlZRo4cKASEhKUm5ur7OxsTZgwQWlpaVq/fr2ampq0aNEiv9UAAADs5Nc1Or4oKSnR1KlTPdu5ubmSpOzsbBUWFmrevHk6d+6cVq9erZqaGqWkpGjHjh0dFigDAAB8nk/30bkaRm5Ffn7+7Q7frW71OnxfcB8dAAC6xq3+/fbpjM6BAwd04MABtba2asyYMZKkjz76SL169dJ9993n6edyuXwZvluwGBkAAPv5FHRmzZqliIgIvfDCC567JH/yySdatGiRpkyZoieeeMKvRXYFFiMDAGA/n24Y+PTTTysvL8/rURADBgzQT3/6U79edQUAANAZPgWdhoYGnTt3rkP7uXPndPHixU4XBQAA4A8+BZ05c+Zo0aJF2rp1q06dOqVTp07pv//7v7V48eJrPoE8ELndbiUlJWnixIlOlwIAALqIT1ddXbp0ScuXL1dBQYFaW1slffr4h8WLF2vdunW64447/F5oV+GqK29cdQUA6Am69Kqr8PBwPfPMM1q3bp2OHTsmSRo5cmSPCjgAAMB+nXp6eXV1taqrqzVq1Cjdcccd8uHkEAAAQJfxKeh8/PHHmjZtmkaPHq0ZM2aourpakrR48eIecWk5AAAIDj4FnX/6p39Snz59dPLkSYWHh3va582bpx07dvitOAAAgM7waY3OH//4R/3hD3/QsGHDvNpHjRqlEydO+KWwrsadkQEAsJ9PZ3Sampq8zuRcdeHCBYWFhXW6qO6Qk5Oj8vJy7du3z+lSAABAF/Ep6EyZMkUvvviiZ9vlcqm9vV1PPfWU15PIAQAAnOTTW1dPPfWUpk2bppKSErW0tOgHP/iBPvjgA124cEHvvPOOv2sEAADwiU9ndMaOHauPPvpIkydP1uzZs9XU1KS5c+fqwIEDGjlypL9rBAAA8Mltn9FpbW3V9OnTtWHDBv3whz/sipoAAAD84rbP6PTp00eHDh3qilq6Fc+6AgDAfj69dfXoo4/q+eef93ct3YqrrgAAsJ9Pi5GvXLmigoIC7dy5U6mpqR2ecZWfn++X4gAAADrjtoLOn//8ZyUmJurw4cO67777JEkfffSRVx+Xy+W/6gAAADrhtoLOqFGjVF1drd27d0v69JEP//Ef/6GYmJguKQ4AAKAzbmuNzuefTv7666+rqanJrwUBAAD4i0+Lka/6fPABAAAIJLcVdFwuV4c1OKzJAQAAgeq21ugYY7Rw4ULPgzsvX76sxx9/vMNVV1u3bvVfhV2Ep5cDAGC/2wo62dnZXtuPPvqoX4vpTjk5OcrJyVFDQ4OioqKcLgcAAHSB2wo6Gzdu7Ko6AAAA/K5Ti5EBAAACGUEHAABYi6ADAACs5dOzrmCvxJXbnS7hth1fO9PpEgAAAYozOgAAwFoEHQAAYC2CDgAAsBZBBwAAWCtog47b7VZSUpImTpzodCkAAKCLBG3QycnJUXl5ufbt2+d0KQAAoIsEbdABAAD2I+gAAABrEXQAAIC1CDoAAMBaBB0AAGAtgg4AALAWQQcAAFiLoAMAAKxF0AEAANYi6AAAAGsRdAAAgLUIOgAAwFpBG3R4ejkAAPYL2qDD08sBALBf0AYdAABgP4IOAACwFkEHAABYi6ADAACsRdABAADWIugAAABrEXQAAIC1CDoAAMBaBB0AAGAtgg4AALAWQQcAAFiLoAMAAKxF0AEAANYi6AAAAGsRdAAAgLUIOgAAwFoEHQAAYC0rgs6cOXM0YMAA/d3f/Z3TpQAAgABiRdBZtmyZXnzxRafLAAAAAcaKoPPAAw8oIiLC6TIAAECAcTzo7NmzR7NmzVJcXJxcLpe2bdvWoY/b7VZiYqL69u2r9PR07d27t/sLBQAAPY7jQaepqUnJyclyu93X3L9582bl5uZqzZo12r9/v5KTk5WZmamzZ892c6UAAKCn6e10AVlZWcrKyrru/vz8fC1ZskSLFi2SJG3YsEHbt29XQUGBVq5cedufr7m5Wc3NzZ7thoaG2y8aAAD0CI6f0bmRlpYWlZaWKiMjw9MWEhKijIwMFRcX+zRmXl6eoqKiPK/4+Hh/lQsAAAJMQAed8+fPq62tTTExMV7tMTExqqmp8WxnZGTo7//+7/Xaa69p2LBhNwxBq1atUn19vedVVVXVZfUDAABnOf7WlT/s3LnzlvuGhYUpLCysC6sBAACBIqDP6ERHR6tXr16qra31aq+trVVsbKxDVQEAgJ4ioINOaGioUlNTVVRU5Glrb29XUVGRJk2a1Kmx3W63kpKSNHHixM6WCQAAApTjb101NjaqoqLCs11ZWamysjINHDhQCQkJys3NVXZ2tiZMmKC0tDStX79eTU1NnquwfJWTk6OcnBw1NDQoKiqqs9MAAAAByPGgU1JSoqlTp3q2c3NzJUnZ2dkqLCzUvHnzdO7cOa1evVo1NTVKSUnRjh07OixQBgAA+DyXMcY4XYSTrp7Rqa+vV2RkpF/HTly53a/j4dqOr53pdAkAgG52q3+/A3qNTldijQ4AAPYL2qCTk5Oj8vJy7du3z+lSAABAFwnaoAMAAOxH0AEAANYi6AAAAGs5fnm5U9xut9xut9ra2pwuBZ3UE69u40oxAOgeQXtGh8XIAADYL2iDDgAAsB9BBwAAWIugAwAArBW0QYc7IwMAYL+gDTosRgYAwH5BG3QAAID9CDoAAMBaBB0AAGAtgg4AALAWQQcAAFgraIMOl5cDAGC/oA06XF4OAID9gjboAAAA+xF0AACAtQg6AADAWgQdAABgLYIOAACwFkEHAABYK2iDDvfRAQDAfkEbdLiPDgAA9gvaoAMAAOxH0AEAANYi6AAAAGsRdAAAgLUIOgAAwFoEHQAAYC2CDgAAsBZBBwAAWKu30wU4xe12y+12q62tzelSAKDHS1y53ekSbtvxtTOdLgHdIGjP6HBnZAAA7Be0QQcAANiPoAMAAKxF0AEAANYi6AAAAGsRdAAAgLUIOgAAwFoEHQAAYC2CDgAAsBZBBwAAWIugAwAArEXQAQAA1iLoAAAAaxF0AACAtXo7XYBT3G633G632tranC4FQShx5XanS/DJ8bUznS4B8Jue+nPY0zj9eyNoz+jk5OSovLxc+/btc7oUAADQRYI26AAAAPsRdAAAgLUIOgAAwFoEHQAAYC2CDgAAsBZBBwAAWIugAwAArEXQAQAA1iLoAAAAaxF0AACAtQg6AADAWgQdAABgLYIOAACwFkEHAABYi6ADAACsRdABAADWIugAAABrWRF0fv/732vMmDEaNWqUfvvb3zpdDgAACBC9nS6gs65cuaLc3Fzt3r1bUVFRSk1N1Zw5czRo0CCnSwMAAA7r8Wd09u7dq3vvvVdDhw7VnXfeqaysLP3xj390uiwAABAAHA86e/bs0axZsxQXFyeXy6Vt27Z16ON2u5WYmKi+ffsqPT1de/fu9ew7c+aMhg4d6tkeOnSoTp8+3R2lAwCAAOd40GlqalJycrLcbvc192/evFm5ublas2aN9u/fr+TkZGVmZurs2bM+fb7m5mY1NDR4vQAAgJ0cX6OTlZWlrKys6+7Pz8/XkiVLtGjRIknShg0btH37dhUUFGjlypWKi4vzOoNz+vRppaWlXXe8vLw8/ehHP/LfBIAgkrhyu9MlBIXja2c6XQJgDcfP6NxIS0uLSktLlZGR4WkLCQlRRkaGiouLJUlpaWk6fPiwTp8+rcbGRr3++uvKzMy87pirVq1SfX2951VVVdXl8wAAAM5w/IzOjZw/f15tbW2KiYnxao+JidGHH34oSerdu7eefvppTZ06Ve3t7frBD35wwyuuwsLCFBYW1qV1AwCAwBDQQedWPfzww3r44YedLgMAAASYgH7rKjo6Wr169VJtba1Xe21trWJjYzs1ttvtVlJSkiZOnNipcQAAQOAK6KATGhqq1NRUFRUVedra29tVVFSkSZMmdWrsnJwclZeXa9++fZ0tEwAABCjH37pqbGxURUWFZ7uyslJlZWUaOHCgEhISlJubq+zsbE2YMEFpaWlav369mpqaPFdhAQAAXI/jQaekpERTp071bOfm5kqSsrOzVVhYqHnz5uncuXNavXq1ampqlJKSoh07dnRYoAwAAPB5jgedBx54QMaYG/ZZunSpli5d6tfP63a75Xa71dbW5tdxAQBA4AjoNTpdiTU6AADYL2iDDgAAsB9BBwAAWIugAwAArBW0QYcbBgIAYL+gDTosRgYAwH5BG3QAAID9HL+PjtOu3sOnoaHB72O3N1/y+5gA7NcVv4+6Gr/vcD1d9f18ddyb3YvPZW7Ww3KnTp1SfHy802UAAAAfVFVVadiwYdfdH/RBp729XWfOnFFERIRcLpffxm1oaFB8fLyqqqoUGRnpt3EDHfNm3sGAeTPvYBDo8zbG6OLFi4qLi1NIyPVX4gT9W1chISE3TIKdFRkZGZDfIF2NeQcX5h1cmHdwCeR5R0VF3bQPi5EBAIC1CDoAAMBaBJ0uEhYWpjVr1igsLMzpUroV82bewYB5M+9gYMu8g34xMgAAsBdndAAAgLUIOgAAwFoEHQAAYC2CDgAAsBZBpwu43W4lJiaqb9++Sk9P1969e50uqVPy8vI0ceJERUREaPDgwfrqV7+qI0eOePV54IEH5HK5vF6PP/64V5+TJ09q5syZCg8P1+DBg/Xkk0/qypUr3TmV2/Kv//qvHeb0hS98wbP/8uXLysnJ0aBBg3TnnXfqb//2b1VbW+s1Rk+bsyQlJiZ2mLfL5VJOTo4ke471nj17NGvWLMXFxcnlcmnbtm1e+40xWr16tYYMGaJ+/fopIyNDR48e9epz4cIFzZ8/X5GRkerfv78WL16sxsZGrz6HDh3SlClT1LdvX8XHx+upp57q6qnd0I3m3draqhUrVmjcuHG64447FBcXpwULFujMmTNeY1zre2Tt2rVefXrSvCVp4cKFHeY0ffp0rz62HW9J1/xZd7lcWrdunadPTzzeXgz8atOmTSY0NNQUFBSYDz74wCxZssT079/f1NbWOl2azzIzM83GjRvN4cOHTVlZmZkxY4ZJSEgwjY2Nnj5f+cpXzJIlS0x1dbXnVV9f79l/5coVM3bsWJORkWEOHDhgXnvtNRMdHW1WrVrlxJRuyZo1a8y9997rNadz58559j/++OMmPj7eFBUVmZKSEvOlL33J/PVf/7Vnf0+cszHGnD171mvOb7zxhpFkdu/ebYyx51i/9tpr5oc//KHZunWrkWReeeUVr/1r1641UVFRZtu2bebgwYPm4YcfNiNGjDB/+ctfPH2mT59ukpOTzXvvvWfeeustc88995hHHnnEs7++vt7ExMSY+fPnm8OHD5uXX37Z9OvXzzz33HPdNc0ObjTvuro6k5GRYTZv3mw+/PBDU1xcbNLS0kxqaqrXGMOHDzc//vGPvb4HPvv7oKfN2xhjsrOzzfTp073mdOHCBa8+th1vY4zXfKurq01BQYFxuVzm2LFjnj498Xh/FkHHz9LS0kxOTo5nu62tzcTFxZm8vDwHq/Kvs2fPGknmf//3fz1tX/nKV8yyZcuu+zGvvfaaCQkJMTU1NZ62Z5991kRGRprm5uauLNdna9asMcnJydfcV1dXZ/r06WO2bNniafu///s/I8kUFxcbY3rmnK9l2bJlZuTIkaa9vd0YY+ex/vwfgPb2dhMbG2vWrVvnaaurqzNhYWHm5ZdfNsYYU15ebiSZffv2efq8/vrrxuVymdOnTxtjjHnmmWfMgAEDvOa9YsUKM2bMmC6e0a251h++z9u7d6+RZE6cOOFpGz58uPnFL35x3Y/pifPOzs42s2fPvu7HBMvxnj17tnnwwQe92nr68eatKz9qaWlRaWmpMjIyPG0hISHKyMhQcXGxg5X5V319vSRp4MCBXu3/+Z//qejoaI0dO1arVq3SpUuXPPuKi4s1btw4xcTEeNoyMzPV0NCgDz74oHsK98HRo0cVFxenu+++W/Pnz9fJkyclSaWlpWptbfU61l/4wheUkJDgOdY9dc6f1dLSopdeekmPPfaY10NvbTzWn1VZWamamhqv4xsVFaX09HSv49u/f39NmDDB0ycjI0MhISF6//33PX3uv/9+hYaGevpkZmbqyJEj+uSTT7ppNp1TX18vl8ul/v37e7WvXbtWgwYN0he/+EWtW7fO663JnjrvN998U4MHD9aYMWP0ne98Rx9//LFnXzAc79raWm3fvl2LFy/usK8nH++gf6inP50/f15tbW1ev+AlKSYmRh9++KFDVflXe3u7/vEf/1Ff/vKXNXbsWE/717/+dQ0fPlxxcXE6dOiQVqxYoSNHjmjr1q2SpJqammt+Xa7uC0Tp6ekqLCzUmDFjVF1drR/96EeaMmWKDh8+rJqaGoWGhnb45R8TE+OZT0+c8+dt27ZNdXV1WrhwoafNxmP9eVfrvNY8Pnt8Bw8e7LW/d+/eGjhwoFefESNGdBjj6r4BAwZ0Sf3+cvnyZa1YsUKPPPKI10Mdv//97+u+++7TwIED9e6772rVqlWqrq5Wfn6+pJ457+nTp2vu3LkaMWKEjh07pn/+539WVlaWiouL1atXr6A43i+88IIiIiI0d+5cr/aefrwJOrgtOTk5Onz4sN5++22v9m9961uef48bN05DhgzRtGnTdOzYMY0cObK7y/SLrKwsz7/Hjx+v9PR0DR8+XL/73e/Ur18/ByvrPs8//7yysrIUFxfnabPxWKOj1tZW/cM//IOMMXr22We99uXm5nr+PX78eIWGhurb3/628vLyeuzjAr72ta95/j1u3DiNHz9eI0eO1Jtvvqlp06Y5WFn3KSgo0Pz589W3b1+v9p5+vHnryo+io6PVq1evDlfe1NbWKjY21qGq/Gfp0qX6/e9/r927d2vYsGE37Jueni5JqqiokCTFxsZe8+tydV9P0L9/f40ePVoVFRWKjY1VS0uL6urqvPp89lj39DmfOHFCO3fu1De/+c0b9rPxWF+t80Y/y7GxsTp79qzX/itXrujChQs9/nvgasg5ceKE3njjDa+zOdeSnp6uK1eu6Pjx45J67rw/6+6771Z0dLTX97Wtx1uS3nrrLR05cuSmP+9SzzveBB0/Cg0NVWpqqoqKijxt7e3tKioq0qRJkxysrHOMMVq6dKleeeUV7dq1q8MpymspKyuTJA0ZMkSSNGnSJP3pT3/y+kVx9RdoUlJSl9Ttb42NjTp27JiGDBmi1NRU9enTx+tYHzlyRCdPnvQc654+540bN2rw4MGaOXPmDfvZeKxHjBih2NhYr+Pb0NCg999/3+v41tXVqbS01NNn165dam9v94S/SZMmac+ePWptbfX0eeONNzRmzBjHT+dfz9WQc/ToUe3cuVODBg266ceUlZUpJCTE89ZOT5z35506dUoff/yx1/e1jcf7queff16pqalKTk6+ad8ed7ydXg1tm02bNpmwsDBTWFhoysvLzbe+9S3Tv39/rytQeprvfOc7Jioqyrz55ptelxdeunTJGGNMRUWF+fGPf2xKSkpMZWWlefXVV83dd99t7r//fs8YVy85fuihh0xZWZnZsWOHueuuuwLukuPPeuKJJ8ybb75pKisrzTvvvGMyMjJMdHS0OXv2rDHm08vLExISzK5du0xJSYmZNGmSmTRpkufje+Kcr2prazMJCQlmxYoVXu02HeuLFy+aAwcOmAMHDhhJJj8/3xw4cMBzddHatWtN//79zauvvmoOHTpkZs+efc3Ly7/4xS+a999/37z99ttm1KhRXpcb19XVmZiYGPONb3zDHD582GzatMmEh4c7etntjebd0tJiHn74YTNs2DBTVlbm9fN+9Yqad9991/ziF78wZWVl5tixY+all14yd911l1mwYIHnc/S0eV+8eNEsX77cFBcXm8rKSrNz505z3333mVGjRpnLly97xrDteF9VX19vwsPDzbPPPtvh43vq8f4sgk4X+NWvfmUSEhJMaGioSUtLM++9957TJXWKpGu+Nm7caIwx5uTJk+b+++83AwcONGFhYeaee+4xTz75pNe9VYwx5vjx4yYrK8v069fPREdHmyeeeMK0trY6MKNbM2/ePDNkyBATGhpqhg4daubNm2cqKio8+//yl7+Y7373u2bAgAEmPDzczJkzx1RXV3uN0dPmfNUf/vAHI8kcOXLEq92mY7179+5rfl9nZ2cbYz69xPxf/uVfTExMjAkLCzPTpk3r8PX4+OOPzSOPPGLuvPNOExkZaRYtWmQuXrzo1efgwYNm8uTJJiwszAwdOtSsXbu2u6Z4TTead2Vl5XV/3q/eR6m0tNSkp6ebqKgo07dvX/NXf/VX5mc/+5lXIDCmZ8370qVL5qGHHjJ33XWX6dOnjxk+fLhZsmRJh/+g2na8r3ruuedMv379TF1dXYeP76nH+7NcxhjTpaeMAAAAHMIaHQAAYC2CDgAAsBZBBwAAWIugAwAArEXQAQAA1iLoAAAAaxF0AACAtQg6AADAWgQdAABgLYIOAACwFkEHAABYi6ADAACs9f8ASsZC+gu3kqEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# anime['episodes'].sort_values(ascending=False).head(25)\n",
    "anime[anime['episodes'] > 0]['episodes'].plot(kind='hist', logy=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of unique genres: 44\n",
      "\n",
      "Unique genres:\n",
      "['Action', 'Adventure', 'Cars', 'Comedy', 'Dementia', 'Demons', 'Drama', 'Ecchi', 'Fantasy', 'Game', 'Harem', 'Hentai', 'Historical', 'Horror', 'Josei', 'Kids', 'Magic', 'Martial Arts', 'Mecha', 'Military', 'Music', 'Mystery', 'Parody', 'Police', 'Psychological', 'Romance', 'Samurai', 'School', 'Sci-Fi', 'Seinen', 'Shoujo', 'Shoujo Ai', 'Shounen', 'Shounen Ai', 'Slice of Life', 'Space', 'Sports', 'Super Power', 'Supernatural', 'Thriller', 'Unknown', 'Vampire', 'Yaoi', 'Yuri']\n"
     ]
    }
   ],
   "source": [
    "# Extract all unique genres from anime\n",
    "all_genres = anime['genre'].dropna().str.split(',')\n",
    "unique_genres = set([genre.strip() for genres in all_genres for genre in genres])\n",
    "\n",
    "print(\"Number of unique genres:\", len(unique_genres))\n",
    "print(\"\\nUnique genres:\")\n",
    "print(sorted(unique_genres))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Genre DataFrame shape: (12294, 44)\n",
      "\n",
      "First few rows of the Genre DataFrame:\n",
      "   Action  Adventure  Cars  Comedy  Dementia  Demons  Drama  Ecchi  Fantasy  \\\n",
      "0       0          0     0       0         0       0      1      0        0   \n",
      "1       1          1     0       0         0       0      1      0        1   \n",
      "2       1          0     0       1         0       0      0      0        0   \n",
      "3       0          0     0       0         0       0      0      0        0   \n",
      "4       1          0     0       1         0       0      0      0        0   \n",
      "\n",
      "   Game  ...  Slice of Life  Space  Sports  Super Power  Supernatural  \\\n",
      "0     0  ...              0      0       0            0             1   \n",
      "1     0  ...              0      0       0            0             0   \n",
      "2     0  ...              0      0       0            0             0   \n",
      "3     0  ...              0      0       0            0             0   \n",
      "4     0  ...              0      0       0            0             0   \n",
      "\n",
      "   Thriller  Unknown  Vampire  Yaoi  Yuri  \n",
      "0         0        0        0     0     0  \n",
      "1         0        0        0     0     0  \n",
      "2         0        0        0     0     0  \n",
      "3         1        0        0     0     0  \n",
      "4         0        0        0     0     0  \n",
      "\n",
      "[5 rows x 44 columns]\n",
      "\n",
      "Sum of genres for each anime:\n",
      "1     2866\n",
      "2     2670\n",
      "3     2579\n",
      "4     1929\n",
      "5     1275\n",
      "6      611\n",
      "7      230\n",
      "8       77\n",
      "9       41\n",
      "10      13\n",
      "11       2\n",
      "13       1\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Create a genre dataframe\n",
    "genres = pd.DataFrame(index=anime.index)\n",
    "\n",
    "# Iterate through unique genres and create binary columns\n",
    "for genre in sorted(unique_genres):\n",
    "    genres[genre] = anime['genre'].str.contains(genre).astype(int)\n",
    "\n",
    "# Display the first few rows and shape of the genre dataframe\n",
    "print(\"Genre DataFrame shape:\", genres.shape)\n",
    "print(\"\\nFirst few rows of the Genre DataFrame:\")\n",
    "print(genres.head())\n",
    "\n",
    "# Verify the sum of genres for each anime\n",
    "print(\"\\nSum of genres for each anime:\")\n",
    "print(genres.sum(axis=1).value_counts().sort_index())\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Duplicate rows in anime DataFrame:\n",
      "0\n",
      "\n",
      "Duplicate anime_id in anime DataFrame:\n",
      "0\n",
      "\n",
      "Duplicate rows in ratings DataFrame:\n",
      "1\n",
      "\n",
      "Duplicate (user_id, anime_id) pairs in ratings DataFrame:\n",
      "7\n"
     ]
    }
   ],
   "source": [
    "# Check for duplicates in anime DataFrame\n",
    "print(\"Duplicate rows in anime DataFrame:\")\n",
    "print(anime.duplicated().sum())\n",
    "print(\"\\nDuplicate anime_id in anime DataFrame:\")\n",
    "print(anime['anime_id'].duplicated().sum())\n",
    "\n",
    "# Check for duplicates in ratings DataFrame\n",
    "print(\"\\nDuplicate rows in ratings DataFrame:\")\n",
    "print(ratings.duplicated().sum())\n",
    "print(\"\\nDuplicate (user_id, anime_id) pairs in ratings DataFrame:\")\n",
    "print(ratings.duplicated(subset=['user_id', 'anime_id']).sum())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings = ratings.drop_duplicates(subset=['user_id', 'anime_id'], keep='last')\n",
    "ratings['total_watch'] = ratings['user_id'].map(ratings['user_id'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original rated DataFrame shape: (6337234, 4)\n",
      "Subset of rated DataFrame shape: (6151960, 4)\n",
      "\n",
      "Number of unique users in original: 69600\n",
      "Number of unique users in subset: 46502\n"
     ]
    }
   ],
   "source": [
    "# Separate ratings into rated and unrated\n",
    "rated = ratings[ratings['rating'] >= 0]\n",
    "# unrated = ratings[ratings['rating'] < 0]\n",
    "\n",
    "# Count the occurrences of each user_id\n",
    "user_counts = rated['user_id'].value_counts()\n",
    "top_user_ratings = rated.groupby('user_id')['rating'].quantile(0.75)\n",
    "bottom_user_ratings = rated.groupby('user_id')['rating'].quantile(0.25)\n",
    "\n",
    "# Create a mask for users with more than 20 ratings\n",
    "frequent_users = user_counts[user_counts > 20].index\n",
    "# Create a mask for rows with rating > user_ratings\n",
    "high_rating_mask = rated['rating'] > rated['user_id'].map(user_ratings)\n",
    "\n",
    "# Display information about the high rating mask\n",
    "print(\"\\nNumber of high ratings:\", high_rating_mask.sum())\n",
    "print(\"Percentage of high ratings: {:.2f}%\".format(high_rating_mask.mean() * 100))\n",
    "\n",
    "\n",
    "# Create a subset of rated df with users who have more than 10 ratings\n",
    "rated_subset = rated[rated['user_id'].isin(frequent_users)]\n",
    "\n",
    "# Display information about the new subset\n",
    "print(\"Original rated DataFrame shape:\", rated.shape)\n",
    "print(\"Subset of rated DataFrame shape:\", rated_subset.shape)\n",
    "print(\"\\nNumber of unique users in original:\", rated['user_id'].nunique())\n",
    "print(\"Number of unique users in subset:\", rated_subset['user_id'].nunique())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      user_id  anime_id  rating  total_watch\n",
      "1668       17      1889      10          601\n",
      "2758       27     11319      10           80\n",
      "761         5     29803       8          467\n",
      "2105       17     25519       8          601\n",
      "1576       14     16782       8          123\n",
      "1957       17     19365       5          601\n",
      "921         7     10110       6          343\n",
      "1796       17     10396       5          601\n",
      "1064        7     24227       6          343\n",
      "725         5     23273       8          467\n",
      "1120        7     31043       9          343\n",
      "2904       29     11209       3          129\n",
      "1571       14     16067       7          123\n",
      "1821       17     11577       9          601\n",
      "      user_id  anime_id  rating  total_watch\n",
      "2919       29     16498       8          129\n",
      "497         5      7817       2          467\n",
      "2584       24     10067       8           66\n",
      "1220       11       790       8          112\n",
      "1668       17      1889      10          601\n",
      "2758       27     11319      10           80\n",
      "761         5     29803       8          467\n",
      "2756       27     10790       9           80\n",
      "550         5     11113       7          467\n",
      "2105       17     25519       8          601\n",
      "1709       17      4280       7          601\n",
      "2815       29       232       7          129\n",
      "2632       25     11633       9           38\n",
      "518         5      9367       1          467\n",
      "1576       14     16782       8          123\n",
      "2461       22       853      10           14\n",
      "2709       26     19429       8           65\n",
      "1957       17     19365       5          601\n",
      "489         5      7593       1          467\n",
      "1678       17      2605       7          601\n",
      "621         5     15775       7          467\n",
      "921         7     10110       6          343\n",
      "1796       17     10396       5          601\n",
      "1744       17      6707       7          601\n",
      "2232       18     10620      10           25\n",
      "515         5      9220       5          467\n",
      "359         5       448       3          467\n",
      "379         5       903       1          467\n",
      "2410       21     18497       7          180\n",
      "1151        8      6702      10           12\n",
      "1064        7     24227       6          343\n",
      "725         5     23273       8          467\n",
      "941         7     11977       8          343\n",
      "1120        7     31043       9          343\n",
      "2818       29       288       8          129\n",
      "1765       17      8769       7          601\n",
      "2540       23     10330       9           72\n",
      "222         3     20507       8           94\n",
      "2904       29     11209       3          129\n",
      "999         7     18247       8          343\n",
      "1884       17     15197       7          601\n",
      "546         5     10805       5          467\n",
      "1571       14     16067       7          123\n",
      "2064       17     23433       6          601\n",
      "1620       17        66       6          601\n",
      "850         7      1563       8          343\n",
      "1603       16        61       8           11\n",
      "2785       27     16049       9           80\n",
      "1821       17     11577       9          601\n",
      "487         5      7088       1          467\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "teste = pd.DataFrame(rated[rated['user_id'] < 30].sample(50))\n",
    "\n",
    "# user_counts = teste['user_id'].value_counts()\n",
    "# print(user_counts)\n",
    "# teste[teste['user_id'] == 5]\n",
    "# teste.groupby('user_id')['rating'].quantile(0.75)\n",
    "user_ratings = teste.groupby('user_id')['rating'].quantile(0.25)\n",
    "user_ratings2 = teste.groupby('user_id')['rating'].quantile(0.75)\n",
    "mask1 = teste['rating'] < teste['user_id'].map(user_ratings)\n",
    "mask2 = teste['rating'] > teste['user_id'].map(user_ratings2)\n",
    "print(teste[mask1 | mask2])\n",
    "print(teste)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get 20% of the rated_subset as test set\n",
    "_, rating_test = train_test_split(rated_subset, test_size=0.2, random_state=42)\n",
    "\n",
    "# Create the train set by removing the test set from the whole ratings DataFrame\n",
    "rating_train = ratings[~ratings.index.isin(rating_test.index)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All test users are present in the training set: True\n"
     ]
    }
   ],
   "source": [
    "# Test if all rating_test user_ids are also shown in rating_train\n",
    "test_users = set(rating_test['user_id'])\n",
    "train_users = set(rating_train['user_id'])\n",
    "\n",
    "all_test_users_in_train = test_users.issubset(train_users)\n",
    "\n",
    "print(f\"All test users are present in the training set: {all_test_users_in_train}\")\n",
    "\n",
    "if not all_test_users_in_train:\n",
    "    missing_users = test_users - train_users\n",
    "    print(f\"Number of test users not in training set: {len(missing_users)}\")\n",
    "    print(f\"Sample of missing users: {list(missing_users)[:5]}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of users needing adjustment: 0\n",
      "Adjustment complete.\n",
      "New test set shape: (1253843, 4)\n",
      "New train set shape: (6559887, 4)\n",
      "Users still lacking quartile representation: 0\n"
     ]
    }
   ],
   "source": [
    "# Ensure that for every user_id in test_users, there are examples of bottom quartile and top quartile ratings\n",
    "\n",
    "# Function to get bottom and top quartile ratings for a user\n",
    "def get_quartile_ratings(user_id, ratings_df):\n",
    "    user_ratings = ratings_df[ratings_df['user_id'] == user_id]['rating']\n",
    "    bottom_quartile = user_ratings.quantile(0.25)\n",
    "    top_quartile = user_ratings.quantile(0.75)\n",
    "    return user_ratings[user_ratings <= bottom_quartile], user_ratings[user_ratings >= top_quartile]\n",
    "\n",
    "# Initialize lists to store users that need adjustment\n",
    "users_to_adjust = []\n",
    "\n",
    "# Check each user in test_users\n",
    "for user_id in test_users:\n",
    "    bottom_q, top_q = get_quartile_ratings(user_id, ratings)\n",
    "    \n",
    "    # Check if both quartiles are represented in rating_test\n",
    "    if not (any(rating_test[(rating_test['user_id'] == user_id) & (rating_test['rating'] <= bottom_q.max())]) and\n",
    "            any(rating_test[(rating_test['user_id'] == user_id) & (rating_test['rating'] >= top_q.min())])):\n",
    "        users_to_adjust.append(user_id)\n",
    "\n",
    "print(f\"Number of users needing adjustment: {len(users_to_adjust)}\")\n",
    "\n",
    "# Adjust the test set for users lacking quartile representation\n",
    "for user_id in users_to_adjust:\n",
    "    bottom_q, top_q = get_quartile_ratings(user_id, ratings)\n",
    "    \n",
    "    # Add a bottom quartile rating to test set if missing\n",
    "    if not any(rating_test[(rating_test['user_id'] == user_id) & (rating_test['rating'] <= bottom_q.max())]):\n",
    "        bottom_sample = ratings[(ratings['user_id'] == user_id) & (ratings['rating'] <= bottom_q.max())].sample(1)\n",
    "        rating_test = pd.concat([rating_test, bottom_sample])\n",
    "        rating_train = rating_train[~rating_train.index.isin(bottom_sample.index)]\n",
    "    \n",
    "    # Add a top quartile rating to test set if missing\n",
    "    if not any(rating_test[(rating_test['user_id'] == user_id) & (rating_test['rating'] >= top_q.min())]):\n",
    "        top_sample = ratings[(ratings['user_id'] == user_id) & (ratings['rating'] >= top_q.min())].sample(1)\n",
    "        rating_test = pd.concat([rating_test, top_sample])\n",
    "        rating_train = rating_train[~rating_train.index.isin(top_sample.index)]\n",
    "\n",
    "print(\"Adjustment complete.\")\n",
    "print(f\"New test set shape: {rating_test.shape}\")\n",
    "print(f\"New train set shape: {rating_train.shape}\")\n",
    "\n",
    "# Verify the adjustment\n",
    "users_lacking_quartiles = sum(1 for user_id in test_users if not (\n",
    "    any(rating_test[(rating_test['user_id'] == user_id) & (rating_test['rating'] <= get_quartile_ratings(user_id, ratings)[0].max())]) and\n",
    "    any(rating_test[(rating_test['user_id'] == user_id) & (rating_test['rating'] >= get_quartile_ratings(user_id, ratings)[1].min())])\n",
    "))\n",
    "\n",
    "print(f\"Users still lacking quartile representation: {users_lacking_quartiles}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NMF components (w1 and h1) have been loaded from 'nmf_components.pkl'\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "redo = False\n",
    "\n",
    "user_item_matrix = rated.pivot(index='user_id', columns='anime_id', values='rating')\n",
    "\n",
    "if redo:\n",
    "    from sklearn.decomposition import NMF\n",
    "    # There are 44 genres, 20 components seems a good place to start\n",
    "    dec = NMF(n_components=20, random_state=42)\n",
    "    w1 = dec.fit_transform(user_item_matrix.fillna(0)) # user-group matrix\n",
    "    h1 = dec.components_ # group-anime matrix\n",
    "\n",
    "    # Save w1 and h1 to a pickle file\n",
    "    with open('data/nmf_components.pkl', 'wb') as f:\n",
    "        pickle.dump({'w1': w1, 'h1': h1}, f)\n",
    "\n",
    "    print(\"NMF components (w1 and h1) have been saved to 'nmf_components.pkl'\")\n",
    "\n",
    "else:\n",
    "    # Load the NMF components from the pickle file\n",
    "    with open('data/nmf_components.pkl', 'rb') as f:\n",
    "        nmf_components = pickle.load(f)\n",
    "    w1 = nmf_components['w1']\n",
    "    h1 = nmf_components['h1']\n",
    "\n",
    "    print(\"NMF components (w1 and h1) have been loaded from 'nmf_components.pkl'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "69600\n",
      "- 69600 -\n",
      "73516\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'numpy.ndarray' object has no attribute 'index'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[69], line 12\u001b[0m\n\u001b[1;32m      9\u001b[0m groups[a] \u001b[38;5;241m==\u001b[39m groups[\u001b[38;5;241m69600\u001b[39m\u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m]\n\u001b[1;32m     10\u001b[0m user_group \u001b[38;5;241m=\u001b[39m groups[a]\n\u001b[0;32m---> 12\u001b[0m \u001b[43mgroups\u001b[49m\u001b[43m[\u001b[49m\u001b[43mgroups\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m==\u001b[39;49m\u001b[43m \u001b[49m\u001b[43muser_group\u001b[49m\u001b[43m]\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mindex\u001b[49m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'numpy.ndarray' object has no attribute 'index'"
     ]
    }
   ],
   "source": [
    "groups = w1.argmax(axis=1)\n",
    "print(len(groups))\n",
    "\n",
    "print('-',len(user_item_matrix), '-')\n",
    "print(max(user_item_matrix.index))\n",
    "teste = user_item_matrix.reset_index()\n",
    "\n",
    "a = teste[teste['user_id'] == 73516].index \n",
    "groups[a] == groups[69600-1]\n",
    "user_group = groups[a]\n",
    "\n",
    "groups[groups == user_group].index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9927 34475\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.00000000e+00, 0.00000000e+00, 4.97000206e-01, ...,\n",
       "        0.00000000e+00, 0.00000000e+00, 0.00000000e+00],\n",
       "       [1.77700107e+01, 1.09564403e+01, 1.32863280e+01, ...,\n",
       "        2.36247912e-03, 0.00000000e+00, 0.00000000e+00],\n",
       "       [0.00000000e+00, 0.00000000e+00, 9.43008610e-02, ...,\n",
       "        0.00000000e+00, 0.00000000e+00, 0.00000000e+00],\n",
       "       ...,\n",
       "       [0.00000000e+00, 1.50334867e-01, 0.00000000e+00, ...,\n",
       "        5.15256983e-04, 0.00000000e+00, 1.05931941e-03],\n",
       "       [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ...,\n",
       "        0.00000000e+00, 0.00000000e+00, 5.19137383e-04],\n",
       "       [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ...,\n",
       "        0.00000000e+00, 0.00000000e+00, 0.00000000e+00]])"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# np.where(groups == 19)[0]\n",
    "print(len(user_item_matrix.columns), max(user_item_matrix.columns))\n",
    "h1.shape\n",
    "h1[:,9926]\n",
    "\n",
    "animes_group = h1.argmax(axis=1)\n",
    "[(row, col) for row, col in enumerate(animes_group)]\n",
    "h1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrix:\n",
      "[[44 47 64]\n",
      " [67 67  9]\n",
      " [83 21 36]]\n",
      "Row names: [18, 19, 18]\n",
      "Column names: [30, 25, 21]\n",
      "Row name: 18, Column name: 21, Max value: 64\n",
      "Row name: 19, Column name: 30, Max value: 67\n",
      "Row name: 18, Column name: 30, Max value: 83\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "# Set a seed for reproducibility\n",
    "np.random.seed(0)\n",
    "\n",
    "# Create a 3x3 matrix with random integers between 0 and 100\n",
    "matrix = np.random.randint(0, 100, size=(3, 3))\n",
    "\n",
    "# Generate random integers greater than 10 for row and column names\n",
    "row_names = [random.randint(11, 20) for _ in range(3)]\n",
    "col_names = [random.randint(21, 30) for _ in range(3)]\n",
    "\n",
    "# Print the matrix and names\n",
    "print(\"Matrix:\")\n",
    "print(matrix)\n",
    "print(\"Row names:\", row_names)\n",
    "print(\"Column names:\", col_names)\n",
    "\n",
    "# Find the indices of the maximum values along each row\n",
    "max_indices = np.argmax(matrix, axis=1)\n",
    "\n",
    "# Retrieve the corresponding row and column names\n",
    "for i, row_index in enumerate(max_indices):\n",
    "    print(f\"Row name: {row_names[i]}, Column name: {col_names[row_index]}, Max value: {matrix[i, row_index]}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of users in each group:\n",
      "0     1175\n",
      "1     4185\n",
      "2     3862\n",
      "3     3372\n",
      "4     6620\n",
      "5     2096\n",
      "6     1508\n",
      "7     2137\n",
      "8     2293\n",
      "9     4247\n",
      "10    4413\n",
      "11    1402\n",
      "12    8727\n",
      "13    1017\n",
      "14    2871\n",
      "15    6453\n",
      "16    3641\n",
      "17    2090\n",
      "18    2541\n",
      "19    4950\n",
      "Name: count, dtype: int64\n",
      "\n",
      "Total number of users: 69600\n",
      "\n",
      "Percentage of users in each group:\n",
      "0      1.69\n",
      "1      6.01\n",
      "2      5.55\n",
      "3      4.84\n",
      "4      9.51\n",
      "5      3.01\n",
      "6      2.17\n",
      "7      3.07\n",
      "8      3.29\n",
      "9      6.10\n",
      "10     6.34\n",
      "11     2.01\n",
      "12    12.54\n",
      "13     1.46\n",
      "14     4.12\n",
      "15     9.27\n",
      "16     5.23\n",
      "17     3.00\n",
      "18     3.65\n",
      "19     7.11\n",
      "Name: count, dtype: float64\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABKUAAAJOCAYAAABm7rQwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYlUlEQVR4nO3de7zVU/4/8Pfudkr3ok6ROgqVW2SQchvRUISMu9xzqREZFCN3YtyFhqFyGdfJtZkUiS9TSRgkKYpI5dKF0O18fn94tH+dqTinTp/dOT2fj8d+POy11v583qu9t7PP66zP2pkkSZIAAAAAgBRVyHUBAAAAAGx8hFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAUIZcccUVkclkUjnXvvvuG/vuu2/2/pgxYyKTycRTTz2VyvlPPvnkaNasWSrnWls//PBDnH766ZGfnx+ZTCbOO++8XJcEAFBmCKUAIEeGDBkSmUwme6tatWo0btw4OnXqFHfccUd8//33pXKeWbNmxRVXXBHvvvtuqRyvNG3ItRXHddddF0OGDImzzz47HnrooTjxxBPXODaTyUSvXr1W2/fUU09FJpOJMWPGrKdK16/ddtstMplM3HPPPbkuZb1auHBhXHvttbHrrrtG7dq1Iy8vL5o2bRpHH310DB8+PNflAUCZUynXBQDAxu6qq66KgoKCWLp0acyePTvGjBkT5513Xtxyyy3x3HPPxY477pgd+5e//CX69u1bouPPmjUrrrzyymjWrFm0adOm2I8bOXJkic6zNn6ttvvuuy8KCwvXew3rYvTo0bHHHnvE5ZdfnutScmbq1KkxYcKEaNasWTzyyCNx9tln57qk9WLatGnRqVOn+Oyzz+Lwww+P7t27R40aNWLmzJnxr3/9K7p06RIPPvjgrwaTAEBRQikAyLGDDjoodt111+z9fv36xejRo6NLly5x6KGHxuTJk6NatWoREVGpUqWoVGn9/vj+8ccfY5NNNokqVaqs1/P8lsqVK+f0/MUxd+7caN26da7LWCsrnud19fDDD0eDBg3i5ptvjiOPPDJmzJhRapddLlq0KKpXr14qx1oXy5Yti8MPPzzmzJkTr776arRv375I/+WXXx4jR46M5cuX/+pxNpT5AMCGwuV7ALAB+v3vfx+XXXZZfPbZZ/Hwww9n21e3p9SoUaOiQ4cOUadOnahRo0Zsu+22cckll0TEL/tA/e53v4uIiFNOOSV7qeCQIUMi4pd9o7bffvuYOHFi7L333rHJJptkH/u/e0qtsHz58rjkkksiPz8/qlevHoceemjMnDmzyJhmzZrFySefvMpjVz7mb9W2uj2lFi1aFBdccEE0adIk8vLyYtttt42bbropkiQpMm7FpXLPPPNMbL/99pGXlxfbbbddjBgxYvX/4P9j7ty5cdppp0XDhg2jatWqsdNOO8XQoUOz/Sv215o+fXoMHz48W/uMGTOKdfzimDp1anTr1i3y8/OjatWqscUWW8QxxxwTCxYsKDLu4YcfjrZt20a1atWiXr16ccwxx6zyfPza8/zWW29Fp06dYtNNN41q1apFQUFBnHrqqcWu8x//+EcceeSR0aVLl6hdu3b84x//WO248ePHx8EHHxx169aN6tWrx4477hi33357tv/kk0+OGjVqxCeffBIHH3xw1KxZM44//viIKP7z/mvvhRXuvPPO2G677WKTTTaJunXrxq677rrGmld48skn44MPPojLLrtslUBqhQMPPDAOOuig7P0Vl+e++uqrcc4550SDBg1iiy22yPbffffdsd1220VeXl40btw4evbsGfPnzy9yzOK8jyL+/+vx8ccf/833ZnFfVwCQBiulAGADdeKJJ8Yll1wSI0eOjDPOOGO1YyZNmhRdunSJHXfcMa666qrIy8uLadOmxRtvvBEREa1atYqrrroq+vfvHz169Ii99torIiL23HPP7DG+/fbbOOigg+KYY46JE044IRo2bPirdV177bWRyWTi4osvjrlz58Ztt90WHTt2jHfffTe7oqs4ilPbypIkiUMPPTReeeWVOO2006JNmzbx4osvxoUXXhhffvll3HrrrUXGv/766zFs2LA455xzombNmnHHHXdEt27d4vPPP4/69euvsa6ffvop9t1335g2bVr06tUrCgoK4sknn4yTTz455s+fH717945WrVrFQw89FOeff35sscUWccEFF0RExGabbVbs+f+aJUuWRKdOnWLx4sXxpz/9KfLz8+PLL7+MF154IebPnx+1a9eOiF+ei8suuyyOOuqoOP300+Prr7+OO++8M/bee+945513ok6dOtljru55njt3bhx44IGx2WabRd++faNOnToxY8aMGDZsWLHqHD9+fEybNi0GDx4cVapUiSOOOCIeeeSRVYKgUaNGRZcuXaJRo0bRu3fvyM/Pj8mTJ8cLL7wQvXv3zo5btmxZdOrUKTp06BA33XRTbLLJJsV+3n/rvRDxyyWh5557bhx55JHRu3fv+Pnnn+O9996L8ePHx3HHHbfGeT7//PMREXHCCScU699lZeecc05sttlm0b9//1i0aFFE/BIuX3nlldGxY8c4++yzY8qUKXHPPffEhAkT4o033ljrVYK/9d4s7usKAFKTAAA5MXjw4CQikgkTJqxxTO3atZOdd945e//yyy9PVv7xfeuttyYRkXz99ddrPMaECROSiEgGDx68St8+++yTREQyaNCg1fbts88+2fuvvPJKEhHJ5ptvnixcuDDb/sQTTyQRkdx+++3ZtqZNmyYnnXTSbx7z12o76aSTkqZNm2bvP/PMM0lEJNdcc02RcUceeWSSyWSSadOmZdsiIqlSpUqRtv/+979JRCR33nnnKuda2W233ZZERPLwww9n25YsWZK0a9cuqVGjRpG5N23aNOncufOvHm/lmnr27LnavieffDKJiOSVV15JkiRJ3nnnnSQikieffHKNx5sxY0ZSsWLF5Nprry3S/v777yeVKlUq0r6m5/npp5/+zdfgr+nVq1fSpEmTpLCwMEmSJBk5cmQSEck777yTHbNs2bKkoKAgadq0aTJv3rwij1/xuCT55fmOiKRv375FxhT3eS/Oe6Fr167JdtttV+J57rzzzkmdOnVWaf/hhx+Sr7/+OntbsGBBtm/F+7tDhw7JsmXLsu1z585NqlSpkhx44IHJ8uXLs+0DBw5MIiJ54IEHsm3FfR8V971ZnNcVAKTJ5XsAsAGrUaPGr34L34qVMM8+++xabwqel5cXp5xySrHHd+/ePWrWrJm9f+SRR0ajRo3iX//611qdv7j+9a9/RcWKFePcc88t0n7BBRdEkiTx73//u0h7x44do3nz5tn7O+64Y9SqVSs+/fTT3zxPfn5+HHvssdm2ypUrx7nnnhs//PBDvPrqq6Uwm1+3YsXKiy++GD/++ONqxwwbNiwKCwvjqKOOim+++SZ7y8/Pj6233jpeeeWVIuNX9zyveP288MILsXTp0hLVuGzZsnj88cfj6KOPzl5S+vvf/z4aNGgQjzzySHbcO++8E9OnT4/zzjuvyMqtiFjlUtSIWGWj9OI+78V5L9SpUye++OKLmDBhQonmunDhwqhRo8Yq7Zdeemlsttlm2dvqVludccYZUbFixez9l156KZYsWRLnnXdeVKhQoci4WrVqrdO3+P3We7M4rysASJNQCgA2YD/88EORXzL/19FHHx3t27eP008/PRo2bBjHHHNMPPHEEyUKqDbffPMSbWq+9dZbF7mfyWSiRYsWpbqf0up89tln0bhx41X+PVq1apXtX9mWW265yjHq1q0b8+bN+83zbL311kUCg187T2laEdIUFBREnz594u9//3tsuumm0alTp7jrrruK7PszderUSJIktt566yLByGabbRaTJ0+OuXPnFjn26p7nffbZJ7p16xZXXnllbLrpptG1a9cYPHhwLF68+DdrHTlyZHz99dex2267xbRp02LatGkxffr02G+//eLRRx/NvgY/+eSTiIjYfvvtf/OYlSpVKrLvUkTxn/fivBcuvvjiqFGjRuy2226x9dZbR8+ePYtc3rcmNWvWjB9++GGV9nPOOSdGjRoVo0aNWuNlrwUFBavMJyJi2223LdJepUqV2Gqrrdbp9fVb783ivK4AIE1CKQDYQH3xxRexYMGCaNGixRrHVKtWLV577bV46aWX4sQTT4z33nsvjj766DjggAN+85vAVj5GaVvdCpiIKHZNpWHl1SkrS/5nc+y05OXlxU8//bTavhWrVqpWrZptu/nmm+O9996LSy65JH766ac499xzY7vttosvvvgiIiIKCwsjk8nEiBEjssHIyre//e1vRc6xuuc5k8nEU089FWPHjo1evXrFl19+Gaeeemq0bdt2tSHMylashjrqqKNi6623zt4ef/zx+PLLL9dqRVleXt4qYWBxFee90KpVq5gyZUo89thj0aFDh/jnP/8ZHTp0iMsvv/xXj92yZcuYP39+fPnll0Xat9lmm+jYsWN07NixyHP3v3WtrfXxPvqt1xUApEkoBQAbqIceeigiIjp16vSr4ypUqBD7779/3HLLLfHhhx/GtddeG6NHj85evrWmX2zX1tSpU4vcT5Ikpk2bVuSb8urWrbvKN4lFrLrKqCS1NW3aNGbNmrXK5YwfffRRtr80NG3aNKZOnbrKarN1PU/Tpk1jypQpq+1b0f6/x95hhx3iL3/5S7z22mvxf//3f/Hll1/GoEGDIiKiefPmkSRJFBQUZIORlW977LFHsWvbY4894tprr4233norHnnkkZg0aVI89thjaxy/aNGiePbZZ+Poo4+OJ598cpVbo0aNsqHViksoP/jgg2LXs7KSPO+/9V6IiKhevXocffTRMXjw4Pj888+jc+fOce2118bPP/+8xhq6dOkSEVHkssS1taLe/30tLFmyJKZPn15kPsV9H61QnPdmxK+/rgAgTUIpANgAjR49Oq6++uooKCiI448/fo3jvvvuu1Xa2rRpExGRvQSrevXqERGr/eV2bTz44INFAoKnnnoqvvrqqzjooIOybc2bN49x48bFkiVLsm0vvPDCKl9PX5LaDj744Fi+fHkMHDiwSPutt94amUymyPnXxcEHHxyzZ8+Oxx9/PNu2bNmyuPPOO6NGjRqxzz77rPVxx40bFxMnTizSPn/+/HjkkUeiTZs2kZ+fHxG/7GG0bNmyIuN22GGHqFChQvZ5PeKII6JixYpx5ZVXrrL6K0mS+Pbbb3+zpnnz5q3y2P99/azO008/HYsWLYqePXvGkUceucqtS5cu8c9//jMWL14cu+yySxQUFMRtt922yvNcnFVrxX3ei/Ne+N9/kypVqkTr1q0jSZJf3VPrqKOOitatW8fVV18d48aNW+2Y4q7A69ixY1SpUiXuuOOOIo+5//77Y8GCBdG5c+dsW3HfRyv81nuzOK8rAEhTpVwXAAAbu3//+9/x0UcfxbJly2LOnDkxevToGDVqVDRt2jSee+65NV4WFBFx1VVXxWuvvRadO3eOpk2bxty5c+Puu++OLbbYIjp06BARv/xiW6dOnRg0aFDUrFkzqlevHrvvvvsqe90UV7169aJDhw5xyimnxJw5c+K2226LFi1axBlnnJEdc/rpp8dTTz0Vf/jDH+Koo46KTz75JB5++OEiG4+XtLZDDjkk9ttvv7j00ktjxowZsdNOO8XIkSPj2WefjfPOO2+VY6+tHj16xN/+9rc4+eSTY+LEidGsWbN46qmn4o033ojbbrvtV/f4+jV9+/aNJ598Mvbee+8488wzo2XLljFr1qwYMmRIfPXVVzF48ODs2NGjR0evXr3ij3/8Y2yzzTaxbNmyeOihh6JixYrRrVu3iPjl3+6aa66Jfv36xYwZM+Kwww6LmjVrxvTp0+Ppp5+OHj16xJ///OdfrWno0KFx9913x+GHHx7NmzeP77//Pu67776oVatWHHzwwWt83COPPBL169ePPffcc7X9hx56aNx3330xfPjwOOKII+Kee+6JQw45JNq0aROnnHJKNGrUKD766KOYNGlSvPjii79aY3Gf9+K8Fw488MDIz8+P9u3bR8OGDWPy5MkxcODA6Ny5868+r5UrV46nn346OnXqFB06dIgjjjgi9tprr6hevXp8+eWX8dxzz2VXXf2WzTbbLPr16xdXXnll/OEPf4hDDz00pkyZEnfffXf87ne/ixNOOCE7trjvoxV+671ZnNcVAKQqF1/5BwD8/6+MX3GrUqVKkp+fnxxwwAHJ7bffXuSr3Ve4/PLLk5V/fL/88stJ165dk8aNGydVqlRJGjdunBx77LHJxx9/XORxzz77bNK6deukUqVKSUQkgwcPTpLkl6+W32677VZb35q+dv7RRx9N+vXrlzRo0CCpVq1a0rlz5+Szzz5b5fE333xzsvnmmyd5eXlJ+/btk7feemuVY/5abSeddFLStGnTImO///775Pzzz08aN26cVK5cOdl6662Tv/71r0lhYWGRcRGR9OzZc5WamjZtmpx00kmrne/K5syZk5xyyinJpptumlSpUiXZYYcdsnX97/E6d+78m8db4YsvvkhOP/30ZPPNN08qVaqU1KtXL+nSpUsybty4IuM+/fTT5NRTT02aN2+eVK1aNalXr16y3377JS+99NIqx/znP/+ZdOjQIalevXpSvXr1pGXLlknPnj2TKVOmZMes6Xl+++23k2OPPTbZcsstk7y8vKRBgwZJly5dkrfeeutX/20qVaqUnHjiiWsc8+OPPyabbLJJcvjhh2fbXn/99eSAAw5IatasmVSvXj3ZcccdkzvvvDPbf9JJJyXVq1df7fGK87wX573wt7/9Ldl7772T+vXrJ3l5eUnz5s2TCy+8MFmwYMEa57Ky+fPnJ1dddVWy8847JzVq1EiqVKmSNGnSJDnyyCOT559/vsjYFe/vCRMmrPZYAwcOTFq2bJlUrlw5adiwYXL22Wcn8+bNW2Vccd5HxX1vluR1BQBpyCRJjnb7BAAA1tmYMWNiv/32iyeffDKOPPLIXJcDAMVmTykAAAAAUieUAgAAACB1QikAAAAAUmdPKQAAAABSZ6UUAAAAAKkTSgEAAACQukq5LqAsKCwsjFmzZkXNmjUjk8nkuhwAAACADVaSJPH9999H48aNo0KFNa+HEkoVw6xZs6JJkya5LgMAAACgzJg5c2ZsscUWa+wXShVDzZo1I+KXf8xatWrluBoAAACADdfChQujSZMm2TxlTYRSxbDikr1atWoJpQAAAACK4be2QLLROQAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpE0oBAAAAkDqhFAAAAACpq5TrAgAAgOJr1nd4aueaMaBzaucCYONjpRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJA6oRQAAAAAqRNKAQAAAJC6nIZSy5cvj8suuywKCgqiWrVq0bx587j66qsjSZLsmCRJon///tGoUaOoVq1adOzYMaZOnVrkON99910cf/zxUatWrahTp06cdtpp8cMPPxQZ895778Vee+0VVatWjSZNmsSNN96YyhwBAAAAWFVOQ6kbbrgh7rnnnhg4cGBMnjw5brjhhrjxxhvjzjvvzI658cYb44477ohBgwbF+PHjo3r16tGpU6f4+eefs2OOP/74mDRpUowaNSpeeOGFeO2116JHjx7Z/oULF8aBBx4YTZs2jYkTJ8Zf//rXuOKKK+Lee+9Ndb4AAAAA/CKTrLwsKWVdunSJhg0bxv33359t69atW1SrVi0efvjhSJIkGjduHBdccEH8+c9/joiIBQsWRMOGDWPIkCFxzDHHxOTJk6N169YxYcKE2HXXXSMiYsSIEXHwwQfHF198EY0bN4577rknLr300pg9e3ZUqVIlIiL69u0bzzzzTHz00Ue/WefChQujdu3asWDBgqhVq9Z6+JcAAIDiadZ3eGrnmjGgc2rnAqD8KG6OktOVUnvuuWe8/PLL8fHHH0dExH//+994/fXX46CDDoqIiOnTp8fs2bOjY8eO2cfUrl07dt999xg7dmxERIwdOzbq1KmTDaQiIjp27BgVKlSI8ePHZ8fsvffe2UAqIqJTp04xZcqUmDdv3nqfJwAAAABFVcrlyfv27RsLFy6Mli1bRsWKFWP58uVx7bXXxvHHHx8REbNnz46IiIYNGxZ5XMOGDbN9s2fPjgYNGhTpr1SpUtSrV6/ImIKCglWOsaKvbt26RfoWL14cixcvzt5fuHDhuk4VAAAAgJXkdKXUE088EY888kj84x//iLfffjuGDh0aN910UwwdOjSXZcX1118ftWvXzt6aNGmS03oAAAAAypuchlIXXnhh9O3bN4455pjYYYcd4sQTT4zzzz8/rr/++oiIyM/Pj4iIOXPmFHncnDlzsn35+fkxd+7cIv3Lli2L7777rsiY1R1j5XOsrF+/frFgwYLsbebMmaUwWwAAAABWyGko9eOPP0aFCkVLqFixYhQWFkZEREFBQeTn58fLL7+c7V+4cGGMHz8+2rVrFxER7dq1i/nz58fEiROzY0aPHh2FhYWx++67Z8e89tprsXTp0uyYUaNGxbbbbrvKpXsREXl5eVGrVq0iNwAAAABKT05DqUMOOSSuvfbaGD58eMyYMSOefvrpuOWWW+Lwww+PiIhMJhPnnXdeXHPNNfHcc8/F+++/H927d4/GjRvHYYcdFhERrVq1ij/84Q9xxhlnxJtvvhlvvPFG9OrVK4455pho3LhxREQcd9xxUaVKlTjttNNi0qRJ8fjjj8ftt98effr0ydXUAQAAADZqOd3o/M4774zLLrsszjnnnJg7d240btw4zjzzzOjfv392zEUXXRSLFi2KHj16xPz586NDhw4xYsSIqFq1anbMI488Er169Yr9998/KlSoEN26dYs77rgj21+7du0YOXJk9OzZM9q2bRubbrpp9O/fP3r06JHqfAEAAAD4RSZJkiTXRWzoFi5cGLVr144FCxa4lA8AgJxq1nd4aueaMaBzaucCoPwobo6S08v3AAAAANg4CaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUVcp1AQClqVnf4ameb8aAzqmeDwAAoLywUgoAAACA1AmlAAAAAEidUAoAAACA1AmlAAAAAEidUAoAAACA1AmlAAAAAEidUAoAAACA1AmlAAAAAEhdpVwXAAAAQNnVrO/wVM83Y0DnVM8HrD9WSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKnLeSj15ZdfxgknnBD169ePatWqxQ477BBvvfVWtj9Jkujfv380atQoqlWrFh07doypU6cWOcZ3330Xxx9/fNSqVSvq1KkTp512Wvzwww9Fxrz33nux1157RdWqVaNJkyZx4403pjI/AAAAAFaV01Bq3rx50b59+6hcuXL8+9//jg8//DBuvvnmqFu3bnbMjTfeGHfccUcMGjQoxo8fH9WrV49OnTrFzz//nB1z/PHHx6RJk2LUqFHxwgsvxGuvvRY9evTI9i9cuDAOPPDAaNq0aUycODH++te/xhVXXBH33ntvqvMFAAAA4BeVcnnyG264IZo0aRKDBw/OthUUFGT/O0mSuO222+Ivf/lLdO3aNSIiHnzwwWjYsGE888wzccwxx8TkyZNjxIgRMWHChNh1110jIuLOO++Mgw8+OG666aZo3LhxPPLII7FkyZJ44IEHokqVKrHddtvFu+++G7fcckuR8AoAAACAdOR0pdRzzz0Xu+66a/zxj3+MBg0axM477xz33Xdftn/69Okxe/bs6NixY7atdu3asfvuu8fYsWMjImLs2LFRp06dbCAVEdGxY8eoUKFCjB8/Pjtm7733jipVqmTHdOrUKaZMmRLz5s1bpa7FixfHwoULi9wAAAAAKD05DaU+/fTTuOeee2LrrbeOF198Mc4+++w499xzY+jQoRERMXv27IiIaNiwYZHHNWzYMNs3e/bsaNCgQZH+SpUqRb169YqMWd0xVj7Hyq6//vqoXbt29takSZNSmC0AAAAAK+Q0lCosLIxddtklrrvuuth5552jR48eccYZZ8SgQYNyWVb069cvFixYkL3NnDkzp/UAAAAAlDc5DaUaNWoUrVu3LtLWqlWr+PzzzyMiIj8/PyIi5syZU2TMnDlzsn35+fkxd+7cIv3Lli2L7777rsiY1R1j5XOsLC8vL2rVqlXkBgAAAEDpyWko1b59+5gyZUqRto8//jiaNm0aEb9sep6fnx8vv/xytn/hwoUxfvz4aNeuXUREtGvXLubPnx8TJ07Mjhk9enQUFhbG7rvvnh3z2muvxdKlS7NjRo0aFdtuu22Rb/oDAAAAIB05DaXOP//8GDduXFx33XUxbdq0+Mc//hH33ntv9OzZMyIiMplMnHfeeXHNNdfEc889F++//3507949GjduHIcddlhE/LKy6g9/+EOcccYZ8eabb8Ybb7wRvXr1imOOOSYaN24cERHHHXdcVKlSJU477bSYNGlSPP7443H77bdHnz59cjV1AAAAgI1apVye/He/+108/fTT0a9fv7jqqquioKAgbrvttjj++OOzYy666KJYtGhR9OjRI+bPnx8dOnSIESNGRNWqVbNjHnnkkejVq1fsv//+UaFChejWrVvccccd2f7atWvHyJEjo2fPntG2bdvYdNNNo3///tGjR49U5wsAAADALzJJkiS5LmJDt3Dhwqhdu3YsWLDA/lKwgWvWd3iq55sxoHOq5wOANH/W+TlHcfj8Bfyv4uYoOb18DwAAAICNk1AKAAAAgNQJpQAAAABInVAKAAAAgNQJpQAAAABInVAKAAAAgNQJpQAAAABInVAKAAAAgNQJpQAAAABInVAKAAAAgNQJpQAAAABInVAKAAAAgNQJpQAAAABInVAKAAAAgNSVOJQaOnRoDB8+PHv/oosuijp16sSee+4Zn332WakWBwAAAED5VOJQ6rrrrotq1apFRMTYsWPjrrvuihtvvDE23XTTOP/880u9QAAAAADKn0olfcDMmTOjRYsWERHxzDPPRLdu3aJHjx7Rvn372HfffUu7PgAAAADKoRKvlKpRo0Z8++23ERExcuTIOOCAAyIiomrVqvHTTz+VbnUAAAAAlEslXil1wAEHxOmnnx4777xzfPzxx3HwwQdHRMSkSZOiWbNmpV0fAAAAAOVQiVdK3XXXXbHnnnvG119/Hf/85z+jfv36ERExceLEOPbYY0u9QAAAAADKnxKtlFq2bFnccccdcfHFF8cWW2xRpO/KK68s1cIAAAAAKL9KtFKqUqVKceONN8ayZcvWVz0AAAAAbARKfPne/vvvH6+++ur6qAUAAACAjUSJNzo/6KCDom/fvvH+++9H27Zto3r16kX6Dz300FIrDgAAAIDyqcSh1DnnnBMREbfccssqfZlMJpYvX77uVQEAAABQrpU4lCosLFwfdQAAAACwESnxnlIr+/nnn0urDgAAAAA2IiUOpZYvXx5XX311bL755lGjRo349NNPIyLisssui/vvv7/UCwQAAACg/ClxKHXttdfGkCFD4sYbb4wqVapk27fffvv4+9//XqrFAQAAAFA+lTiUevDBB+Pee++N448/PipWrJht32mnneKjjz4q1eIAAAAAKJ9KHEp9+eWX0aJFi1XaCwsLY+nSpaVSFAAAAADlW4lDqdatW8f//d//rdL+1FNPxc4771wqRQEAAABQvlUq6QP69+8fJ510Unz55ZdRWFgYw4YNiylTpsSDDz4YL7zwwvqoEQAAAIBypsQrpbp27RrPP/98vPTSS1G9evXo379/TJ48OZ5//vk44IAD1keNAAAAAJQzJV4pFRGx1157xahRo0q7FgAAAAA2EiVeKTVz5sz44osvsvfffPPNOO+88+Lee+8t1cIAAAAAKL9KHEodd9xx8corr0RExOzZs6Njx47x5ptvxqWXXhpXXXVVqRcIAAAAQPlT4lDqgw8+iN122y0iIp544onYYYcd4j//+U888sgjMWTIkNKuDwAAAIByqMSh1NKlSyMvLy8iIl566aU49NBDIyKiZcuW8dVXX5VudQAAAACUSyUOpbbbbrsYNGhQ/N///V+MGjUq/vCHP0RExKxZs6J+/fqlXiAAAAAA5U+JQ6kbbrgh/va3v8W+++4bxx57bOy0004REfHcc89lL+sDAAAAgF9TqaQP2HfffeObb76JhQsXRt26dbPtPXr0iE022aRUiwMAAACgfCpxKBURUbFixSKBVEREs2bNSqMeAAAAADYCxQ6l6tatG5lMZpX22rVrxzbbbBN//vOf44ADDijV4gAAAAAon4odSt12222rbZ8/f35MnDgxunTpEk899VQccsghpVUbAAAAAOVUsUOpk0466Vf727RpE9dff71QCgAAAIDfVOJv31uTLl26xEcffVRahwMAAACgHCu1UGrx4sVRpUqV0jocAAAAAOVYqYVS999/f7Rp06a0DgcAAABAOVbsPaX69Omz2vYFCxbE22+/HR9//HG89tprpVYYAAAAAOVXsUOpd955Z7XttWrVigMOOCCGDRsWBQUFpVYYAAAAAOVXsUOpV155ZX3WAQAAAEAxNOs7PLVzzRjQeb0du9T2lAIAAACA4hJKAQAAAJA6oRQAAAAAqRNKAQAAAJC6YoVSu+yyS8ybNy8iIq666qr48ccf12tRAAAAAJRvxQqlJk+eHIsWLYqIiCuvvDJ++OGH9VoUAAAAAOVbpeIMatOmTZxyyinRoUOHSJIkbrrppqhRo8Zqx/bv379UCwQAAACg/ClWKDVkyJC4/PLL44UXXohMJhP//ve/o1KlVR+ayWSEUgAAAAD8pmKFUttuu2089thjERFRoUKFePnll6NBgwbrtTAAAAAAyq9ihVIrKywsXB91AAAAALARKXEoFRHxySefxG233RaTJ0+OiIjWrVtH7969o3nz5qVaHAAAAADlU7G+fW9lL774YrRu3TrefPPN2HHHHWPHHXeM8ePHx3bbbRejRo1aHzUCAAAAUM6UeKVU37594/zzz48BAwas0n7xxRfHAQccUGrFAQAAAFA+lXil1OTJk+O0005bpf3UU0+NDz/8sFSKAgAAAKB8K3Eotdlmm8W77767Svu7777rG/kAAAAAKJYSX753xhlnRI8ePeLTTz+NPffcMyIi3njjjbjhhhuiT58+pV4gAAAAAOVPiUOpyy67LGrWrBk333xz9OvXLyIiGjduHFdccUWce+65pV4gAAAAAOVPiUOpTCYT559/fpx//vnx/fffR0REzZo1S70wAAAAAMqvEodSKxNGAQAAALA2SrzROQAAAACsK6EUAAAAAKkTSgEAAACQuhKFUkuXLo39998/pk6dur7qAQAAAGAjUKJQqnLlyvHee++tr1oAAAAA2EiU+PK9E044Ie6///71UQsAAAAAG4lKJX3AsmXL4oEHHoiXXnop2rZtG9WrVy/Sf8stt5RacQAAAACUTyUOpT744IPYZZddIiLi448/LtKXyWRKpyoAAAAAyrUSh1KvvPLK+qgDAAAAgI1IiUOpFaZNmxaffPJJ7L333lGtWrVIksRKKQCAMqJZ3+Gpnm/GgM6png8A2PCVeKPzb7/9Nvbff//YZptt4uCDD46vvvoqIiJOO+20uOCCC0q9QAAAAADKnxKHUueff35Urlw5Pv/889hkk02y7UcffXSMGDGiVIsDAAAAoHwq8eV7I0eOjBdffDG22GKLIu1bb711fPbZZ6VWGAAAAADlV4lXSi1atKjICqkVvvvuu8jLyyuVogAAAAAo30ocSu21117x4IMPZu9nMpkoLCyMG2+8Mfbbb79SLQ4AAACA8qnEl+/deOONsf/++8dbb70VS5YsiYsuuigmTZoU3333Xbzxxhvro0YAAAAAypkSr5Tafvvt4+OPP44OHTpE165dY9GiRXHEEUfEO++8E82bN18fNQIAAABQzpR4pVRERO3atePSSy8t7VoAAAAA2EisVSg1b968uP/++2Py5MkREdG6des45ZRTol69eqVaHABALjXrOzy1c80Y0Dm1cwEAbAhKfPnea6+9Fs2aNYs77rgj5s2bF/PmzYs77rgjCgoK4rXXXlsfNQIAAABQzpR4pVTPnj3j6KOPjnvuuScqVqwYERHLly+Pc845J3r27Bnvv/9+qRcJAAAAQPlS4lBq2rRp8dRTT2UDqYiIihUrRp8+feLBBx8s1eLY8LmsAQAAAFgbJb58b5dddsnuJbWyyZMnx0477VQqRQEAAABQvhVrpdR7772X/e9zzz03evfuHdOmTYs99tgjIiLGjRsXd911VwwYMGD9VAkAAABAuVKsUKpNmzaRyWQiSZJs20UXXbTKuOOOOy6OPvro0qsOAAAAgHKpWKHU9OnT13cdAAAAAGxEihVKNW3adH3XAQAAAMBGpMQbnUdEzJo1K5544okYOHBg3HHHHUVua2vAgAGRyWTivPPOy7b9/PPP0bNnz6hfv37UqFEjunXrFnPmzCnyuM8//zw6d+4cm2yySTRo0CAuvPDCWLZsWZExY8aMiV122SXy8vKiRYsWMWTIkLWuEwAAAIB1V6yVUisbMmRInHnmmVGlSpWoX79+ZDKZbF8mk4lzzz23xEVMmDAh/va3v8WOO+5YpP3888+P4cOHx5NPPhm1a9eOXr16xRFHHBFvvPFGREQsX748OnfuHPn5+fGf//wnvvrqq+jevXtUrlw5rrvuuoj45dLDzp07x1lnnRWPPPJIvPzyy3H66adHo0aNolOnTiWuFQAAAIB1V+KVUpdddln0798/FixYEDNmzIjp06dnb59++mmJC/jhhx/i+OOPj/vuuy/q1q2bbV+wYEHcf//9ccstt8Tvf//7aNu2bQwePDj+85//xLhx4yIiYuTIkfHhhx/Gww8/HG3atImDDjoorr766rjrrrtiyZIlERExaNCgKCgoiJtvvjlatWoVvXr1iiOPPDJuvfXWEtcKAAAAQOkocSj1448/xjHHHBMVKqzVlX+r6NmzZ3Tu3Dk6duxYpH3ixImxdOnSIu0tW7aMLbfcMsaOHRsREWPHjo0ddtghGjZsmB3TqVOnWLhwYUyaNCk75n+P3alTp+wxVmfx4sWxcOHCIjcAAAAASk+Jk6XTTjstnnzyyVI5+WOPPRZvv/12XH/99av0zZ49O6pUqRJ16tQp0t6wYcOYPXt2dszKgdSK/hV9vzZm4cKF8dNPP622ruuvvz5q166dvTVp0mSt5gcAAADA6pV4T6nrr78+unTpEiNGjIgddtghKleuXKT/lltuKdZxZs6cGb17945Ro0ZF1apVS1rGetWvX7/o06dP9v7ChQsFUwAAAAClaK1CqRdffDG23XbbiIhVNjovrokTJ8bcuXNjl112ybYtX748XnvttRg4cGC8+OKLsWTJkpg/f36R1VJz5syJ/Pz8iIjIz8+PN998s8hxV3w738pj/vcb++bMmRO1atWKatWqrba2vLy8yMvLK/ZcAAAAACiZEodSN998czzwwANx8sknr9OJ999//3j//feLtJ1yyinRsmXLuPjii6NJkyZRuXLlePnll6Nbt24RETFlypT4/PPPo127dhER0a5du7j22mtj7ty50aBBg4iIGDVqVNSqVStat26dHfOvf/2ryHlGjRqVPQYAAAAA6StxKJWXlxft27df5xPXrFkztt9++yJt1atXj/r162fbTzvttOjTp0/Uq1cvatWqFX/605+iXbt2sccee0RExIEHHhitW7eOE088MW688caYPXt2/OUvf4mePXtmVzqdddZZMXDgwLjooovi1FNPjdGjR8cTTzwRw4cPX+c5AAAAALB2SrzRee/evePOO+9cH7Ws4tZbb40uXbpEt27dYu+99478/PwYNmxYtr9ixYrxwgsvRMWKFaNdu3ZxwgknRPfu3eOqq67KjikoKIjhw4fHqFGjYqeddoqbb745/v73v0enTp1SmQMAAAAAqyrxSqk333wzRo8eHS+88EJst912q2x0vnJoVFJjxowpcr9q1apx1113xV133bXGxzRt2nSVy/P+17777hvvvPPOWtcFAAAAQOkqcShVp06dOOKII9ZHLQAAAABsJEocSg0ePHh91AEbnGZ909t3bMaAzqmdCwAAADYEJd5TCgAAAADWVYlXShUUFEQmk1lj/6effrpOBQEAAABQ/pU4lDrvvPOK3F+6dGm88847MWLEiLjwwgtLqy4AAAAAyrESh1K9e/debftdd90Vb7311joXBAAAAED5V2p7Sh100EHxz3/+s7QOBwAAAEA5Vmqh1FNPPRX16tUrrcMBAAAAUI6V+PK9nXfeuchG50mSxOzZs+Prr7+Ou+++u1SLAwAAAKB8KnEoddhhhxW5X6FChdhss81i3333jZYtW5ZWXQAAAACUYyUOpS6//PL1UQcAAAAAG5FS21MKAAAAAIqr2CulKlSoUGQvqdXJZDKxbNmydS4KAAAAgPKt2KHU008/vca+sWPHxh133BGFhYWlUhQAAABsCJr1HZ7auWYM6JzauWBDUOxQqmvXrqu0TZkyJfr27RvPP/98HH/88XHVVVeVanEAAAAAlE9rtafUrFmz4owzzogddtghli1bFu+++24MHTo0mjZtWtr1AQAAAFAOlSiUWrBgQVx88cXRokWLmDRpUrz88svx/PPPx/bbb7++6gMAAACgHCr25Xs33nhj3HDDDZGfnx+PPvroai/nAwAAAIDiKHYo1bdv36hWrVq0aNEihg4dGkOHDl3tuGHDhpVacQAAAACUT8UOpbp37x6ZTGZ91gIAAADARqLYodSQIUPWYxkAAAAAbEzW6tv3AAAAAGBdCKUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUVcp1AUD6mvUdnur5ZgzonOr5AAAA2PBZKQUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKSuUq4LAAAAKO+a9R2e6vlmDOic6vkA1oaVUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkrlKuCwAAAAAoTc36Dk/1fDMGdE71fOWFlVIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApK5SrgsAAMquZn2Hp3q+GQM6p3o+AADWHyulAAAAAEidUAoAAACA1AmlAAAAAEidUAoAAACA1AmlAAAAAEidUAoAAACA1AmlAAAAAEhdpVwXAADlXbO+w1M934wBnVM9HwAArA0rpQAAAABInZVSAGVImiturLYBAADWp5yGUtdff30MGzYsPvroo6hWrVrsueeeccMNN8S2226bHfPzzz/HBRdcEI899lgsXrw4OnXqFHfffXc0bNgwO+bzzz+Ps88+O1555ZWoUaNGnHTSSXH99ddHpUr/f3pjxoyJPn36xKRJk6JJkybxl7/8JU4++eQ0pwsAAAAbDFsMkGs5vXzv1VdfjZ49e8a4ceNi1KhRsXTp0jjwwANj0aJF2THnn39+PP/88/Hkk0/Gq6++GrNmzYojjjgi2798+fLo3LlzLFmyJP7zn//E0KFDY8iQIdG/f//smOnTp0fnzp1jv/32i3fffTfOO++8OP300+PFF19Mdb4AAAAA/CKnK6VGjBhR5P6QIUOiQYMGMXHixNh7771jwYIFcf/998c//vGP+P3vfx8REYMHD45WrVrFuHHjYo899oiRI0fGhx9+GC+99FI0bNgw2rRpE1dffXVcfPHFccUVV0SVKlVi0KBBUVBQEDfffHNERLRq1Spef/31uPXWW6NTp06pzxsAAABgY7dBbXS+YMGCiIioV69eRERMnDgxli5dGh07dsyOadmyZWy55ZYxduzYiIgYO3Zs7LDDDkUu5+vUqVMsXLgwJk2alB2z8jFWjFlxDAAAAADStcFsdF5YWBjnnXdetG/fPrbffvuIiJg9e3ZUqVIl6tSpU2Rsw4YNY/bs2dkxKwdSK/pX9P3amIULF8ZPP/0U1apVK9K3ePHiWLx4cfb+woUL132CAAAAAGRtMCulevbsGR988EE89thjuS4lrr/++qhdu3b21qRJk1yXBAAAAFCubBChVK9eveKFF16IV155JbbYYotse35+fixZsiTmz59fZPycOXMiPz8/O2bOnDmr9K/o+7UxtWrVWmWVVEREv379YsGCBdnbzJkz13mOAAAAAPx/OQ2lkiSJXr16xdNPPx2jR4+OgoKCIv1t27aNypUrx8svv5xtmzJlSnz++efRrl27iIho165dvP/++zF37tzsmFGjRkWtWrWidevW2TErH2PFmBXH+F95eXlRq1atIjcAAAAASk9O95Tq2bNn/OMf/4hnn302atasmd0Dqnbt2lGtWrWoXbt2nHbaadGnT5+oV69e1KpVK/70pz9Fu3btYo899oiIiAMPPDBat24dJ554Ytx4440xe/bs+Mtf/hI9e/aMvLy8iIg466yzYuDAgXHRRRfFqaeeGqNHj44nnngihg8fnrO5AwAAAGzMcrpS6p577okFCxbEvvvuG40aNcreHn/88eyYW2+9Nbp06RLdunWLvffeO/Lz82PYsGHZ/ooVK8YLL7wQFStWjHbt2sUJJ5wQ3bt3j6uuuio7pqCgIIYPHx6jRo2KnXbaKW6++eb4+9//Hp06dUp1vgAAAAD8IqcrpZIk+c0xVatWjbvuuivuuuuuNY5p2rRp/Otf//rV4+y7777xzjvvlLhGAAAAAErfBrHROQAAAAAbF6EUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQOqEUAAAAAKkTSgEAAACQukq5LgAAIiKa9R2e6vlmDOic6vkAAICirJQCAAAAIHVCKQAAAABSJ5QCAAAAIHVCKQAAAABSZ6NzAADKFV+cAABlg5VSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6oRSAAAAAKROKAUAAABA6irluoDyrlnf4ameb8aAzqmeDwAAAGBtWCkFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOoq5boAAACAiIhmfYener4ZAzqnej4AirJSCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASJ1QCgAAAIDUCaUAAAAASN1GFUrddddd0axZs6hatWrsvvvu8eabb+a6JAAAAICN0kYTSj3++OPRp0+fuPzyy+Ptt9+OnXbaKTp16hRz587NdWkAAAAAG52NJpS65ZZb4owzzohTTjklWrduHYMGDYpNNtkkHnjggVyXBgAAALDR2ShCqSVLlsTEiROjY8eO2bYKFSpEx44dY+zYsTmsDAAAAGDjVCnXBaThm2++ieXLl0fDhg2LtDds2DA++uijVcYvXrw4Fi9enL2/YMGCiIhYuHBhic9duPjHEj9mXaxNjesizfmZW+kpz6/L8jy3CK/L0uR1WTrK89wivOdKk9dl6fG6LD1el6WjPM8twnuuNHldlp4N/XW54jFJkvzquEzyWyPKgVmzZsXmm28e//nPf6Jdu3bZ9osuuiheffXVGD9+fJHxV1xxRVx55ZVplwkAAABQbsycOTO22GKLNfZvFCulNt1006hYsWLMmTOnSPucOXMiPz9/lfH9+vWLPn36ZO8XFhbGd999F/Xr149MJrPe6124cGE0adIkZs6cGbVq1Vrv50tTeZ5bRPmen7mVTeV5bhHle37mVnaV5/mZW9lUnucWUb7nZ25lV3men7mVTWnPLUmS+P7776Nx48a/Om6jCKWqVKkSbdu2jZdffjkOO+ywiPglaHr55ZejV69eq4zPy8uLvLy8Im116tRJodKiatWqVe7eCCuU57lFlO/5mVvZVJ7nFlG+52duZVd5np+5lU3leW4R5Xt+5lZ2lef5mVvZlObcateu/ZtjNopQKiKiT58+cdJJJ8Wuu+4au+22W9x2222xaNGiOOWUU3JdGgAAAMBGZ6MJpY4++uj4+uuvo3///jF79uxo06ZNjBgxYpXNzwEAAABY/zaaUCoiolevXqu9XG9Dk5eXF5dffvkqlxCWB+V5bhHle37mVjaV57lFlO/5mVvZVZ7nZ25lU3meW0T5np+5lV3leX7mVjZtqHPbKL59DwAAAIANS4VcFwAAAADAxkcoBQAAAEDqhFIAAAAApE4oBRARttcDAABI10b17Xsbqm+++SYeeOCBGDt2bMyePTsiIvLz82PPPfeMk08+OTbbbLMcVwjlX15eXvz3v/+NVq1a5boUAACAjYJv38uxCRMmRKdOnWKTTTaJjh07RsOGDSMiYs6cOfHyyy/Hjz/+GC+++GLsuuuuOa6U1fnpp59i4sSJUa9evWjdunWRvp9//jmeeOKJ6N69e46qWzeTJ0+OcePGRbt27aJly5bx0Ucfxe233x6LFy+OE044IX7/+9/nusS10qdPn9W233777XHCCSdE/fr1IyLilltuSbOs9WbRokXxxBNPxLRp06JRo0Zx7LHHZufIhuNPf/pTHHXUUbHXXnvluhRK6Kuvvop77rknXn/99fjqq6+iQoUKsdVWW8Vhhx0WJ598clSsWDHXJQIAbLCEUjm2xx57xE477RSDBg2KTCZTpC9JkjjrrLPivffei7Fjx+aowvVr5syZcfnll8cDDzyQ61JK7OOPP44DDzwwPv/888hkMtGhQ4d47LHHolGjRhHxS7DYuHHjWL58eY4rLbkRI0ZE165do0aNGvHjjz/G008/Hd27d4+ddtopCgsL49VXX42RI0eWyWCqQoUKsdNOO0WdOnWKtL/66qux6667RvXq1SOTycTo0aNzU+A6at26dbz++utRr169mDlzZuy9994xb9682GabbeKTTz6JSpUqxbhx46KgoCDXpZbY22+/HXXr1s3W/tBDD8WgQYPi888/j6ZNm0avXr3imGOOyXGVa6dChQqRyWSiefPmcdppp8VJJ50U+fn5uS6r1AwcODDefPPNOPjgg+OYY46Jhx56KK6//vooLCyMI444Iq666qqoVKnsLd5+6623omPHjtGiRYuoVq1ajB07No477rhYsmRJvPjii9G6desYMWJE1KxZM9elrrUlS5bEM888s9rV3F27do0qVarkuML1Z86cOfG3v/0t+vfvn+tS1toXX3wRderUiRo1ahRpX7p0aYwdOzb23nvvHFW2br799tt47733Yqeddop69erFN998E/fff38sXrw4/vjHP5bLVc9bbbVVvPjii7H11lvnupRSkyRJjBkzJvuHs06dOkXlypVzXdZa+eKLL6Jq1aqx6aabRkTE//3f/xX5jNKzZ89o165djqtcOzfffHMceeSR0bRp01yXst688MIL8eabb0anTp2iffv2MXr06Ljpppuyn1N69OiR6xLX2k8//RSPPvroav94tv/+++e6vF8k5FTVqlWTyZMnr7F/8uTJSdWqVVOsKF3vvvtuUqFChVyXsVYOO+ywpHPnzsnXX3+dTJ06NencuXNSUFCQfPbZZ0mSJMns2bPL7NzatWuXXHrppUmSJMmjjz6a1K1bN7nkkkuy/X379k0OOOCAXJW3Tq6//vqkoKAgefnll4u0V6pUKZk0aVKOqio9mUwmmTNnTpIkSXL88ccne+65ZzJ//vwkSZLk+++/Tzp27Jgce+yxuSxxre24447JqFGjkiRJkvvuuy+pVq1acu655yb33HNPct555yU1atRI7r///hxXuXYymUzy0ksvJb1790423XTTpHLlysmhhx6aPP/888ny5ctzXd46ufrqq5OaNWsm3bp1S/Lz85MBAwYk9evXT6655prkuuuuSzbbbLOkf//+uS5zrbRv3z654oorsvcfeuihZPfdd0+SJEm+++67pE2bNsm5556bq/LW2dSpU5OtttoqqVq1arLPPvskRx11VHLUUUcl++yzT1K1atWkRYsWydSpU3Nd5npTlj+jzJo1K/nd736XVKhQIalYsWJy4oknJt9//322vyx/Rhk/fnxSu3btJJPJJHXr1k3eeuutpKCgINl6662T5s2bJ9WqVUsmTpyY6zLX2u23377aW8WKFZN+/fpl75dFBx10UPYzybfffpvsvvvuSSaTSTbbbLOkQoUKScuWLZO5c+fmuMq1s9tuuyXPP/98kiRJ8swzzyQVKlRIDj300OTiiy9ODj/88KRy5crZ/rImk8kkFStWTDp27Jg89thjyeLFi3NdUqkaNGhQUqlSpaRt27ZJrVq1koceeiipWbNmcvrppydnnnlmUq1ateS2227LdZlrZerUqUnTpk2TBg0aJE2aNEkymUzSuXPnZPfdd08qVqyY/PGPf0yWLl2a6zIToVSONWvWLBk6dOga+4cOHZo0bdo0vYJK2bPPPvurt1tvvbXMfihq0KBB8t5772XvFxYWJmeddVay5ZZbJp988kmZ/sBXq1at7C8ay5cvTypVqpS8/fbb2f73338/adiwYa7KW2dvvvlmss022yQXXHBBsmTJkiRJymcotdVWWyUjR44s0v/GG28kTZo0yUVp66xatWrJjBkzkiRJkp133jm59957i/Q/8sgjSevWrXNR2jpb+XlbsmRJ8vjjjyedOnVKKlasmDRu3Di55JJLyuwv/82bN0/++c9/Jknyyy/5FStWTB5++OFs/7Bhw5IWLVrkqrx1Uq1ateSTTz7J3l++fHlSuXLlZPbs2UmSJMnIkSOTxo0b56q8ddaxY8eka9euyYIFC1bpW7BgQdK1a9fkwAMPzEFlpeO///3vr94ef/zxMvtzvHv37snuu++eTJgwIRk1alTStm3bZNddd02+++67JEl+CaUymUyOq1w7HTt2TE4//fRk4cKFyV//+tdkiy22SE4//fRs/ymnnJIcdthhOaxw3WQymWSLLbZImjVrVuSWyWSSzTffPGnWrFlSUFCQ6zLXyso/684+++ykdevWyaeffpokSZLMnDkzadu2bXLWWWflssS1Vr169excdt9992TAgAFF+u+8885k5513zkVp6yyTySSDBw9OunbtmlSuXDmpX79+0rt37+T999/PdWmlonXr1tnPlKNHj06qVq2a3HXXXdn+wYMHJ61atcpVeevkoIMOSs4888yksLAwSZIkGTBgQHLQQQclSZIkH3/8cdKsWbPk8ssvz2GFvxBK5djAgQOTvLy85Nxzz02effbZZNy4ccm4ceOSZ599Njn33HOTatWqFXlTlDWZTCapUKFCkslk1ngrqx/4atasmXz44YertPfs2TPZYostktdee63Mzq1WrVrJtGnTsvdr1KhR5BevGTNmlPkVfN9//33SvXv3ZMcdd0zef//9pHLlyuUmlFrxV8bGjRuv8oGhLD939evXT956660kSX4Jhd99990i/dOmTUuqVauWi9LW2cof1Ff22WefJZdffnnStGnTMvv/k2rVqmVXkCZJklSuXDn54IMPsvdnzJiRbLLJJrkobZ01bdo0ef3117P3Z82alWQymeTHH39MkiRJpk+fXmbfb0nyy3P3a790vPfee2X2PZckv/4ZZUV7WX3fNW7cOBk/fnz2/s8//5wccsghSZs2bZJvv/22TP/hrG7dutnPX0uWLEkqVKhQZK4TJ05MNt9881yVt87OPPPMpE2bNqt8xiwPfzxb+Wfdtttumzz77LNF+l966aUyG7jVrl07+e9//5skyS+fUVb89wrTpk0rsz/rVn7e5syZk9xwww1Jy5YtkwoVKiS/+93vknvvvTdZuHBhjqtce6v7nLLyz77p06eX2eduk002ST7++OPs/cWLFyeVK1dOvvnmmyRJflnV16xZs1yVl1Uh15cPbux69uwZQ4cOjfHjx0e3bt2iXbt20a5du+jWrVuMHz8+hgwZEuecc06uy1xrjRo1imHDhkVhYeFqb2+//XauS1xrLVu2jLfeemuV9oEDB0bXrl3j0EMPzUFVpaNZs2YxderU7P2xY8fGlltumb3/+eefZ/fOKqtq1KgRQ4cOjX79+kXHjh3L5N5fa7L//vvHLrvsEgsXLowpU6YU6fvss8/K7EbnBx10UNxzzz0REbHPPvvEU089VaT/iSeeiBYtWuSitPVmyy23jCuuuCKmT58eI0aMyHU5ayU/Pz8+/PDDiIiYOnVqLF++PHs/ImLSpEnRoEGDXJW3Tg477LA466yzYsSIEfHKK6/E8ccfH/vss09Uq1YtIiKmTJkSm2++eY6rXHt16tSJGTNmrLF/xowZq+zPV5bUq1cv7rvvvpg+ffoqt08//TReeOGFXJe41hYsWBB169bN3s/Ly4thw4ZFs2bNYr/99ou5c+fmsLp1s2TJkux7rHLlyrHJJptk9/GJiNh0003j22+/zVV562zQoEHRv3//6NSpUwwcODDX5ZS6FXvozps3L5o3b16kr0WLFjFr1qxclLXO9tlnn3j00UcjImLnnXeOMWPGFOl/5ZVXyvTPgxUaNGgQF110UUyePDnGjBkTrVu3jvPPP79M/15Qv379+OyzzyIiYtasWbFs2bL4/PPPs/2fffZZ1KtXL1flrZM6derE999/n73/448/xrJly7L7Qe64447x1Vdf5aq8rLK3q2g5dPTRR8fRRx8dS5cujW+++SYifvmBWlY3+ltZ27ZtY+LEidG1a9fV9mcymUjK6F77hx9+eDz66KNx4oknrtI3cODAKCwsjEGDBuWgsnV39tlnFwlptt9++yL9//73v8vkJuerc8wxx0SHDh1i4sSJ5WIDx8svv7zI/f/d3Pb5558vs9/wdsMNN0T79u1jn332iV133TVuvvnmGDNmTLRq1SqmTJkS48aNi6effjrXZa6Vpk2b/uq3tGUymTjggANSrKj0HH/88dG9e/fo2rVrvPzyy3HRRRfFn//85/j2228jk8nEtddeG0ceeWSuy1wr11xzTXz11VdxyCGHxPLly6Ndu3bx8MMPZ/szmUxcf/31Oaxw3Zx++unRvXv3uOyyy2L//fdf5RuCr7nmmvjTn/6U4yrXXtu2bWPWrFlr/H///Pnzy+xnlK222iree++9IptiV6pUKZ588sn44x//GF26dMlhdeumSZMm8emnn0azZs0iIop8yUzEL9+IuXJIVRYdfvjhsdtuu0X37t1j+PDhMXjw4FyXVGpOPvnkyMvLi6VLl8b06dNju+22y/bNnj27zAbdAwYMiL322itmzZoVHTp0iEsvvTQmTJiQ/Yzy+OOPl9nfC/73y7hW2GuvvWKvvfaKO+64Ix5//PGUqyo9Xbt2zX7JzHPPPRfdu3ePCy64IPslNBdeeGEceOCBuS5zrRxwwAHRp0+fGDRoUOTl5UW/fv2iTZs22S9g+fzzzzeMPwzmeqkW5dtrr72W/Pvf/15j/w8//JCMGTMmxYqAsmzevHnJxRdfnLRu3TqpWrVqUqVKlaRp06bJcccdl0yYMCHX5bEay5cvT6699tqkS5cuyXXXXZcUFhYmjz76aNKkSZOkfv36ycknn5z88MMPuS5znfz0009FNpEuTwYMGJA0atQoeynbisvaGjVqlNxwww25Lm+dDBs2LHnooYfW2P/dd98lQ4YMSbGi0nPRRRetcb+vpUuXJoceemiZ3VPqiiuuSB599NE19l9yySXJEUcckWJF609hYWFy3XXXJfn5+UnFihXL/OV7J598cpHb448/XqT/wgsvTDp16pSj6tbdtGnTkmOOOSapWbNm9lLgypUrJ3vuuWfy9NNP57q8tbamLQbKix9++CE544wzku233z7p0aNHsnjx4uSvf/1rUqVKlSSTyST77rtvmZ3/nDlzkj322CP7M7xp06ZF9gl+8sknkzvuuCOHFf4ikyRl9E9AAACkYvr06TF79uyI+OWSzIKCghxXxK9ZtmxZ/Pjjj1GrVq019n/55ZflYoXw//rxxx+jYsWKkZeXl+tSSs3EiRPj9ddfj+7duxe5LLO8WbRoUVSsWDGqVq2a61LWSZIkMXfu3CgsLCw3V79sjH7++edYunRpdlVRWTZ16tRYvHhxtGzZMipV2vAulrOnFAAAv6qgoCC77+WKQGrmzJlx6qmn5riy9acsz69SpUprDKQifrnE7corr0yxovR8++23cfbZZ+e6jFLVtm3b6N27d9StW7dMvy5/y3fffVem99JdIZPJRMOGDaNRo0bZQKo8P2/ldW5Vq1aNmjVrlov5bb311rH99tuvEkhtKHOzUgoAgBL773//G7vssku5+qKIlZXn+Zlb2VWe52duZVN5nltE+Z7fhjK3DW/tFgAAOffcc8/9av+nn36aUiXrR3men7mVXeV5fuZWNpXnuUWU7/mVlblZKQUAwCpWfPPQr31UzGQyOf8L69oqz/Mzt7I5t4jyPT9zM7cNUXmeX1mZmz2lAABYRaNGjWLYsGFRWFi42tvbb7+d6xLXSXmen7mVXeV5fuZWNpXnuUWU7/mVlbkJpQAAWEXbtm1j4sSJa+z/rb++bujK8/zMrWzOLaJ8z8/czG1DVJ7nV1bmZk8pAABWceGFF8aiRYvW2N+iRYt45ZVXUqyodJXn+Zlb2ZxbRPmen7mZ24aoPM+vrMzNnlIAAAAApM7lewAAAACkTigFAAAAQOqEUgAAAACkTigFAAAAQOqEUgAAAACkTigFAJADs2fPjt69e0eLFi2iatWq0bBhw2jfvn3cc8898eOPP+a6PACA9a5SrgsAANjYfPrpp9G+ffuoU6dOXHfddbHDDjtEXl5evP/++3HvvffG5ptvHoceeugqj1u6dGlUrlw5BxUDAJQ+K6UAAFJ2zjnnRKVKleKtt96Ko446Klq1ahVbbbVVdO3aNYYPHx6HHHJIRERkMpm455574tBDD43q1avHtddeGxER99xzTzRv3jyqVKkS2267bTz00EPZY8+YMSMymUy8++672bb58+dHJpOJMWPGRETEmDFjIpPJxPDhw2PHHXeMqlWrxh577BEffPBBav8GAABCKQCAFH377bcxcuTI6NmzZ1SvXn21YzKZTPa/r7jiijj88MPj/fffj1NPPTWefvrp6N27d1xwwQXxwQcfxJlnnhmnnHJKvPLKKyWu5cILL4ybb745JkyYEJtttlkccsghsXTp0rWeGwBASQilAABSNG3atEiSJLbddtsi7ZtuumnUqFEjatSoERdffHG2/bjjjotTTjklttpqq9hyyy3jpptuipNPPjnOOeec2GabbaJPnz5xxBFHxE033VTiWi6//PI44IADYocddoihQ4fGnDlz4umnn17nOQIAFIdQCgBgA/Dmm2/Gu+++G9ttt10sXrw4277rrrsWGTd58uRo3759kbb27dvH5MmTS3zOdu3aZf+7Xr16se22267VcQAA1oaNzgEAUtSiRYvIZDIxZcqUIu1bbbVVRERUq1atSPuaLvFbkwoVfvmbY5Ik2TaX5AEAGyIrpQAAUlS/fv044IADYuDAgbFo0aISP75Vq1bxxhtvFGl74403onXr1hERsdlmm0VExFdffZXtX3nT85WNGzcu+9/z5s2Ljz/+OFq1alXimgAA1oaVUgAAKbv77rujffv2seuuu8YVV1wRO+64Y1SoUCEmTJgQH330UbRt23aNj73wwgvjqKOOip133jk6duwYzz//fAwbNixeeumliPhlpdUee+wRAwYMiIKCgpg7d2785S9/We2xrrrqqqhfv340bNgwLr300th0003jsMMOWx9TBgBYhVAKACBlzZs3j3feeSeuu+666NevX3zxxReRl5cXrVu3jj//+c9xzjnnrPGxhx12WNx+++1x0003Re/evaOgoCAGDx4c++67b3bMAw88EKeddlq0bds2tt1227jxxhvjwAMPXOVYAwYMiN69e8fUqVOjTZs28fzzz0eVKlXWx5QBAFaRSVbecAAAgHJvzJgxsd9++8W8efOiTp06uS4HANhI2VMKAAAAgNQJpQAAAABIncv3AAAAAEidlVIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApE4oBQAAAEDqhFIAAAAApO7/ARb8ej2JCljwAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "# Get the group with the highest number for each user\n",
    "user_groups = w1.argmax(axis=1)\n",
    "\n",
    "# Count the number of users in each group\n",
    "group_counts = pd.Series(user_groups).value_counts().sort_index()\n",
    "\n",
    "# Display the results\n",
    "print(\"Number of users in each group:\")\n",
    "print(group_counts)\n",
    "\n",
    "# Calculate and display the total number of users\n",
    "total_users = group_counts.sum()\n",
    "print(f\"\\nTotal number of users: {total_users}\")\n",
    "\n",
    "# Calculate and display the percentage of users in each group\n",
    "group_percentages = (group_counts / total_users * 100).round(2)\n",
    "print(\"\\nPercentage of users in each group:\")\n",
    "print(group_percentages)\n",
    "\n",
    "# Visualize the distribution\n",
    "plt.figure(figsize=(12, 6))\n",
    "group_counts.plot(kind='bar')\n",
    "plt.title('Distribution of Users Across Groups')\n",
    "plt.xlabel('Group')\n",
    "plt.ylabel('Number of Users')\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    20.000000\n",
      "mean      4.998500\n",
      "std       2.919319\n",
      "min       1.460000\n",
      "25%       3.007500\n",
      "50%       4.480000\n",
      "75%       6.160000\n",
      "max      12.540000\n",
      "Name: count, dtype: float64\n",
      "Mean: 5.00\n",
      "Standard deviation: 2.92\n",
      "Lower bound: -0.84\n",
      "Upper bound: 10.84\n",
      "Percentage of distribution within 2 standard deviations of the mean: 95.00%\n"
     ]
    }
   ],
   "source": [
    "print(group_percentages.describe())\n",
    "lower_bound = group_percentages.mean() - 2 * group_percentages.std()\n",
    "upper_bound = group_percentages.mean() + 2 * group_percentages.std()\n",
    "within_2std = group_percentages[(group_percentages >= lower_bound) & (group_percentages <= upper_bound)]\n",
    "percentage_within_2std = (len(within_2std) / len(group_percentages)) * 100\n",
    "\n",
    "print(f\"Mean: {group_percentages.mean():.2f}\")\n",
    "print(f\"Standard deviation: {group_percentages.std():.2f}\")\n",
    "print(f\"Lower bound: {lower_bound:.2f}\")\n",
    "print(f\"Upper bound: {upper_bound:.2f}\")\n",
    "print(f\"Percentage of distribution within 2 standard deviations of the mean: {percentage_within_2std:.2f}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_users(w1, user, n=10):\n",
    "    \"\"\"\n",
    "    Get the top n similar users for a given user\n",
    "    \"\"\"\n",
    "    user_groups = pd.DataFrame({'group':w1.argmax(axis=1), 'weight':w1.max(axis=1)})\n",
    "    user_group = user_groups.iloc[user] # get the group of the user\n",
    "    similar_users = user_groups[user_groups['group'] == user_group['group']] # get all users in the same group\n",
    "    similar_users = similar_users[similar_users.index != user] # remove the user itself\n",
    "    similar_users = similar_users.sort_values(by='weight', ascending=False) # sort by weight\n",
    "    return similar_users.head(n).index\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([677, 22009, 34995, 1366, 18222, 20708, 36790, 29792, 13223, 22917], dtype='int64')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_users(w1, 42, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 10 recommended animes for user 11137\n",
      "                                                                     name  rating\n",
      "anime_id                                                                         \n",
      "26                                                             Texhnolyze    10.0\n",
      "6377                                         Zan Sayonara Zetsubou Sensei    10.0\n",
      "376                                                    Elfen Lied Special    10.0\n",
      "14349                                               Little Witch Academia    10.0\n",
      "14267                         Persona 4 the Animation: The Factor of Hope    10.0\n",
      "4565                         Tengen Toppa Gurren Lagann Movie: Lagann-hen    10.0\n",
      "2246                                                             Mononoke    10.0\n",
      "4181                                                 Clannad: After Story    10.0\n",
      "1117                          Pokemon: Maboroshi no Pokemon Lugia Bakutan    10.0\n",
      "12711                                 Uta no☆Prince-sama♪ Maji Love 2000%    10.0\n",
      "2605                                             Sayonara Zetsubou Sensei    10.0\n",
      "597                                                      Neko no Ongaeshi    10.0\n",
      "441                         Shoujo Kakumei Utena: Adolescence Mokushiroku    10.0\n",
      "12079                                                  Black★Rock Shooter    10.0\n",
      "1033                                                         Sennen Joyuu    10.0\n",
      "10321                                 Uta no☆Prince-sama♪ Maji Love 1000%    10.0\n",
      "1002                                            Top wo Nerae 2! Diebuster    10.0\n",
      "9989             Ano Hi Mita Hana no Namae wo Bokutachi wa Mada Shiranai.    10.0\n",
      "3228                                        Zoku Sayonara Zetsubou Sensei    10.0\n",
      "9515                         Highschool of the Dead: Drifters of the Dead    10.0\n",
      "9201          Air Gear: Kuro no Hane to Nemuri no Mori - Break on the Sky    10.0\n",
      "3396                                          Gloria: Kindan no Ketsuzoku    10.0\n",
      "7044                               Zan Sayonara Zetsubou Sensei Bangaichi    10.0\n",
      "8919                                             Licca-chan no Nichiyoubi    10.0\n",
      "8795                                 Panty &amp; Stocking with Garterbelt    10.0\n",
      "3784                                Evangelion: 2.0 You Can (Not) Advance    10.0\n",
      "877                                                                  Nana    10.0\n",
      "567                                                             The Big O    10.0\n",
      "14515                                               Sasami-san@Ganbaranai    10.0\n",
      "15315                     Mondaiji-tachi ga Isekai kara Kuru Sou Desu yo?    10.0\n",
      "32189            Danganronpa 3: The End of Kibougamine Gakuen - Mirai-hen    10.0\n",
      "29803                                                            Overlord    10.0\n",
      "28621                            Subete ga F ni Naru: The Perfect Insider    10.0\n",
      "28025                                                     Tsukimonogatari    10.0\n",
      "1430                                       Lupin III: Cagliostro no Shiro    10.0\n",
      "25345                          Rance 01: Hikari wo Motomete The Animation    10.0\n",
      "24705                                          Ore, Twintail ni Narimasu.    10.0\n",
      "4872                                        Goku Sayonara Zetsubou Sensei    10.0\n",
      "339                                               Serial Experiments Lain    10.0\n",
      "759                                                      Tokyo Godfathers    10.0\n",
      "22319                                                         Tokyo Ghoul    10.0\n",
      "21855                                                      Hanamonogatari    10.0\n",
      "31181                                                     Owarimonogatari    10.0\n",
      "21439                           Uta no☆Prince-sama♪ Maji Love Revolutions    10.0\n",
      "1829                                                           Gedo Senki    10.0\n",
      "20709                                                           Sabagebu!    10.0\n",
      "1441                                               Ginga Tetsudou no Yoru    10.0\n",
      "721                                                         Princess Tutu    10.0\n",
      "16444     Mondaiji-tachi ga Isekai kara Kuru Sou Desu yo?: Onsen Manyuuki    10.0\n",
      "1943                                                              Paprika    10.0\n"
     ]
    }
   ],
   "source": [
    "def get_top_animes_for_user(user_id, w1, ratings, n_users=10, n_animes=10):\n",
    "    # Get top n similar users\n",
    "    similar_users = top_users(w1, user_id, n=n_users)\n",
    "    \n",
    "    # Get animes watched by these users\n",
    "    similar_users_ratings = ratings[ratings['user_id'].isin(similar_users)]\n",
    "    \n",
    "    # Get animes watched by the target user\n",
    "    target_user_animes = ratings[ratings['user_id'] == user_id]\n",
    "    watched_animes = set(target_user_animes['anime_id'])\n",
    "    \n",
    "    # Filter out animes that the target user has already watched\n",
    "    unwatched_animes = similar_users_ratings[~similar_users_ratings['anime_id'].isin(watched_animes)]\n",
    "    \n",
    "    # Rank the remaining animes based on their average rating\n",
    "    top_animes = unwatched_animes.groupby('anime_id')['rating'].mean().sort_values(ascending=False).head(n_animes)\n",
    "    \n",
    "    return top_animes\n",
    "\n",
    "# Example usage\n",
    "random_user_id = rating_test['user_id'].sample(n=1).iloc[0]\n",
    "user_id = random_user_id\n",
    "top_animes = get_top_animes_for_user(user_id, w1, ratings, n_users=10, n_animes=50)\n",
    "print(\"Top 10 recommended animes for user\", user_id)\n",
    "# Get the anime names for the top recommended animes\n",
    "top_anime_names = anime[anime['anime_id'].isin(top_animes.index)][['anime_id', 'name']]\n",
    "top_anime_names = top_anime_names.set_index('anime_id')\n",
    "\n",
    "# Merge the anime names with the top animes ratings\n",
    "top_animes_with_names = top_animes.to_frame().join(top_anime_names)\n",
    "\n",
    "# Print the top animes with their names and average ratings\n",
    "print(top_animes_with_names[['name', 'rating']].to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>anime_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>total_watch</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [user_id, anime_id, rating, total_watch]\n",
       "Index: []"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Choose 1 user_id randomly from rating_test\n",
    "rating_test[rating_test['user_id'] == user_id & rating_test['anime_id'].isin(top_animes.index)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature engineering\n",
    "- How popular the user likes their anime?\n",
    "- How many times the user has watched the anime genre?\n",
    "- How well the user rated a genre?\n",
    "- How to consider what's the base rate of liking an anime?\n",
    "\n",
    "### Evaluation\n",
    "- Instead of asking for a suggestion, get a validation example and see how well it would perform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   user_id  anime_id  rating  total_watch        members\n",
      "0        1        20      -1          153  198889.784314\n",
      "1        1        24      -1          153  198889.784314\n",
      "2        1        79      -1          153  198889.784314\n",
      "3        1       226      -1          153  198889.784314\n",
      "4        1       241      -1          153  198889.784314\n"
     ]
    }
   ],
   "source": [
    "# Calculate average members and average rating for each user\n",
    "user_stats = ratings.merge(anime[['anime_id', 'members']], on='anime_id', how='left')\n",
    "user_stats = user_stats.groupby('user_id').mean()['members'].reset_index()\n",
    "\n",
    "# Merge the calculated averages back to the ratings dataframe\n",
    "ratings = ratings.merge(user_stats, on='user_id', how='left')\n",
    "\n",
    "# Display the first few rows of the updated ratings dataframe\n",
    "print(ratings.head())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAG+CAYAAADP4E3NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAeQklEQVR4nO3df5DU9X348ddxHIsIp0EGA3qo4KBGfkhADFEaSESDDgmTSeIMiaWmJm2Djg1jGylJgaqRGLB0LDWpMVBmgiRkxDZqBGpLGK22esoEI8UgRm0MIqVy/GjW5fbz/cMvl578ur2895bP+XjMMM5++NxnX7zc4Z7s3t3WZVmWBQBAAj1qPQAA0H0ICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIJmahcXGjRtj2rRpMXjw4Kirq4sHH3yw4mtkWRaLFi2K4cOHR6FQiDPOOCNuv/329MMCAB3Ss1Z3vH///hg9enR84QtfiE996lOdusZNN90U69ati0WLFsXIkSNj9+7dsXv37sSTAgAdVXcivAlZXV1drFmzJqZPn952rFgsxty5c+P++++Pt956K0aMGBHf/OY3Y9KkSRERsWXLlhg1alQ8//zzcd5559VmcACgnRP2ayxuuOGGePLJJ2PVqlXxs5/9LD7zmc/Exz/+8fjFL34RERE//vGPY+jQofHQQw/FOeecE2effXZcf/31nrEAgBo6IcPi1VdfjWXLlsXq1atj4sSJMWzYsLj55pvjsssui2XLlkVExPbt2+OVV16J1atXx4oVK2L58uXR3Nwcn/70p2s8PQC8d9XsayyOZfPmzdHa2hrDhw9vd7xYLMZpp50WERHlcjmKxWKsWLGi7bz77rsvxo4dG1u3bvXyCADUwAkZFvv27Yv6+vpobm6O+vr6dr/Xt2/fiIgYNGhQ9OzZs118XHDBBRHxzjMewgIAut4JGRZjxoyJ1tbW2LlzZ0ycOPGI51x66aVx8ODBeOmll2LYsGEREfHiiy9GRMRZZ53VZbMCAL9Vs+8K2bdvX2zbti0i3gmJu+66KyZPnhz9+/ePIUOGxOc///l44oknYvHixTFmzJh4880347HHHotRo0bF1VdfHeVyOS6++OLo27dvLFmyJMrlcsyaNSsaGxtj3bp1tfgjAcB7Xs3CYsOGDTF58uTDjs+cOTOWL18epVIpbrvttlixYkX86le/igEDBsSHPvShWLBgQYwcOTIiIl5//fW48cYbY926dXHyySfH1KlTY/HixdG/f/+u/uMAAHGC/BwLAKB7OCG/3RQAyCdhAQAk0+XfFVIul+P111+Pfv36RV1dXVffPQDQCVmWxd69e2Pw4MHRo8fRn5fo8rB4/fXXo6mpqavvFgBI4LXXXoszzzzzqL/f5WHRr1+/iHhnsMbGxmTXLZVKsW7durjiiiuioaEh2XX5LTuuPjuuPjuuPjuuvlrsuKWlJZqamto+jx9Nl4fFoZc/Ghsbk4dFnz59orGx0QO5Suy4+uy4+uy4+uy4+mq54+N9GYMv3gQAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJNPlb5tOe2ff8nCtR+iwQn0Wd46v9RQAnMg8YwEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAyv1NYLFy4MOrq6uJP//RPE40DAORZp8Pi6aefju985zsxatSolPMAADnWszMftG/fvvjc5z4X9957b9x2223HPLdYLEaxWGy73dLSEhERpVIpSqVSZ+7+iA5dK+U1u0KhPqv1CB1W6PHOrHnbcZ7k9XGcJ3ZcfXZcfbXYcUfvqy7Lsoo/s82cOTP69+8ff/3Xfx2TJk2Kiy66KJYsWXLEc+fPnx8LFiw47PjKlSujT58+ld41AFADBw4ciBkzZsSePXuisbHxqOdV/IzFqlWr4tlnn42nn366Q+fPmTMnZs+e3Xa7paUlmpqa4oorrjjmYJUqlUqxfv36mDJlSjQ0NCS7brWNmL+21iN0WKFHFreOK+dux3mS18dxnthx9dlx9dVix4decTieisLitddei5tuuinWr18fvXv37tDHFAqFKBQKhx1vaGioyjKqdd1qKbbW1XqEiuVtx3lkx9Vnx9Vnx9XXlTvu6P1UFBbNzc2xc+fO+OAHP9h2rLW1NTZu3Bh/+7d/G8ViMerr6yubFADoNioKi4997GOxefPmdseuu+66OP/88+OrX/2qqACA97iKwqJfv34xYsSIdsdOPvnkOO200w47DgC89/jJmwBAMp36ORb/14YNGxKMAQB0B56xAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMhWFxT333BOjRo2KxsbGaGxsjAkTJsRPfvKTas0GAORMRWFx5plnxsKFC6O5uTmeeeaZ+OhHPxqf/OQn4+c//3m15gMAcqRnJSdPmzat3e3bb7897rnnnnjqqafiwgsvTDoYAJA/FYXF/9Xa2hqrV6+O/fv3x4QJE456XrFYjGKx2Ha7paUlIiJKpVKUSqXO3v1hDl0r5TW7QqE+q/UIHVbo8c6sedtxnuT1cZwndlx9dlx9tdhxR++rLsuyij6zbd68OSZMmBC/+c1vom/fvrFy5cq46qqrjnr+/PnzY8GCBYcdX7lyZfTp06eSuwYAauTAgQMxY8aM2LNnTzQ2Nh71vIrD4u23345XX3019uzZEz/60Y/iu9/9bvz0pz+ND3zgA0c8/0jPWDQ1NcWuXbuOOVilSqVSrF+/PqZMmRINDQ3JrlttI+avrfUIHVbokcWt48q523Ge5PVxnCd2XH12XH212HFLS0sMGDDguGFR8UshvXr1inPPPTciIsaOHRtPP/10/M3f/E185zvfOeL5hUIhCoXCYccbGhqqsoxqXbdaiq11tR6hYnnbcR7ZcfXZcfXZcfV15Y47ej+/88+xKJfL7Z6RAADeuyp6xmLOnDkxderUGDJkSOzduzdWrlwZGzZsiLVr8/N0PgBQPRWFxc6dO+P3f//349e//nWccsopMWrUqFi7dm1MmTKlWvMBADlSUVjcd9991ZoDAOgGvFcIAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDIVhcUdd9wRF198cfTr1y8GDhwY06dPj61bt1ZrNgAgZyoKi5/+9Kcxa9aseOqpp2L9+vVRKpXiiiuuiP3791drPgAgR3pWcvKjjz7a7vby5ctj4MCB0dzcHL/3e7+XdDBOXCPmr41ia12tx6jILxdeXesRAN4TKgqLd9uzZ09ERPTv3/+o5xSLxSgWi223W1paIiKiVCpFqVT6Xe6+nUPXSnnNrlCoz2o9QocVemTt/psneXlc5PVxnCd2XH12XH212HFH76suy7JOfZYol8vxiU98It566614/PHHj3re/PnzY8GCBYcdX7lyZfTp06czdw0AdLEDBw7EjBkzYs+ePdHY2HjU8zodFn/yJ38SP/nJT+Lxxx+PM88886jnHekZi6ampti1a9cxB6tUqVSK9evXx5QpU6KhoSHZdattxPy1tR6hwwo9srh1XDm+/kyPKJbz9VLI8/OvrPUIHZLXx3Ge2HH12XH11WLHLS0tMWDAgOOGRadeCrnhhhvioYceio0bNx4zKiIiCoVCFAqFw443NDRUZRnVum615O1rFSIiiuW63M2dp8dERP4ex3lkx9Vnx9XXlTvu6P1UFBZZlsWNN94Ya9asiQ0bNsQ555zTqeEAgO6porCYNWtWrFy5Mv7xH/8x+vXrFzt27IiIiFNOOSVOOumkqgwIAORHRT/H4p577ok9e/bEpEmTYtCgQW2/fvCDH1RrPgAgRyp+KQQA4Gi8VwgAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMhWHxcaNG2PatGkxePDgqKuriwcffLAKYwEAeVRxWOzfvz9Gjx4dS5curcY8AECO9az0A6ZOnRpTp07t8PnFYjGKxWLb7ZaWloiIKJVKUSqVKr37ozp0rZTX7AqF+qzWI3RYoUfW7r95kpfHRV4fx3lix9Vnx9VXix139L7qsizr9GeJurq6WLNmTUyfPv2o58yfPz8WLFhw2PGVK1dGnz59OnvXAEAXOnDgQMyYMSP27NkTjY2NRz2v6mFxpGcsmpqaYteuXcccrFKlUinWr18fU6ZMiYaGhmTXrbYR89fWeoQOK/TI4tZx5fj6Mz2iWK6r9TgVeX7+lbUeoUPy+jjOEzuuPjuuvlrsuKWlJQYMGHDcsKj4pZBKFQqFKBQKhx1vaGioyjKqdd1qKbbm6xN0RESxXJe7ufP0mIjI3+M4j+y4+uy4+rpyxx29H99uCgAkIywAgGQqfilk3759sW3btrbbL7/8cmzatCn69+8fQ4YMSTocAJAvFYfFM888E5MnT267PXv27IiImDlzZixfvjzZYABA/lQcFpMmTYrf4RtJAIBuzNdYAADJCAsAIBlhAQAkIywAgGSq/pM3u9qI+Wtz91MhAaC78IwFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJNOz1gNAVzj7lodrPUKHFOqzuHN8xIj5a6PYWhe/XHh1rUcCqIhnLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIpmdnPmjp0qXxrW99K3bs2BGjR4+Ou+++O8aPH596NnjPO/uWh2s9QsV+ufDqWo8A1FDFz1j84Ac/iNmzZ8e8efPi2WefjdGjR8eVV14ZO3furMZ8AECOVBwWd911V3zxi1+M6667Lj7wgQ/Et7/97ejTp09873vfq8Z8AECOVPRSyNtvvx3Nzc0xZ86ctmM9evSIyy+/PJ588skjfkyxWIxisdh2e8+ePRERsXv37iiVSp2Z+YhKpVIcOHAgepZ6RGu5Ltl1+a2e5SwOHCjbcRV1hx2fe/MPaz3CMRV6ZPG1MeW4aO4DUfz/O/73OR+r8VTdy6G/j//7v/87Ghoaaj1Ot1SLHe/duzciIrIsO+Z5FYXFrl27orW1NU4//fR2x08//fT4z//8zyN+zB133BELFiw47Pg555xTyV1zgphR6wHeA+y4+t694wGLazIG5NLevXvjlFNOOervd+qLNysxZ86cmD17dtvtcrkcu3fvjtNOOy3q6tL9i6ylpSWampritddei8bGxmTX5bfsuPrsuPrsuPrsuPpqseMsy2Lv3r0xePDgY55XUVgMGDAg6uvr44033mh3/I033oj3v//9R/yYQqEQhUKh3bFTTz21krutSGNjowdyldlx9dlx9dlx9dlx9XX1jo/1TMUhFX3xZq9evWLs2LHx2GOPtR0rl8vx2GOPxYQJEyqfEADoVip+KWT27Nkxc+bMGDduXIwfPz6WLFkS+/fvj+uuu64a8wEAOVJxWFxzzTXx5ptvxl/+5V/Gjh074qKLLopHH330sC/o7GqFQiHmzZt32MsupGPH1WfH1WfH1WfH1Xci77guO973jQAAdJD3CgEAkhEWAEAywgIASEZYAADJ5Cosli5dGmeffXb07t07LrnkkviP//iPY56/evXqOP/886N3794xcuTIeOSRR7po0vyqZMf33ntvTJw4Md73vvfF+973vrj88suP+/+Eyh/Hh6xatSrq6upi+vTp1R2wG6h0x2+99VbMmjUrBg0aFIVCIYYPH+7vi+OodMdLliyJ8847L0466aRoamqKr3zlK/Gb3/ymi6bNl40bN8a0adNi8ODBUVdXFw8++OBxP2bDhg3xwQ9+MAqFQpx77rmxfPnyqs95VFlOrFq1KuvVq1f2ve99L/v5z3+effGLX8xOPfXU7I033jji+U888URWX1+f3XnnndkLL7yQfe1rX8saGhqyzZs3d/Hk+VHpjmfMmJEtXbo0e+6557ItW7Zkf/AHf5Cdcsop2X/913918eT5UemOD3n55ZezM844I5s4cWL2yU9+smuGzalKd1wsFrNx48ZlV111Vfb4449nL7/8crZhw4Zs06ZNXTx5flS64+9///tZoVDIvv/972cvv/xytnbt2mzQoEHZV77ylS6ePB8eeeSRbO7cudkDDzyQRUS2Zs2aY56/ffv2rE+fPtns2bOzF154Ibv77ruz+vr67NFHH+2agd8lN2Exfvz4bNasWW23W1tbs8GDB2d33HHHEc//7Gc/m1199dXtjl1yySXZH/3RH1V1zjyrdMfvdvDgwaxfv37ZP/zDP1RrxNzrzI4PHjyYffjDH86++93vZjNnzhQWx1Hpju+5555s6NCh2dtvv91VI+ZepTueNWtW9tGPfrTdsdmzZ2eXXnppVefsDjoSFn/+53+eXXjhhe2OXXPNNdmVV15ZxcmOLhcvhRx6u/bLL7+87djx3q79ySefbHd+RMSVV1551PPf6zqz43c7cOBAlEql6N+/f7XGzLXO7viv/uqvYuDAgfGHf/iHXTFmrnVmx//0T/8UEyZMiFmzZsXpp58eI0aMiG984xvR2traVWPnSmd2/OEPfziam5vbXi7Zvn17PPLII3HVVVd1yczd3Yn2+a7q726aQmfern3Hjh1HPH/Hjh1VmzPPOrPjd/vqV78agwcPPuwBzjs6s+PHH3887rvvvti0aVMXTJh/ndnx9u3b41/+5V/ic5/7XDzyyCOxbdu2+PKXvxylUinmzZvXFWPnSmd2PGPGjNi1a1dcdtllkWVZHDx4MP74j/84/uIv/qIrRu72jvb5rqWlJf73f/83TjrppC6dJxfPWHDiW7hwYaxatSrWrFkTvXv3rvU43cLevXvj2muvjXvvvTcGDBhQ63G6rXK5HAMHDoy///u/j7Fjx8Y111wTc+fOjW9/+9u1Hq3b2LBhQ3zjG9+Iv/u7v4tnn302HnjggXj44Yfj1ltvrfVoVEEunrHozNu1v//976/o/Pe6zuz4kEWLFsXChQvjn//5n2PUqFHVHDPXKt3xSy+9FL/85S9j2rRpbcfK5XJERPTs2TO2bt0aw4YNq+7QOdOZx/GgQYOioaEh6uvr245dcMEFsWPHjnj77bejV69eVZ05bzqz469//etx7bXXxvXXXx8RESNHjoz9+/fHl770pZg7d2706OHfuL+Lo32+a2xs7PJnKyJy8oxFZ96ufcKECe3Oj4hYv369t3c/is7sOCLizjvvjFtvvTUeffTRGDduXFeMmluV7vj888+PzZs3x6ZNm9p+feITn4jJkyfHpk2boqmpqSvHz4XOPI4vvfTS2LZtW1u0RUS8+OKLMWjQIFFxBJ3Z8YEDBw6Lh0Mhl3m7qt/ZCff5riZfMtoJq1atygqFQrZ8+fLshRdeyL70pS9lp556arZjx44sy7Ls2muvzW655Za285944omsZ8+e2aJFi7ItW7Zk8+bN8+2mx1HpjhcuXJj16tUr+9GPfpT9+te/bvu1d+/eWv0RTniV7vjdfFfI8VW641dffTXr169fdsMNN2Rbt27NHnrooWzgwIHZbbfdVqs/wgmv0h3Pmzcv69evX3b//fdn27dvz9atW5cNGzYs++xnP1urP8IJbe/evdlzzz2XPffcc1lEZHfddVf23HPPZa+88kqWZVl2yy23ZNdee23b+Ye+3fTP/uzPsi1btmRLly717aYddffdd2dDhgzJevXqlY0fPz576qmn2n7vIx/5SDZz5sx25//whz/Mhg8fnvXq1Su78MILs4cffriLJ86fSnZ81llnZRFx2K958+Z1/eA5Uunj+P8SFh1T6Y7/7d/+LbvkkkuyQqGQDR06NLv99tuzgwcPdvHU+VLJjkulUjZ//vxs2LBhWe/evbOmpqbsy1/+cvY///M/XT94Dvzrv/7rEf9uPbTTmTNnZh/5yEcO+5iLLroo69WrVzZ06NBs2bJlXT73Id42HQBIJhdfYwEA5IOwAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAHQDGzdujGnTpsXgwYOjrq4uHnzwwYqvkWVZLFq0KIYPHx6FQiHOOOOMuP322yu6Ri7e3RQAOLb9+/fH6NGj4wtf+EJ86lOf6tQ1brrppli3bl0sWrQoRo4cGbt3747du3dXdA0/0hsAupm6urpYs2ZNTJ8+ve1YsViMuXPnxv333x9vvfVWjBgxIr75zW/GpEmTIiJiy5YtMWrUqHj++efjvPPO6/R9eykEAN4DbrjhhnjyySdj1apV8bOf/Sw+85nPxMc//vH4xS9+ERERP/7xj2Po0KHx0EMPxTnnnBNnn312XH/99RU/YyEsAKCbe/XVV2PZsmWxevXqmDhxYgwbNixuvvnmuOyyy2LZsmUREbF9+/Z45ZVXYvXq1bFixYpYvnx5NDc3x6c//emK7svXWABAN7d58+ZobW2N4cOHtzteLBbjtNNOi4iIcrkcxWIxVqxY0XbefffdF2PHjo2tW7d2+OURYQEA3dy+ffuivr4+mpubo76+vt3v9e3bNyIiBg0aFD179mwXHxdccEFEvPOMh7AAACIiYsyYMdHa2ho7d+6MiRMnHvGcSy+9NA4ePBgvvfRSDBs2LCIiXnzxxYiIOOusszp8X74rBAC6gX379sW2bdsi4p2QuOuuu2Ly5MnRv3//GDJkSHz+85+PJ554IhYvXhxjxoyJN998Mx577LEYNWpUXH311VEul+Piiy+Ovn37xpIlS6JcLsesWbOisbEx1q1b1+E5hAUAdAMbNmyIyZMnH3Z85syZsXz58iiVSnHbbbfFihUr4le/+lUMGDAgPvShD8WCBQti5MiRERHx+uuvx4033hjr1q2Lk08+OaZOnRqLFy+O/v37d3gOYQEAJOPbTQGAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJL5fxYLTPSa/d4WAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ratings.rename(columns={'members': 'average_members'}, inplace=True)\n",
    "ratings['average_members'].hist()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   user_id  average_members user_type\n",
      "0        1    198889.784314       mid\n",
      "1        1    198889.784314       mid\n",
      "2        1    198889.784314       mid\n",
      "3        1    198889.784314       mid\n",
      "4        1    198889.784314       mid\n",
      "5        1    198889.784314       mid\n",
      "6        1    198889.784314       mid\n",
      "7        1    198889.784314       mid\n",
      "8        1    198889.784314       mid\n",
      "9        1    198889.784314       mid\n",
      "user_type\n",
      "mid        0.500020\n",
      "hipster    0.250013\n",
      "trendy     0.249967\n",
      "Name: proportion, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Calculate the 25th and 75th percentiles of average_members\n",
    "percentile_25 = ratings['average_members'].quantile(0.25)\n",
    "percentile_75 = ratings['average_members'].quantile(0.75)\n",
    "\n",
    "# Create a new column 'user_type' based on the percentiles\n",
    "ratings['user_type'] = pd.cut(ratings['average_members'], \n",
    "                              bins=[-float('inf'), percentile_25, percentile_75, float('inf')],\n",
    "                              labels=['hipster', 'mid', 'trendy'])\n",
    "\n",
    "# Display the first few rows to verify the new column\n",
    "print(ratings[['user_id', 'average_members', 'user_type']].head(10))\n",
    "\n",
    "# Display the distribution of user types\n",
    "print(ratings['user_type'].value_counts(normalize=True))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "genre  user_id    Action  Adventure      Cars    Comedy  Dementia    Demons  \\\n",
      "0            1  0.093284   0.018657  0.000000  0.120647  0.000000  0.017413   \n",
      "1            2  0.000000   0.000000  0.000000  0.222222  0.000000  0.000000   \n",
      "2            3  0.139434   0.104575  0.000000  0.084967  0.000000  0.008715   \n",
      "3            4  0.129707   0.054393  0.000000  0.112971  0.000000  0.008368   \n",
      "4            5  0.076288   0.045190  0.001944  0.158406  0.000486  0.009718   \n",
      "\n",
      "genre     Drama     Ecchi   Fantasy  ...  Slice of Life     Space    Sports  \\\n",
      "0      0.036070  0.080846  0.057214  ...       0.008706  0.000000  0.001244   \n",
      "1      0.000000  0.000000  0.000000  ...       0.222222  0.000000  0.111111   \n",
      "2      0.067538  0.010893  0.113290  ...       0.010893  0.000000  0.021786   \n",
      "3      0.020921  0.025105  0.104603  ...       0.025105  0.000000  0.008368   \n",
      "4      0.037901  0.048591  0.046647  ...       0.035957  0.001944  0.019436   \n",
      "\n",
      "genre  Super Power  Supernatural  Thriller  Unknown   Vampire  Yaoi  Yuri  \n",
      "0         0.018657      0.073383  0.000000      0.0  0.007463   0.0   0.0  \n",
      "1         0.000000      0.000000  0.000000      0.0  0.000000   0.0   0.0  \n",
      "2         0.021786      0.056645  0.008715      0.0  0.002179   0.0   0.0  \n",
      "3         0.029289      0.062762  0.008368      0.0  0.020921   0.0   0.0  \n",
      "4         0.020408      0.048591  0.003401      0.0  0.003887   0.0   0.0  \n",
      "\n",
      "[5 rows x 45 columns]\n",
      "\n",
      "Shape of the genre counts table: (73515, 45)\n"
     ]
    }
   ],
   "source": [
    "# Merge ratings with anime to get genre information\n",
    "user_genre_data = ratings.merge(anime[['anime_id', 'genre']], on='anime_id', how='left')\n",
    "\n",
    "# Split the genre string into a list\n",
    "user_genre_data['genre'] = user_genre_data['genre'].fillna('Unknown').str.split(',')\n",
    "\n",
    "# Explode the genre list so each genre gets its own row\n",
    "user_genre_data = user_genre_data.explode('genre')\n",
    "\n",
    "# Remove leading/trailing whitespace from genre names\n",
    "user_genre_data['genre'] = user_genre_data['genre'].str.strip()\n",
    "\n",
    "# Create a pivot table to count genre occurrences for each user\n",
    "genre_counts = pd.pivot_table(user_genre_data, values='rating', index='user_id', \n",
    "                              columns='genre', aggfunc='count', fill_value=0)\n",
    "\n",
    "# Normalize the counts for each user\n",
    "genre_counts_normalized = genre_counts.div(genre_counts.sum(axis=1), axis=0)\n",
    "\n",
    "# Reset the index to make user_id a column\n",
    "genre_counts_normalized = genre_counts_normalized.reset_index()\n",
    "\n",
    "# Display the first few rows of the normalized genre counts\n",
    "print(genre_counts_normalized.head())\n",
    "\n",
    "# Display the shape of the resulting dataframe\n",
    "print(f\"\\nShape of the genre counts table: {genre_counts_normalized.shape}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>genre</th>\n",
       "      <th>user_id</th>\n",
       "      <th>Action</th>\n",
       "      <th>Adventure</th>\n",
       "      <th>Cars</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>Dementia</th>\n",
       "      <th>Demons</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Ecchi</th>\n",
       "      <th>Fantasy</th>\n",
       "      <th>...</th>\n",
       "      <th>Slice of Life</th>\n",
       "      <th>Space</th>\n",
       "      <th>Sports</th>\n",
       "      <th>Super Power</th>\n",
       "      <th>Supernatural</th>\n",
       "      <th>Thriller</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Vampire</th>\n",
       "      <th>Yaoi</th>\n",
       "      <th>Yuri</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>73510</th>\n",
       "      <td>73512</td>\n",
       "      <td>0.071429</td>\n",
       "      <td>0.028571</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.085714</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.128571</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057143</td>\n",
       "      <td>...</td>\n",
       "      <td>0.042857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73511</th>\n",
       "      <td>73513</td>\n",
       "      <td>0.078947</td>\n",
       "      <td>0.059211</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.118421</td>\n",
       "      <td>0.006579</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.105263</td>\n",
       "      <td>0.032895</td>\n",
       "      <td>0.065789</td>\n",
       "      <td>...</td>\n",
       "      <td>0.026316</td>\n",
       "      <td>0.026316</td>\n",
       "      <td>0.006579</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.039474</td>\n",
       "      <td>0.006579</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.006579</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73512</th>\n",
       "      <td>73514</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73513</th>\n",
       "      <td>73515</td>\n",
       "      <td>0.144896</td>\n",
       "      <td>0.058178</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.065862</td>\n",
       "      <td>0.001098</td>\n",
       "      <td>0.017563</td>\n",
       "      <td>0.049396</td>\n",
       "      <td>0.007684</td>\n",
       "      <td>0.066959</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007684</td>\n",
       "      <td>0.007684</td>\n",
       "      <td>0.003293</td>\n",
       "      <td>0.041712</td>\n",
       "      <td>0.079034</td>\n",
       "      <td>0.018661</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.012075</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.001098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73514</th>\n",
       "      <td>73516</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 45 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "genre  user_id    Action  Adventure  Cars    Comedy  Dementia    Demons  \\\n",
       "73510    73512  0.071429   0.028571   0.0  0.085714  0.014286  0.014286   \n",
       "73511    73513  0.078947   0.059211   0.0  0.118421  0.006579  0.000000   \n",
       "73512    73514  0.000000   0.166667   0.0  0.166667  0.000000  0.000000   \n",
       "73513    73515  0.144896   0.058178   0.0  0.065862  0.001098  0.017563   \n",
       "73514    73516  0.142857   0.000000   0.0  0.000000  0.000000  0.000000   \n",
       "\n",
       "genre     Drama     Ecchi   Fantasy  ...  Slice of Life     Space    Sports  \\\n",
       "73510  0.128571  0.000000  0.057143  ...       0.042857  0.000000  0.000000   \n",
       "73511  0.105263  0.032895  0.065789  ...       0.026316  0.026316  0.006579   \n",
       "73512  0.166667  0.000000  0.166667  ...       0.000000  0.000000  0.000000   \n",
       "73513  0.049396  0.007684  0.066959  ...       0.007684  0.007684  0.003293   \n",
       "73514  0.000000  0.142857  0.000000  ...       0.000000  0.000000  0.000000   \n",
       "\n",
       "genre  Super Power  Supernatural  Thriller  Unknown   Vampire  Yaoi      Yuri  \n",
       "73510     0.000000      0.100000  0.000000      0.0  0.000000   0.0  0.000000  \n",
       "73511     0.000000      0.039474  0.006579      0.0  0.006579   0.0  0.000000  \n",
       "73512     0.000000      0.000000  0.000000      0.0  0.000000   0.0  0.000000  \n",
       "73513     0.041712      0.079034  0.018661      0.0  0.012075   0.0  0.001098  \n",
       "73514     0.000000      0.142857  0.000000      0.0  0.000000   0.0  0.000000  \n",
       "\n",
       "[5 rows x 45 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genre_counts_normalized.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of duplicate user IDs: 0\n",
      "There are no duplicate user IDs in genre_counts_normalized.\n",
      "\n",
      "The index is missing 1 numbers.\n",
      "First few missing numbers: [46809]\n",
      "\n",
      "Total number of rows: 73515\n",
      "Number of unique user IDs: 73515\n",
      "Minimum user ID: 1\n",
      "Maximum user ID: 73516\n"
     ]
    }
   ],
   "source": [
    "# Check for duplicate user IDs in genre_counts_normalized\n",
    "duplicate_users = genre_counts_normalized['user_id'].duplicated().sum()\n",
    "\n",
    "print(f\"Number of duplicate user IDs: {duplicate_users}\")\n",
    "\n",
    "if duplicate_users == 0:\n",
    "    print(\"There are no duplicate user IDs in genre_counts_normalized.\")\n",
    "else:\n",
    "    print(\"There are duplicate user IDs in genre_counts_normalized.\")\n",
    "    print(\"\\nFirst few rows with duplicate user IDs:\")\n",
    "    print(genre_counts_normalized[genre_counts_normalized['user_id'].duplicated(keep=False)].sort_values('user_id').head())\n",
    "\n",
    "# Check if the index is missing any numbers\n",
    "min_user_id = genre_counts_normalized['user_id'].min()\n",
    "max_user_id = genre_counts_normalized['user_id'].max()\n",
    "expected_range = set(range(min_user_id, max_user_id + 1))\n",
    "actual_user_ids = set(genre_counts_normalized['user_id'])\n",
    "\n",
    "missing_ids = expected_range - actual_user_ids\n",
    "\n",
    "if len(missing_ids) > 0:\n",
    "    print(f\"\\nThe index is missing {len(missing_ids)} numbers.\")\n",
    "    print(f\"First few missing numbers: {sorted(list(missing_ids))[:5]}\")\n",
    "else:\n",
    "    print(\"\\nThe index is not missing any numbers.\")\n",
    "\n",
    "print(f\"\\nTotal number of rows: {len(genre_counts_normalized)}\")\n",
    "print(f\"Number of unique user IDs: {genre_counts_normalized['user_id'].nunique()}\")\n",
    "print(f\"Minimum user ID: {min_user_id}\")\n",
    "print(f\"Maximum user ID: {max_user_id}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>genre</th>\n",
       "      <th>user_id</th>\n",
       "      <th>Action</th>\n",
       "      <th>Adventure</th>\n",
       "      <th>Cars</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>Dementia</th>\n",
       "      <th>Demons</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Ecchi</th>\n",
       "      <th>Fantasy</th>\n",
       "      <th>...</th>\n",
       "      <th>Slice of Life</th>\n",
       "      <th>Space</th>\n",
       "      <th>Sports</th>\n",
       "      <th>Super Power</th>\n",
       "      <th>Supernatural</th>\n",
       "      <th>Thriller</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Vampire</th>\n",
       "      <th>Yaoi</th>\n",
       "      <th>Yuri</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "      <td>73515.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>36758.363286</td>\n",
       "      <td>0.095044</td>\n",
       "      <td>0.051017</td>\n",
       "      <td>0.000461</td>\n",
       "      <td>0.106547</td>\n",
       "      <td>0.002376</td>\n",
       "      <td>0.010501</td>\n",
       "      <td>0.064284</td>\n",
       "      <td>0.021307</td>\n",
       "      <td>0.058439</td>\n",
       "      <td>...</td>\n",
       "      <td>0.029680</td>\n",
       "      <td>0.004209</td>\n",
       "      <td>0.006433</td>\n",
       "      <td>0.024489</td>\n",
       "      <td>0.059601</td>\n",
       "      <td>0.013054</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>0.007039</td>\n",
       "      <td>0.001133</td>\n",
       "      <td>0.000164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>21222.497501</td>\n",
       "      <td>0.041535</td>\n",
       "      <td>0.036348</td>\n",
       "      <td>0.003015</td>\n",
       "      <td>0.042583</td>\n",
       "      <td>0.007325</td>\n",
       "      <td>0.014598</td>\n",
       "      <td>0.034697</td>\n",
       "      <td>0.024441</td>\n",
       "      <td>0.032864</td>\n",
       "      <td>...</td>\n",
       "      <td>0.030720</td>\n",
       "      <td>0.010330</td>\n",
       "      <td>0.014500</td>\n",
       "      <td>0.025111</td>\n",
       "      <td>0.033316</td>\n",
       "      <td>0.020818</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>0.012306</td>\n",
       "      <td>0.008733</td>\n",
       "      <td>0.002060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>18379.500000</td>\n",
       "      <td>0.071429</td>\n",
       "      <td>0.028409</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.084175</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.044496</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.041176</td>\n",
       "      <td>...</td>\n",
       "      <td>0.008772</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.011385</td>\n",
       "      <td>0.042424</td>\n",
       "      <td>0.002096</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>36758.000000</td>\n",
       "      <td>0.093878</td>\n",
       "      <td>0.045597</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.107353</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.007812</td>\n",
       "      <td>0.061538</td>\n",
       "      <td>0.014493</td>\n",
       "      <td>0.056090</td>\n",
       "      <td>...</td>\n",
       "      <td>0.024490</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.001594</td>\n",
       "      <td>0.020571</td>\n",
       "      <td>0.057522</td>\n",
       "      <td>0.008584</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.004673</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>55137.500000</td>\n",
       "      <td>0.117720</td>\n",
       "      <td>0.067721</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.129199</td>\n",
       "      <td>0.002354</td>\n",
       "      <td>0.013889</td>\n",
       "      <td>0.081081</td>\n",
       "      <td>0.032164</td>\n",
       "      <td>0.072807</td>\n",
       "      <td>...</td>\n",
       "      <td>0.041667</td>\n",
       "      <td>0.004808</td>\n",
       "      <td>0.007812</td>\n",
       "      <td>0.030973</td>\n",
       "      <td>0.073118</td>\n",
       "      <td>0.016575</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.009174</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>73516.000000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.022472</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 45 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "genre       user_id        Action     Adventure          Cars        Comedy  \\\n",
       "count  73515.000000  73515.000000  73515.000000  73515.000000  73515.000000   \n",
       "mean   36758.363286      0.095044      0.051017      0.000461      0.106547   \n",
       "std    21222.497501      0.041535      0.036348      0.003015      0.042583   \n",
       "min        1.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "25%    18379.500000      0.071429      0.028409      0.000000      0.084175   \n",
       "50%    36758.000000      0.093878      0.045597      0.000000      0.107353   \n",
       "75%    55137.500000      0.117720      0.067721      0.000000      0.129199   \n",
       "max    73516.000000      0.500000      0.500000      0.200000      1.000000   \n",
       "\n",
       "genre      Dementia        Demons         Drama         Ecchi       Fantasy  \\\n",
       "count  73515.000000  73515.000000  73515.000000  73515.000000  73515.000000   \n",
       "mean       0.002376      0.010501      0.064284      0.021307      0.058439   \n",
       "std        0.007325      0.014598      0.034697      0.024441      0.032864   \n",
       "min        0.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "25%        0.000000      0.000000      0.044496      0.000000      0.041176   \n",
       "50%        0.000000      0.007812      0.061538      0.014493      0.056090   \n",
       "75%        0.002354      0.013889      0.081081      0.032164      0.072807   \n",
       "max        0.250000      0.333333      0.500000      0.333333      0.500000   \n",
       "\n",
       "genre  ...  Slice of Life         Space        Sports   Super Power  \\\n",
       "count  ...   73515.000000  73515.000000  73515.000000  73515.000000   \n",
       "mean   ...       0.029680      0.004209      0.006433      0.024489   \n",
       "std    ...       0.030720      0.010330      0.014500      0.025111   \n",
       "min    ...       0.000000      0.000000      0.000000      0.000000   \n",
       "25%    ...       0.008772      0.000000      0.000000      0.011385   \n",
       "50%    ...       0.024490      0.000000      0.001594      0.020571   \n",
       "75%    ...       0.041667      0.004808      0.007812      0.030973   \n",
       "max    ...       0.500000      0.500000      0.333333      0.333333   \n",
       "\n",
       "genre  Supernatural      Thriller       Unknown       Vampire          Yaoi  \\\n",
       "count  73515.000000  73515.000000  73515.000000  73515.000000  73515.000000   \n",
       "mean       0.059601      0.013054      0.000001      0.007039      0.001133   \n",
       "std        0.033316      0.020818      0.000094      0.012306      0.008733   \n",
       "min        0.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "25%        0.042424      0.002096      0.000000      0.000000      0.000000   \n",
       "50%        0.057522      0.008584      0.000000      0.004673      0.000000   \n",
       "75%        0.073118      0.016575      0.000000      0.009174      0.000000   \n",
       "max        0.500000      0.500000      0.022472      0.333333      0.500000   \n",
       "\n",
       "genre          Yuri  \n",
       "count  73515.000000  \n",
       "mean       0.000164  \n",
       "std        0.002060  \n",
       "min        0.000000  \n",
       "25%        0.000000  \n",
       "50%        0.000000  \n",
       "75%        0.000000  \n",
       "max        0.333333  \n",
       "\n",
       "[8 rows x 45 columns]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# genre_counts_normalized = genre_counts_normalized.reset_index(drop=True)\n",
    "genre_counts_normalized.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "genre  user_id     Action  Adventure  Cars     Comedy  Dementia     Demons  \\\n",
      "0            1  10.000000  10.000000   0.0  10.000000  0.000000  10.000000   \n",
      "1            2   0.000000   0.000000   0.0  10.000000  0.000000   0.000000   \n",
      "2            3   7.290323   7.531915   0.0   7.657895  0.000000   6.500000   \n",
      "3            5   4.701299   5.186813   6.0   4.155763  1.000000   4.157895   \n",
      "4            7   7.484127   7.700000   0.0   7.297980  9.333333   7.250000   \n",
      "\n",
      "genre     Drama      Ecchi    Fantasy  ...  Slice of Life  Space     Sports  \\\n",
      "0      0.000000  10.000000  10.000000  ...       0.000000   0.00   0.000000   \n",
      "1      0.000000   0.000000   0.000000  ...       0.000000   0.00  10.000000   \n",
      "2      7.451613   7.800000   7.576923  ...       7.800000   0.00   8.600000   \n",
      "3      5.217949   2.510000   4.410526  ...       4.506849   7.25   6.000000   \n",
      "4      7.750000   6.909091   7.337209  ...       7.137255   0.00   7.333333   \n",
      "\n",
      "genre  Super Power  Supernatural  Thriller  Unknown   Vampire  Yaoi  Yuri  \n",
      "0         0.000000     10.000000  0.000000      0.0  0.000000   0.0   0.0  \n",
      "1         0.000000      0.000000  0.000000      0.0  0.000000   0.0   0.0  \n",
      "2         7.777778      7.583333  8.000000      0.0  6.000000   0.0   0.0  \n",
      "3         4.170732      4.346939  7.285714      0.0  4.428571   0.0   0.0  \n",
      "4         7.939394      7.631579  8.363636      0.0  7.444444   0.0   0.0  \n",
      "\n",
      "[5 rows x 45 columns]\n"
     ]
    }
   ],
   "source": [
    "# Merge ratings with anime to get genre information\n",
    "user_genre_data = ratings.merge(anime[['anime_id', 'genre']], on='anime_id', how='left')\n",
    "\n",
    "# Split the genre string into a list\n",
    "user_genre_data['genre'] = user_genre_data['genre'].fillna('Unknown').str.split(',')\n",
    "\n",
    "# Remove rows with negative ratings\n",
    "positive_ratings = user_genre_data[user_genre_data['rating'] > 0]\n",
    "\n",
    "# Explode the genre list so each genre gets its own row\n",
    "positive_ratings = positive_ratings.explode('genre')\n",
    "\n",
    "# Remove leading/trailing whitespace from genre names\n",
    "positive_ratings['genre'] = positive_ratings['genre'].str.strip()\n",
    "\n",
    "# Create a pivot table to count genre occurrences for each user\n",
    "genre_ratings = pd.pivot_table(positive_ratings, values='rating', index='user_id', \n",
    "                              columns='genre', aggfunc='mean', fill_value=0)\n",
    "\n",
    "# Reset the index to make user_id a column\n",
    "genre_ratings = genre_ratings.reset_index()\n",
    "\n",
    "# Display the first few rows of the normalized genre counts\n",
    "print(genre_ratings.head())\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>7.565217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>4.355120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7</td>\n",
       "      <td>7.387755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69595</th>\n",
       "      <td>73512</td>\n",
       "      <td>8.583333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69596</th>\n",
       "      <td>73513</td>\n",
       "      <td>7.515152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69597</th>\n",
       "      <td>73514</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69598</th>\n",
       "      <td>73515</td>\n",
       "      <td>8.547486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69599</th>\n",
       "      <td>73516</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>69600 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       user_id     rating\n",
       "0            1  10.000000\n",
       "1            2  10.000000\n",
       "2            3   7.565217\n",
       "3            5   4.355120\n",
       "4            7   7.387755\n",
       "...        ...        ...\n",
       "69595    73512   8.583333\n",
       "69596    73513   7.515152\n",
       "69597    73514  10.000000\n",
       "69598    73515   8.547486\n",
       "69599    73516   9.000000\n",
       "\n",
       "[69600 rows x 2 columns]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings[ratings['rating'] > 0][['user_id', 'rating']].groupby('user_id').mean().reset_index()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>episodes</th>\n",
       "      <th>episodes_t</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>15796</td>\n",
       "      <td>-1</td>\n",
       "      <td>Unknown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>861</td>\n",
       "      <td>1</td>\n",
       "      <td>Movie</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>54887</td>\n",
       "      <td>35</td>\n",
       "      <td>Short</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6266</td>\n",
       "      <td>34</td>\n",
       "      <td>Short</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>37195</td>\n",
       "      <td>36</td>\n",
       "      <td>Some_Seasons</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  episodes    episodes_t\n",
       "0    15796        -1       Unknown\n",
       "1      861         1         Movie\n",
       "2    54887        35         Short\n",
       "3     6266        34         Short\n",
       "4    37195        36  Some_Seasons"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a test DataFrame with random user_ids and random episode numbers\n",
    "import numpy as np\n",
    "\n",
    "# Set random seed for reproducibility\n",
    "np.random.seed(42)\n",
    "\n",
    "# Generate random user_ids (assuming user_ids range from 1 to 73516 based on previous context)\n",
    "n_samples = 10\n",
    "user_ids = np.random.randint(1, 73517, n_samples)\n",
    "\n",
    "# Generate random episode numbers (assuming episodes range from 1 to 1818 based on previous context)\n",
    "# episodes = np.random.randint(1, 1819, n_samples)\n",
    "episodes = [-1, 1, 35, 34, 36, 500, 499, 501, 10000000, 100000000]\n",
    "\n",
    "# Create the test DataFrame\n",
    "test_df = pd.DataFrame({\n",
    "    'user_id': user_ids,\n",
    "    'episodes': episodes\n",
    "})\n",
    "\n",
    "# Display the first few rows and basic information about the test DataFrame\n",
    "# print(test_df.head())\n",
    "# print(\"\\nDataFrame Info:\")\n",
    "# print(test_df.info())\n",
    "# print(\"\\nDataFrame Description:\")\n",
    "# print(test_df.describe())\n",
    "test_df['episodes_t'] = pd.cut(test_df['episodes'], bins=[-np.inf, 0, 1, 35, 500, np.inf], labels=['Unknown', 'Movie', 'Short', 'Some_Seasons', 'Huge'])\n",
    "test_df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import json\n",
    "\n",
    "# Load the anime and rating CSV files\n",
    "anime_df = pd.read_csv('data/anime.csv')\n",
    "rating_df = pd.read_csv('data/rating.csv')\n",
    "\n",
    "# Convert each DataFrame to a JSON object\n",
    "anime_json = anime_df.to_json(orient='records')\n",
    "rating_json = rating_df.to_json(orient='records')\n",
    "\n",
    "# Parse the JSON objects into Python dictionaries\n",
    "anime_dict = json.loads(anime_json)\n",
    "rating_dict = json.loads(rating_json)\n",
    "\n",
    "# Unify both dictionaries in a bigger dictionary\n",
    "unified_dict = {\n",
    "    'anime': anime_dict,\n",
    "    'rating': rating_dict\n",
    "}\n",
    "\n",
    "# Convert the unified dictionary to a JSON file\n",
    "with open('unified_data.json', 'w') as f:\n",
    "    json.dump(unified_dict, f, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}