classif.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Phishing Email Generation and Classification\n",
    "\n",
    "Experimental notebook for classification and generation of phishing emails using several techniques.\n",
    "\n",
    "Roadmap:\n",
    "- Preprocessing\n",
    "    - Sentiment Analysis\n",
    "    - Extract features based on persuasive features of speech / common phishing tropes\n",
    "    - Steal some feature ideas from webpage classification and apply these\n",
    "- Classification\n",
    "    - Logistic Regression on emails\n",
    "    - word2vec/GLoVE word representation -> logistic regression\n",
    "    - Transformer-based model (BERT)\n",
    "    - Transfer learning\n",
    "- Generation\n",
    "    - With GANs\n",
    "    - With T5\n",
    "    - With transformers\n",
    "- Adversarial Learning\n",
    "    - EA-based method of classifying\n",
    "    - Cyclical GANs\n",
    "    - Use EA to seed an NN?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install requirements - uncomment this line the first time you run this notebook\n",
    "# !pip install -r ./requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#utilities\n",
    "from functools import reduce\n",
    "from tqdm import tqdm\n",
    "import itertools\n",
    "from collections import Counter\n",
    "import pickle\n",
    "import os\n",
    "import re\n",
    "import gc\n",
    "\n",
    "# scikit\n",
    "from sklearn import feature_extraction, model_selection, naive_bayes, pipeline, manifold, preprocessing, metrics\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# word embeddings\n",
    "import gensim\n",
    "import gensim.downloader as gensim_api\n",
    "from gensim.models.word2vec import Word2Vec as w2v\n",
    "\n",
    "#graphing\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "#nltk\n",
    "import nltk\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk.corpus import stopwords as NLTK_STOP\n",
    "import string\n",
    "\n",
    "#spacy imports\n",
    "import spacy\n",
    "from spacy.tokenizer import Tokenizer\n",
    "from spacy.lang.en import English\n",
    "from spacy.lang.en.stop_words import STOP_WORDS as SPACY_STOP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "myseed = 10897"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Email-Spam-Dataset\n",
    "\n",
    "Use this dataset as preliminary exploration of classification techniques. Classifier can then be applied to Enron dataset if it works\n",
    "\n",
    "0: not spam\n",
    "1: spam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "spam_col_names = ['id', 'Body', 'Label']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfSA = pd.read_csv('./kaggle-datasets/Email-Spam-Dataset/completeSpamAssassin.csv', names=spam_col_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfEnron = pd.read_csv('./kaggle-datasets/Email-Spam-Dataset/enronSpamSubset.csv', names=spam_col_names).tail(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfLing = pd.read_csv('./kaggle-datasets/Email-Spam-Dataset/lingSpam.csv', names=spam_col_names)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a peek at the body of the emails"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Body</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Body</td>\n",
       "      <td>Label</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>Subject: great part-time or summer job !\\n \\n ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>Subject: auto insurance rates too high ?\\n \\n ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.0</td>\n",
       "      <td>Subject: do want the best and economical hunti...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.0</td>\n",
       "      <td>Subject: email 57 million people for $ 99\\n \\n...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id                                               Body  Label\n",
       "0  NaN                                               Body  Label\n",
       "1  0.0  Subject: great part-time or summer job !\\n \\n ...      1\n",
       "2  1.0  Subject: auto insurance rates too high ?\\n \\n ...      1\n",
       "3  2.0  Subject: do want the best and economical hunti...      1\n",
       "4  3.0  Subject: email 57 million people for $ 99\\n \\n...      1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfLing.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfEdu = pd.read_csv('./educational-institute-dataset/PhishingEmailData.csv', encoding=\"ISO-8859-1\", usecols=['Email_Subject', 'Email_Content', 'Closing_Remarks'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfEdu['Body'] = dfEdu[dfEdu.columns[1:]].apply(\n",
    "    lambda x: '\\n'.join(x.dropna().astype(str)),\n",
    "    axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfEdu = dfEdu.drop(columns=['Email_Subject', 'Email_Content', 'Closing_Remarks'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process Enron Dataset\n",
    "\n",
    "Extract the message bodies from the Enron dataset\n",
    "\n",
    "The Enron dataset is very large. To save RAM, the first time you process it you can save it to a csv, then restart the jupyter and load it from CSV (so temporary rows are not stored in RAM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_enron():\n",
    "    dfEnronFull = pd.read_csv('./kaggle-datasets/The-Enron-Email-Dataset/emails.csv', usecols=['message'])\n",
    "\n",
    "    #TODO: for now we are overwriting the message column for memory-saving purposes,\n",
    "    # but in future we may wnat to extract other features from the message column such as email metadata\n",
    "    #the issue is that no other datasets contain this data\n",
    "    #TODO: even overwriting, this still seems to use the same amount of ram\n",
    "    #TODO: when reading from csv in load_processed_enron it gets an index, but not when do initial read - here need to ensure has index\n",
    "    dfEnronFull['message'] = dfEnronFull['message'].str.split(\"\\n\\n\", n=1).str[1]\n",
    "    dfEnronFull[\"id\"] = dfEnronFull.index + 1\n",
    "    dfEnronFull[\"label\"] = 0\n",
    "\n",
    "    dfEnronFull.to_csv('./Processed-Datasets/Enron-Bodies/emails.csv', columns=['message'])\n",
    "\n",
    "    return dfEnronFull"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_processed_enron():\n",
    "    dfEnronFull = pd.read_csv('./Processed-Datasets/Enron-Bodies/emails.csv', usecols=['message'])\n",
    "    dfEnronFull[\"id\"] = dfEnronFull.index + 1\n",
    "    dfEnronFull[\"Label\"] = 0\n",
    "\n",
    "    return dfEnronFull"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#uncomment if you haven't run enron yet\n",
    "# dfEnronFull = process_enron()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#comment if you're running enron for first time\n",
    "dfEnronFull = load_processed_enron()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Body</th>\n",
       "      <th>id</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Here is our forecast\\n\\n</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Traveling to have a business meeting takes the...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>test successful.  way to go!!!</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Randy,\\n\\n Can you send me a schedule of the s...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Let's shoot for Tuesday at 11:45.</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517396</th>\n",
       "      <td>This is a trade with OIL-SPEC-HEDGE-NG (John L...</td>\n",
       "      <td>517397</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517397</th>\n",
       "      <td>Some of my position is with the Alberta Term b...</td>\n",
       "      <td>517398</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517398</th>\n",
       "      <td>2\\n\\n -----Original Message-----\\nFrom: \\tDouc...</td>\n",
       "      <td>517399</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517399</th>\n",
       "      <td>Analyst\\t\\t\\t\\t\\tRank\\n\\nStephane Brodeur\\t\\t\\...</td>\n",
       "      <td>517400</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517400</th>\n",
       "      <td>i think the YMCA has a class that is for peopl...</td>\n",
       "      <td>517401</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>517401 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     Body      id  Label\n",
       "0                               Here is our forecast\\n\\n        1      0\n",
       "1       Traveling to have a business meeting takes the...       2      0\n",
       "2                          test successful.  way to go!!!       3      0\n",
       "3       Randy,\\n\\n Can you send me a schedule of the s...       4      0\n",
       "4                     Let's shoot for Tuesday at 11:45.         5      0\n",
       "...                                                   ...     ...    ...\n",
       "517396  This is a trade with OIL-SPEC-HEDGE-NG (John L...  517397      0\n",
       "517397  Some of my position is with the Alberta Term b...  517398      0\n",
       "517398  2\\n\\n -----Original Message-----\\nFrom: \\tDouc...  517399      0\n",
       "517399  Analyst\\t\\t\\t\\t\\tRank\\n\\nStephane Brodeur\\t\\t\\...  517400      0\n",
       "517400  i think the YMCA has a class that is for peopl...  517401      0\n",
       "\n",
       "[517401 rows x 3 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfEnronFull.rename(columns={\"message\":\"Body\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfEnronFull = dfEnronFull.rename(columns={\"message\":\"Body\"})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing\n",
    "\n",
    "In avoidance of 'self-plagiarisation', much of the code for the `Dataset` class is adapted from my Text Processing Sentiment Analysis assignment. Code can be provided to markers on request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#download wordnet for lemmatization\n",
    "#uncomment appropriate line if you get error: \"Resource wordnet not found.\", \"Resource punkt not found.\", etc...\n",
    "\n",
    "# nltk.download('wordnet')\n",
    "# nltk.download('punkt')\n",
    "# nltk.download('stopwords')\n",
    "# nltk.download('averaged_perceptron_tagger')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#download spacy dataset\n",
    "#uncomment the line below if you get error \"Can't find model 'en_core_web_sm'\"\n",
    "# !python -m spacy download en_core_web_sm\n",
    "# !python3 -m spacy download en_core_web_sm\n",
    "\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Dataset` Class - this takes a `Dataframe` and applies various preprocessing/feature selection techniques to it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Dataset:\n",
    "    def __init__(self, df, preprocessing=[\"lower\"], feature_selection=[\"alltokens\"], retList=False, skip_data_loading=False):\n",
    "        if skip_data_loading:\n",
    "            pass\n",
    "        \n",
    "        #read preprocessing and feature_selection configuration\n",
    "        self.data = df\n",
    "        self.preprocessing = preprocessing\n",
    "        self.feature_selection = feature_selection\n",
    "        self.retList = retList\n",
    "\n",
    "        # === define various processors and regexes for various preprocessing/feature selection methods ===\n",
    "\n",
    "        # NLTK Stemming Engine\n",
    "        self.porter = PorterStemmer()\n",
    "\n",
    "        #NLTK Lemmatizing Engine\n",
    "        self.wn_lt = WordNetLemmatizer()\n",
    "\n",
    "        #tokenizers\n",
    "\n",
    "        #words regex - splits on word boundaries, doesn't include punctuation etc\n",
    "        self.word_tokenizer = RegexpTokenizer(r'\\w+')\n",
    "\n",
    "        #create tokenizer based on NLTK-provided regex from Labs\n",
    "        nltk_pat = r'''(?x) # set flag to allow verbose regexps\n",
    "            (?:[A-Z]\\.)+ # abbreviations, e.g. U.S.A.\n",
    "            | \\w+(?:-\\w+)* # words with optional internal hyphens\n",
    "            | \\$?\\d+(?:\\.\\d+)?%? # currency and percentages, e.g. $12.40, 82%\n",
    "            | \\.\\.\\. # ellipsis\n",
    "            | [][.,;\"'?():_`-]\n",
    "            | [>]?[:;][\\']?[\\(\\)\\[\\]]+ # these are separate tokens; includes ], [\n",
    "            '''\n",
    "        self.nltk_tokenizer = RegexpTokenizer(nltk_pat)\n",
    "\n",
    "        #create tokenizer based on custom regex based on above, with less features\n",
    "        custom_pat = r'''(?x)\n",
    "                \\w+(?:-\\w+)*\n",
    "                |\\$?\\d+(?:\\.\\d+)?%?\n",
    "                |\\.\\.\\.'''\n",
    "        self.custom_tokenizer = RegexpTokenizer(custom_pat)\n",
    "\n",
    "        # stop words - SPACY_STOP defined above in imports\n",
    "\n",
    "        self.NLTK_ENGLISH_STOP = set(NLTK_STOP.words('english'))\n",
    "    \n",
    "    def preprocess_phrase(self, phrase):\n",
    "        \"\"\"define preprocessing function for phrases\n",
    "        can call any number of these options - however, some may not combine well\"\"\"\n",
    "\n",
    "        phrase = str(phrase)\n",
    "\n",
    "        if self.preprocessing == []:\n",
    "            #no preprocessing\n",
    "            return phrase\n",
    "        if \"multi_space\" in self.preprocessing:\n",
    "            phrase = re.sub(' +', ' ', phrase)\n",
    "        if \"lower\" in self.preprocessing:\n",
    "            #lowercase\n",
    "            phrase = phrase.lower()\n",
    "        if \"newlines\" in self.preprocessing:\n",
    "            # phrase = \" \".join(filter(None, phrase.split(\"\\n\")))\n",
    "            phrase = phrase.replace(\"\\n\", \" \")\n",
    "        if \"returns\" in self.preprocessing:\n",
    "            # phrase = \" \".join(filter(None, phrase.split(\"\\r\")))\n",
    "            phrase = phrase.replace(\"\\r\", \" \")\n",
    "        if \"tabs\" in self.preprocessing:\n",
    "            # phrase = \" \".join(filter(None, phrase.split(\"\\t\")))\n",
    "            phrase = phrase.replace(\"\\t\", \" \")\n",
    "        if \"punc\" in self.preprocessing:\n",
    "            #remove punctuation\n",
    "            phrase = phrase.translate(str.maketrans('','',string.punctuation))\n",
    "        if \"stemming\" in self.preprocessing:\n",
    "            #use NLTK stemming\n",
    "            phrase = self.porter.stem(phrase)\n",
    "        if \"nltk_lemmatize\" in self.preprocessing:\n",
    "            #use NLTK's lemmatization method\n",
    "            new_phrase = \"\"\n",
    "            words = nltk.word_tokenize(phrase)\n",
    "            for word in words:\n",
    "                new_phrase += self.wn_lt.lemmatize(word)\n",
    "            phrase = new_phrase\n",
    "        if \"spacy_lemmatize\" in self.preprocessing:\n",
    "            #use spacy's lemmatization method\n",
    "            nlp_phrase = nlp(phrase)\n",
    "            new_phrase = \"\"\n",
    "            for token in nlp_phrase:\n",
    "                new_phrase += (token.lemma_ + \" \")\n",
    "            phrase = new_phrase\n",
    "\n",
    "        return phrase\n",
    "\n",
    "    def extract_features_from_phrase(self, phrase):\n",
    "        \"\"\"define feature extraction function for phrases\n",
    "        extracts all words from all phrases as features for document set\n",
    "        for each method, if one has already been applied the phrase must be treated as a list\"\"\"\n",
    "\n",
    "        #list of negation words from \n",
    "        negation_words = [\"neither\", \"never\", \"no\", \"nobody\", \"none\", \"noone\", \"nor\", \"not\", \"nothing\", \"nowhere\"]\n",
    "\n",
    "        intensifer_words = [\"absolutely\", \"completely\", \"extremely\", \"highly\", \"rather\", \"really\", \"so\", \"too\", \"totally\", \"utterly\", \"very\"]\n",
    "\n",
    "        #TODO: add words for authority, bank accounts, urgency, click here, ...\n",
    "\n",
    "        #use if len(self.feature_selection) > 1 to check if any preprocessing already occurred, as input will be in a list\n",
    "\n",
    "        if self.feature_selection == []:\n",
    "            #we have to do some feature selection, absolute minimum is alltokens\n",
    "            return phrase.split(\" \")\n",
    "        if \"alltokens\" in self.feature_selection:\n",
    "            #split based on whitespace\n",
    "            phrase = phrase.split(\" \")\n",
    "        if \"nltk_tokenize\" in self.feature_selection:\n",
    "            #tokenize using NLTK tokenizer, with words regex\n",
    "            if isinstance(phrase, list):\n",
    "                phrase =  list(itertools.chain.from_iterable([self.word_tokenizer.tokenize(partial_phrase) for partial_phrase in phrase]))\n",
    "            else:\n",
    "                phrase = self.word_tokenizer.tokenize(phrase)\n",
    "                # print(phrase)\n",
    "        if \"nltk_tokenize_2\" in self.feature_selection:\n",
    "            #tokenize using NLTK's regex\n",
    "            if isinstance(phrase, list):\n",
    "                phrase =  list(itertools.chain.from_iterable([self.nltk_tokenizer.tokenize(partial_phrase) for partial_phrase in phrase]))\n",
    "            else:\n",
    "                phrase = self.nltk_tokenizer.tokenize(phrase)\n",
    "        if \"custom_tokenize\" in self.feature_selection:\n",
    "            #tokenize with custom regex\n",
    "            if isinstance(phrase, list):\n",
    "                phrase = list(itertools.chain.from_iterable([self.custom_tokenizer.tokenize(partial_phrase) for partial_phrase in phrase]))\n",
    "            else:\n",
    "                phrase = self.custom_tokenizer.tokenize(phrase)\n",
    "        if \"nltk_stoplist\" in self.feature_selection:\n",
    "            #use an NLTK stoplist\n",
    "            if isinstance(phrase, list):#check words if already split into list\n",
    "                #TODO: quicker way to run this?\n",
    "                phrases = []\n",
    "                for partial_phrase in phrase:\n",
    "                    if partial_phrase not in self.NLTK_ENGLISH_STOP:\n",
    "                        phrases.append(partial_phrase)\n",
    "                phrase = phrases\n",
    "            else:#split into list then eval words\n",
    "                phrase = [word for word in phrase.split(\" \") if word not in self.NLTK_ENGLISH_STOP]\n",
    "        if \"spacy_stoplist\" in self.feature_selection:\n",
    "            #use a spacy stoplist\n",
    "            if isinstance(phrase, list):\n",
    "                phrases = []\n",
    "                for partial_phrase in phrase:\n",
    "                    if partial_phrase not in list(SPACY_STOP):\n",
    "                        phrases.append(partial_phrase)\n",
    "                phrase = phrases\n",
    "            else:\n",
    "                phrase = [word for word in phrase.split(\" \") if word not in list(SPACY_STOP)]\n",
    "        if \"custom_stoplist\" in self.feature_selection:\n",
    "            #use a custom stoplist\n",
    "            stoplist = [\"subject\"]\n",
    "            if isinstance(phrase, list):\n",
    "                phrases = []\n",
    "                for partial_phrase in phrase:\n",
    "                    if partial_phrase not in stoplist:\n",
    "                        phrases.append(partial_phrase)\n",
    "                phrase = phrases\n",
    "            else:\n",
    "                phrase = [word for word in phrase.split(\" \") if word not in stoplist]\n",
    "        if \"nltk_pos_tag\" in self.feature_selection:\n",
    "            #use POS tagging - must be a list, so split by spaces if not already\n",
    "            if isinstance(phrase, list):\n",
    "                phrase = nltk.pos_tag(phrase)\n",
    "            else:\n",
    "                phrase = phrase.split(\" \")\n",
    "                phrase = nltk.pos_tag(phrase)\n",
    "        if \"negation_bigrams\" in self.feature_selection:\n",
    "            #create bigrams with negation words + their successors\n",
    "            if len(self.feature_selection) > 1:\n",
    "                phrases = []\n",
    "                for j in range(0, len(phrase)):\n",
    "                    partial_phrase = phrase[j]\n",
    "                    split = partial_phrase.split(\" \")\n",
    "\n",
    "                    for i in range(0, len(split)):\n",
    "                        if split[i] in negation_words and i != len(split) - 1:\n",
    "                            #add the negation word and its successor to the list of phrases\n",
    "                            phrases.append(split[i] + \" \" + split[i+1])\n",
    "                        else:\n",
    "                            #skip to next partial if negation is at end of partial phrase\n",
    "                            if split[i] in negation_words and j != len(phrase) - 1:\n",
    "                                # print(\"Next partial\")\n",
    "                                phrases.append(split[i] + \" \" + phrase[j+1].split(\" \")[0])\n",
    "                            else:\n",
    "                                #skip entirely if this is the last partial phrase\n",
    "                                phrases.append(split[i])\n",
    "                phrase = phrases\n",
    "            else:\n",
    "                split = phrase.split(\" \")\n",
    "                phrase = []\n",
    "                for i in range(0, len(split)):\n",
    "                    if split[i] in negation_words and i != len(split) - 1:\n",
    "                        phrase.append(split[i] + \" \" + split[i+1])\n",
    "                    else:\n",
    "                        phrase.append(split[i])\n",
    "        if \"intensifier_bigrams\" in self.feature_selection:\n",
    "            #create bigrams with intensifier words + their successors\n",
    "            if len(self.feature_selection) > 1:\n",
    "                phrases = []\n",
    "                for j in range(0, len(phrase)):\n",
    "                    partial_phrase = phrase[j]\n",
    "                    split = partial_phrase.split(\" \")\n",
    "\n",
    "                    for i in range(0, len(split)):\n",
    "                        if split[i] in intensifer_words and i != len(split) - 1:\n",
    "                            #add the intensifier word and its successor to the list of phrases\n",
    "                            phrases.append(split[i] + \" \" + split[i+1])\n",
    "                        else:\n",
    "                            #skip to next partial if intensifier is at end of partial phrase\n",
    "                            if split[i] in intensifer_words and  j != len(phrase) - 1:\n",
    "                                phrases.append(split[i] + phrase[j+1].split(\" \")[0])\n",
    "                            else:\n",
    "                                #skip entirely if this is the last partial phrase\n",
    "                                phrases.append(split[i])\n",
    "                phrase = phrases\n",
    "            else:\n",
    "                split = phrase.split(\" \")\n",
    "                phrase = []\n",
    "                for i in range(0, len(split)):\n",
    "                    if split[i] in intensifer_words and i != len(split) - 1:\n",
    "                        phrase.append(split[i] + \" \" + split[i+1])\n",
    "                    else:\n",
    "                        phrase.append(split[i])\n",
    "        if \"remove_short_tokens\" in self.feature_selection:\n",
    "            #remove tokens with < 3 characters\n",
    "            if len(self.feature_selection) > 1:\n",
    "                phrases = []\n",
    "                for j in range(0, len(phrase)):\n",
    "                    partial_phrase = phrase[j]\n",
    "                    split = partial_phrase.split(\" \")\n",
    "\n",
    "                    #recombine it, filtering out tokens of less than 3 characters, but keeping bigrams\n",
    "                    recombined = \" \".join([p for p in split if len(p) > 2])\n",
    "                    phrases.append(recombined)\n",
    "                phrase = phrases\n",
    "            else:\n",
    "                split = phrase.split(\" \")\n",
    "                phrase = [partial_phrase for partial_phrase in split if len(partial_phrase) > 2]\n",
    "\n",
    "        if self.retList:\n",
    "            return phrase\n",
    "        \n",
    "        #rejoin into one text body in case want to do different tokenisation/one-hot later on\n",
    "        else:\n",
    "            return \" \".join(filter(None, phrase))\n",
    "        \n",
    "    #subclass must implement this\n",
    "    def process_phrases(self):\n",
    "        raise NotImplementedError\n",
    "        \n",
    "    #subclass must implement this\n",
    "    def process_dataset(self):\n",
    "        raise NotImplementedError"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subclass for emails"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EmailDataset(Dataset):\n",
    "    def __init__(self, df, preprocessing=[], feature_selection=[], retList=False, bodyColumn='Body', skip_process_phrases=False, skip_data_loading=False):\n",
    "        super().__init__(df, preprocessing, feature_selection, retList, skip_data_loading=skip_data_loading)\n",
    "        self.bodyColumn = bodyColumn\n",
    "\n",
    "        # === apply preprocessing and feature selection ===\n",
    "\n",
    "        if not skip_process_phrases:\n",
    "            self.process_phrases()\n",
    "\n",
    "    def process_phrases(self):\n",
    "        \"\"\"extract bodies\"\"\"\n",
    "\n",
    "        self.bodies = self.data[self.bodyColumn]\n",
    "        print(\"Bodies\")\n",
    "        \n",
    "        #apply preprocessing function to all phrases using list comprehension\n",
    "        self.preprocessed_phrases = [self.preprocess_phrase(phrase) for phrase in self.bodies]\n",
    "        print(\"pp\")\n",
    "        self.features = [self.extract_features_from_phrase(phrase) for phrase in self.preprocessed_phrases]\n",
    "        print(\"features\")\n",
    "\n",
    "        data = {'id': self.data['id'], 'ppBody': self.features, 'label': self.data['Label']}\n",
    "        print(\"dict\")\n",
    "        self.pp_df = pd.DataFrame(data=data)\n",
    "        print(\"df\")\n",
    "\n",
    "    def process_dataset(self):\n",
    "        \"\"\"applies post-processing to the dataframe rows\"\"\"\n",
    "\n",
    "        #load the text body\n",
    "        df = self.pp_df\n",
    "\n",
    "        #find the negative and positive classes\n",
    "        positive = df[df['label'] == \"1\"]\n",
    "        negative = df[df['label'] == \"0\"]\n",
    "\n",
    "        l_positive = len(positive)\n",
    "        l_negative = len(negative)\n",
    "\n",
    "        #which class is bigger?\n",
    "        min_len = min(l_positive, l_negative)\n",
    "\n",
    "        #take a random sample of the smaller class to remove class imbalance\n",
    "        if l_positive > l_negative:\n",
    "            positive = positive.sample(min_len, random_state=myseed)\n",
    "        elif l_negative > l_positive:\n",
    "            negative = negative.sample(min_len, random_state=myseed)\n",
    "\n",
    "        #create counters from the text bodies\n",
    "        self.pos_word_counter = positive['ppBody'].str.split().explode().value_counts()\n",
    "        self.neg_word_counter = negative['ppBody'].str.split().explode().value_counts()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subclass for sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SentimentDataset(Dataset):\n",
    "    def __init__(self, df, preprocessing, feature_selection, retList, bodyColumn='Phrase'):\n",
    "        super().__init__(df, preprocessing, feature_selection, retList)\n",
    "\n",
    "        self.bodyColumn = bodyColumn\n",
    "\n",
    "        # === apply preprocessing and feature selection ===\n",
    "\n",
    "        self.process_phrases()\n",
    "\n",
    "    def process_phrases(self):\n",
    "        # return super().process_phrases()\n",
    "        #TODO: implement\n",
    "        pass"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Representation Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "class WordRep:\n",
    "    def __init__(self, corpus):\n",
    "        self.corpus = corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TFIDF(WordRep):\n",
    "    def __init__(self, corpus, max_features=10000, ngram_range=(1,2)):\n",
    "        super().__init__(corpus)\n",
    "\n",
    "        self.vectorizer = feature_extraction.text.TfidfVectorizer(max_features=max_features, ngram_range=ngram_range)\n",
    "        self.vectorizer.fit(self.corpus)\n",
    "\n",
    "        self.representation = self.vectorizer.transform(self.corpus)\n",
    "        self.vocabulary = self.vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Word2Vec(WordRep):\n",
    "    def __init__(self, corpus):\n",
    "        super().__init__(corpus)\n",
    "    \n",
    "    def create(self, name, size=300, window=8, min_count=1, sg=1, iter=5, callbacks=[]):\n",
    "        self.representation = w2v(self.corpus, vector_size=size, window=window, min_count=min_count, sg=sg, epochs=iter, callbacks=callbacks)\n",
    "        self.representation.save(f'./Word-Representations/{name}.model')\n",
    "\n",
    "    def load(self, path):\n",
    "        self.representation = w2v.load(path)\n",
    "\n",
    "    def get_average_doc_embedding(self, document):\n",
    "        embeddings = []\n",
    "        for word in document.split(\" \"):\n",
    "            if word in self.representation.wv:\n",
    "                embeddings.append(self.representation.wv[word])\n",
    "        \n",
    "        if len(embeddings) > 0:\n",
    "            return np.mean(embeddings, axis=0)\n",
    "        else:\n",
    "            return np.zeros(self.representation.vector_size)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifier Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Classifier:\n",
    "    def __init__(self, dataset, word_rep, bodyColumn='ppBody'):\n",
    "        self.dataset = dataset\n",
    "        self.bodyColumn = bodyColumn\n",
    "\n",
    "        self.split_data()\n",
    "\n",
    "        self.word_rep = word_rep\n",
    "    \n",
    "    def split_data(self, label_index=-1):\n",
    "        #get X and y dataframes - assumes label is last column\n",
    "        self.X = self.dataset.pp_df.iloc[:, :label_index]\n",
    "        self.y = self.dataset.pp_df.iloc[:, label_index]\n",
    "\n",
    "        #random split\n",
    "        ss = model_selection.ShuffleSplit(n_splits=1, test_size=0.25, random_state=myseed)\n",
    "        indexes = list(ss.split(self.X, self.y))\n",
    "        train_set  = indexes[0][0]\n",
    "        test_set  = indexes[0][1]\n",
    "\n",
    "        self.X_train = self.X.iloc[train_set, :]\n",
    "        self.y_train = self.y.iloc[train_set]\n",
    "        self.X_test = self.X.iloc[test_set, :]\n",
    "        self.y_test = self.y.iloc[test_set]\n",
    "\n",
    "    def get_numerical_rep(self, data, method='word_embedding'):\n",
    "        if method == 'one_hot':\n",
    "            #TODO add an option for one hot? not necessary\n",
    "            pass\n",
    "        elif method == 'word_embedding':\n",
    "            rep = []\n",
    "\n",
    "            docs = data[self.bodyColumn]\n",
    "            for doc in docs:\n",
    "                rep.append(self.word_rep.get_average_doc_embedding(doc))\n",
    "\n",
    "        return rep\n",
    "\n",
    "    def train(self, method='logistic_regression'):\n",
    "        self.train_method = method\n",
    "\n",
    "        if method == 'logistic_regression':\n",
    "            self.train_embeddings = self.get_numerical_rep(self.X_train)\n",
    "            self.model = LogisticRegression(random_state=0).fit(self.train_embeddings, self.y_train)\n",
    "            \n",
    "        if method == 'naive_bayes':\n",
    "            self.method = naive_bayes.MultinomialNB()\n",
    "\n",
    "            self.model = pipeline.Pipeline([(\"vectorizer\", self.word_rep.vectorizer),\n",
    "                                            (\"classifier\", self.method)])\n",
    "\n",
    "            self.model[\"classifier\"].fit(self.X_train, self.y_train)\n",
    "\n",
    "    def predict(self, data, evaluate=False, labels=None):\n",
    "        if self.train_method == 'logistic_regression':\n",
    "            data = self.get_numerical_rep(data)\n",
    "        else:\n",
    "            data = data[self.bodyColumn]\n",
    "            \n",
    "        predicted = self.model.predict(data)\n",
    "\n",
    "        if evaluate:\n",
    "            if labels is None:\n",
    "                print(\"Must supply labels option\")\n",
    "            else:\n",
    "                accuracy = metrics.accuracy_score(labels, predicted)\n",
    "                print(f\"Model accuracy: {accuracy}\")\n",
    "\n",
    "        return predicted"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Utility Methods for Saving Models etc"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below methods allowed saving and loading the pickled counters, but this is less necessary now that `process_dataset()` is faster - can still be used to save objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_pickle(object, path):\n",
    "    if not os.path.exists('./Pickle'):\n",
    "        os.makedirs('./Pickle')\n",
    "\n",
    "    with open('./Pickle/' + path, 'wb') as output:\n",
    "        pickle.dump(object, output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_pickle(path):\n",
    "    with open('./Pickle/' + path, 'rb') as input:\n",
    "        return pickle.load(input)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the below to save your counters to a file for later use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save_pickle(d.pos_word_counter, 'd-pos.pickle')\n",
    "# save_pickle(d.neg_word_counter, 'd-neg.pickle')\n",
    "# save_pickle(d2.pos_word_counter, 'd2-pos.pickle')\n",
    "# save_pickle(d2.neg_word_counter, 'd2-neg.pickle')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the below to load your saved counters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d1_pos_counts = load_pickle('d-pos.pickle')\n",
    "# d1_neg_counts = load_pickle('d-neg.pickle')\n",
    "# d2_pos_counts = load_pickle('d2-pos.pickle')\n",
    "# d2_neg_counts = load_pickle('d2-neg.pickle')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Instantiate some datasets "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate our dataset `d` with basic feature selection + tokenisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiate a new dataset\n",
    "#run the preprocessing with lowercasing, removing newlines, stemming, and tokenisation using the nltk regex\n",
    "# d = EmailDataset(dfSpam, preprocessing=[\"lower\",\"newlines\",\"stemming\"], feature_selection=[\"nltk_tokenize\"], retList=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d2 = EmailDataset(dfSpam, preprocessing=[\"lower\",\"newlines\",\"stemming\"], feature_selection=[\"nltk_tokenize\", \"nltk_stoplist\"], retList=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d3 = EmailDataset(dfSpam, preprocessing=[\"lower\",\"newlines\",\"stemming\",\"returns\"], feature_selection=[\"nltk_tokenize\", \"nltk_stoplist\", \"remove_short_tokens\"], retList=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d.process_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d2.process_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d3.process_dataset()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identifying Duplicates in Enron Dataset\n",
    "\n",
    "As the Email-Spam-Dataset contains Enron emails marked as spam, we should remove these from the Enron dataset. Then we can relatively safely assume everything else in the Enron dataset is non-spam\n",
    "\n",
    "The Enron dataset is very large. To save RAM, the first time you process it you can save it to a csv, then restart the jupyter and load it from CSV (so temporary rows are not stored in RAM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def strip_spam_from_enron():\n",
    "    dEnronSpam = EmailDataset(dfEnron, preprocessing=[\"lower\",\"newlines\",\"stemming\",\"returns\"], feature_selection=[\"nltk_tokenize\", \"nltk_stoplist\", \"remove_short_tokens\", \"custom_stoplist\"], retList=False)\n",
    "    \n",
    "    dEnron = EmailDataset(dfEnronFull, preprocessing=[\"lower\",\"newlines\",\"stemming\",\"returns\"], feature_selection=[\"nltk_tokenize\", \"nltk_stoplist\", \"remove_short_tokens\", \"custom_stoplist\"], retList=False, bodyColumn='message')\n",
    "\n",
    "    #remove the items that appear in spam list, as best we can by comparing preprocessed email bodies - might not catch all of them\n",
    "    #requires that the same preprocessing/feature selection is applied to dEnronSpam\n",
    "    #making a big assumption here,but dataset is too big to manually inspect the duplicates\n",
    "    dEnron.pp_df = dEnron.pp_df[~dEnron.pp_df.ppBody.isin(dEnronSpam.pp_df.ppBody)]\n",
    "\n",
    "    dEnron.pp_df.to_csv('./Processed-Datasets/Enron-Bodies/enron-ham-only.csv')\n",
    "\n",
    "    return dEnron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "#uncomment this if you're stripping spam from enron for the first time\n",
    "# strip_spam_from_enron()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_enron_ham_only():\n",
    "    return pd.read_csv('./Processed-Datasets/Enron-Bodies/enron-ham-only.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dEnronHam = load_enron_ham_only()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Combine the datasets together\n",
    "\n",
    "Once loaded the ham-only Enron (i.e. duplicates removed), combine them all and apply preprocessing (applying it twice to enron shouldn't matter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfAll = pd.concat([dfSA, dfEnron, dfLing, dfEdu, dfEnronFull])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Body</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Body</td>\n",
       "      <td>Label</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>\\nSave up to 70% on Life Insurance.\\nWhy Spend...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2.0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1) Fight The Risk of Cancer!\\nhttp://www.adcli...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3.0</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1) Fight The Risk of Cancer!\\nhttp://www.adcli...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4.0</th>\n",
       "      <td>3.0</td>\n",
       "      <td>##############################################...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517396.0</th>\n",
       "      <td>517397</td>\n",
       "      <td>This is a trade with OIL-SPEC-HEDGE-NG (John L...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517397.0</th>\n",
       "      <td>517398</td>\n",
       "      <td>Some of my position is with the Alberta Term b...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517398.0</th>\n",
       "      <td>517399</td>\n",
       "      <td>2\\n\\n -----Original Message-----\\nFrom: \\tDouc...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517399.0</th>\n",
       "      <td>517400</td>\n",
       "      <td>Analyst\\t\\t\\t\\t\\tRank\\n\\nStephane Brodeur\\t\\t\\...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517400.0</th>\n",
       "      <td>517401</td>\n",
       "      <td>i think the YMCA has a class that is for peopl...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>536243 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id                                               Body  Label\n",
       "0.0          NaN                                               Body  Label\n",
       "1.0          0.0  \\nSave up to 70% on Life Insurance.\\nWhy Spend...      1\n",
       "2.0          1.0  1) Fight The Risk of Cancer!\\nhttp://www.adcli...      1\n",
       "3.0          2.0  1) Fight The Risk of Cancer!\\nhttp://www.adcli...      1\n",
       "4.0          3.0  ##############################################...      1\n",
       "...          ...                                                ...    ...\n",
       "517396.0  517397  This is a trade with OIL-SPEC-HEDGE-NG (John L...      0\n",
       "517397.0  517398  Some of my position is with the Alberta Term b...      0\n",
       "517398.0  517399  2\\n\\n -----Original Message-----\\nFrom: \\tDouc...      0\n",
       "517399.0  517400  Analyst\\t\\t\\t\\t\\tRank\\n\\nStephane Brodeur\\t\\t\\...      0\n",
       "517400.0  517401  i think the YMCA has a class that is for peopl...      0\n",
       "\n",
       "[536243 rows x 3 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfAll"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bodies\n",
      "pp\n"
     ]
    }
   ],
   "source": [
    "dAllFullCorpus = EmailDataset(dfAll, preprocessing=[\"newlines\", \"returns\", \"tabs\"], feature_selection=[], retList=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "dAll = EmailDataset(dfAll, preprocessing=[\"lower\",\"newlines\",\"stemming\",\"returns\"], feature_selection=[\"nltk_tokenize\", \"nltk_stoplist\", \"remove_short_tokens\"], retList=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "dAll.pp_df.to_csv('./Processed-Datasets/all-preprocessed.csv')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Instantiate Word Representations for each Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models.callbacks import CallbackAny2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EpochLogger(CallbackAny2Vec):\n",
    "    '''Callback to log information about training'''\n",
    "\n",
    "    def __init__(self):\n",
    "        self.epoch = 0\n",
    "\n",
    "    def on_epoch_begin(self, model):\n",
    "        print(\"Epoch #{} start\".format(self.epoch))\n",
    "\n",
    "    def on_epoch_end(self, model):\n",
    "        print(\"Epoch #{} end\".format(self.epoch))\n",
    "        self.epoch += 1"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following to train a model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_word2vec(data, name):\n",
    "    \"\"\"data: the ppBody column of a Dataset's pp_df dataframe\n",
    "    name: where to save the model\"\"\"\n",
    "    epoch_logger = EpochLogger()\n",
    "    \n",
    "    w2v_model = Word2Vec(data)\n",
    "    w2v_model.create(name, callbacks=[epoch_logger])\n",
    "\n",
    "    return w2v_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch #0 start\n",
      "Epoch #0 end\n",
      "Epoch #1 start\n",
      "Epoch #1 end\n",
      "Epoch #2 start\n",
      "Epoch #2 end\n",
      "Epoch #3 start\n",
      "Epoch #3 end\n",
      "Epoch #4 start\n",
      "Epoch #4 end\n"
     ]
    }
   ],
   "source": [
    "w2v1 = create_word2vec(dAll.pp_df['ppBody'], 'fullDataw2v')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load an existing model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# w2v_1 = Word2Vec(d.pp_df['ppBody'])\n",
    "# w2v_1.load('./Word-Representations/w2v_1.model')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier_1 = Classifier(dAll, w2v1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Input contains NaN",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[62], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m classifier_1\u001b[39m.\u001b[39;49mtrain()\n",
      "Cell \u001b[0;32mIn[21], line 44\u001b[0m, in \u001b[0;36mClassifier.train\u001b[0;34m(self, method)\u001b[0m\n\u001b[1;32m     42\u001b[0m \u001b[39mif\u001b[39;00m method \u001b[39m==\u001b[39m \u001b[39m'\u001b[39m\u001b[39mlogistic_regression\u001b[39m\u001b[39m'\u001b[39m:\n\u001b[1;32m     43\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mtrain_embeddings \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mget_numerical_rep(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mX_train)\n\u001b[0;32m---> 44\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mmodel \u001b[39m=\u001b[39m LogisticRegression(random_state\u001b[39m=\u001b[39;49m\u001b[39m0\u001b[39;49m)\u001b[39m.\u001b[39;49mfit(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mtrain_embeddings, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49my_train)\n\u001b[1;32m     46\u001b[0m \u001b[39mif\u001b[39;00m method \u001b[39m==\u001b[39m \u001b[39m'\u001b[39m\u001b[39mnaive_bayes\u001b[39m\u001b[39m'\u001b[39m:\n\u001b[1;32m     47\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mmethod \u001b[39m=\u001b[39m naive_bayes\u001b[39m.\u001b[39mMultinomialNB()\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/sklearn/base.py:1151\u001b[0m, in \u001b[0;36m_fit_context.<locals>.decorator.<locals>.wrapper\u001b[0;34m(estimator, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1144\u001b[0m     estimator\u001b[39m.\u001b[39m_validate_params()\n\u001b[1;32m   1146\u001b[0m \u001b[39mwith\u001b[39;00m config_context(\n\u001b[1;32m   1147\u001b[0m     skip_parameter_validation\u001b[39m=\u001b[39m(\n\u001b[1;32m   1148\u001b[0m         prefer_skip_nested_validation \u001b[39mor\u001b[39;00m global_skip_validation\n\u001b[1;32m   1149\u001b[0m     )\n\u001b[1;32m   1150\u001b[0m ):\n\u001b[0;32m-> 1151\u001b[0m     \u001b[39mreturn\u001b[39;00m fit_method(estimator, \u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:1207\u001b[0m, in \u001b[0;36mLogisticRegression.fit\u001b[0;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[1;32m   1204\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m   1205\u001b[0m     _dtype \u001b[39m=\u001b[39m [np\u001b[39m.\u001b[39mfloat64, np\u001b[39m.\u001b[39mfloat32]\n\u001b[0;32m-> 1207\u001b[0m X, y \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_validate_data(\n\u001b[1;32m   1208\u001b[0m     X,\n\u001b[1;32m   1209\u001b[0m     y,\n\u001b[1;32m   1210\u001b[0m     accept_sparse\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mcsr\u001b[39;49m\u001b[39m\"\u001b[39;49m,\n\u001b[1;32m   1211\u001b[0m     dtype\u001b[39m=\u001b[39;49m_dtype,\n\u001b[1;32m   1212\u001b[0m     order\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mC\u001b[39;49m\u001b[39m\"\u001b[39;49m,\n\u001b[1;32m   1213\u001b[0m     accept_large_sparse\u001b[39m=\u001b[39;49msolver \u001b[39mnot\u001b[39;49;00m \u001b[39min\u001b[39;49;00m [\u001b[39m\"\u001b[39;49m\u001b[39mliblinear\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39msag\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39msaga\u001b[39;49m\u001b[39m\"\u001b[39;49m],\n\u001b[1;32m   1214\u001b[0m )\n\u001b[1;32m   1215\u001b[0m check_classification_targets(y)\n\u001b[1;32m   1216\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mclasses_ \u001b[39m=\u001b[39m np\u001b[39m.\u001b[39munique(y)\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/sklearn/base.py:621\u001b[0m, in \u001b[0;36mBaseEstimator._validate_data\u001b[0;34m(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)\u001b[0m\n\u001b[1;32m    619\u001b[0m         y \u001b[39m=\u001b[39m check_array(y, input_name\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39my\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mcheck_y_params)\n\u001b[1;32m    620\u001b[0m     \u001b[39melse\u001b[39;00m:\n\u001b[0;32m--> 621\u001b[0m         X, y \u001b[39m=\u001b[39m check_X_y(X, y, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mcheck_params)\n\u001b[1;32m    622\u001b[0m     out \u001b[39m=\u001b[39m X, y\n\u001b[1;32m    624\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m no_val_X \u001b[39mand\u001b[39;00m check_params\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mensure_2d\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mTrue\u001b[39;00m):\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py:1163\u001b[0m, in \u001b[0;36mcheck_X_y\u001b[0;34m(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)\u001b[0m\n\u001b[1;32m   1143\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\n\u001b[1;32m   1144\u001b[0m         \u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mestimator_name\u001b[39m}\u001b[39;00m\u001b[39m requires y to be passed, but the target y is None\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m   1145\u001b[0m     )\n\u001b[1;32m   1147\u001b[0m X \u001b[39m=\u001b[39m check_array(\n\u001b[1;32m   1148\u001b[0m     X,\n\u001b[1;32m   1149\u001b[0m     accept_sparse\u001b[39m=\u001b[39maccept_sparse,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m   1160\u001b[0m     input_name\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mX\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[1;32m   1161\u001b[0m )\n\u001b[0;32m-> 1163\u001b[0m y \u001b[39m=\u001b[39m _check_y(y, multi_output\u001b[39m=\u001b[39;49mmulti_output, y_numeric\u001b[39m=\u001b[39;49my_numeric, estimator\u001b[39m=\u001b[39;49mestimator)\n\u001b[1;32m   1165\u001b[0m check_consistent_length(X, y)\n\u001b[1;32m   1167\u001b[0m \u001b[39mreturn\u001b[39;00m X, y\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py:1185\u001b[0m, in \u001b[0;36m_check_y\u001b[0;34m(y, multi_output, y_numeric, estimator)\u001b[0m\n\u001b[1;32m   1183\u001b[0m     estimator_name \u001b[39m=\u001b[39m _check_estimator_name(estimator)\n\u001b[1;32m   1184\u001b[0m     y \u001b[39m=\u001b[39m column_or_1d(y, warn\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m)\n\u001b[0;32m-> 1185\u001b[0m     _assert_all_finite(y, input_name\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39my\u001b[39;49m\u001b[39m\"\u001b[39;49m, estimator_name\u001b[39m=\u001b[39;49mestimator_name)\n\u001b[1;32m   1186\u001b[0m     _ensure_no_complex_data(y)\n\u001b[1;32m   1187\u001b[0m \u001b[39mif\u001b[39;00m y_numeric \u001b[39mand\u001b[39;00m y\u001b[39m.\u001b[39mdtype\u001b[39m.\u001b[39mkind \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mO\u001b[39m\u001b[39m\"\u001b[39m:\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py:109\u001b[0m, in \u001b[0;36m_assert_all_finite\u001b[0;34m(X, allow_nan, msg_dtype, estimator_name, input_name)\u001b[0m\n\u001b[1;32m    107\u001b[0m \u001b[39mif\u001b[39;00m X\u001b[39m.\u001b[39mdtype \u001b[39m==\u001b[39m np\u001b[39m.\u001b[39mdtype(\u001b[39m\"\u001b[39m\u001b[39mobject\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39mand\u001b[39;00m \u001b[39mnot\u001b[39;00m allow_nan:\n\u001b[1;32m    108\u001b[0m     \u001b[39mif\u001b[39;00m _object_dtype_isnan(X)\u001b[39m.\u001b[39many():\n\u001b[0;32m--> 109\u001b[0m         \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\u001b[39m\"\u001b[39m\u001b[39mInput contains NaN\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m    111\u001b[0m \u001b[39m# We need only consider float arrays, hence can early return for all else.\u001b[39;00m\n\u001b[1;32m    112\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m xp\u001b[39m.\u001b[39misdtype(X\u001b[39m.\u001b[39mdtype, (\u001b[39m\"\u001b[39m\u001b[39mreal floating\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m\"\u001b[39m\u001b[39mcomplex floating\u001b[39m\u001b[39m\"\u001b[39m)):\n",
      "\u001b[0;31mValueError\u001b[0m: Input contains NaN"
     ]
    }
   ],
   "source": [
    "classifier_1.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model accuracy: 0.6178425906069054\n"
     ]
    }
   ],
   "source": [
    "predictions = classifier_1.predict(classifier_1.X_test, evaluate=True, labels=classifier_1.y_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Common Words\n",
    "\n",
    "Pure count-based word analysis - better measure of informativeness can be achieved using TFIDF, but this also helps us identify what \"useless\" tokens we may have failed to remove"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_most_common_n(counter_df, n, name):\n",
    "    print(f\"{n} most common tokens in {name}:\\n\")\n",
    "\n",
    "    head = counter_df.head(n)\n",
    "    most_common = list(zip(head.index, head))\n",
    "\n",
    "    if n < 6:\n",
    "        output = \", \".join([f\"\\\"{word}\\\" ({count})\" for (word, count) in most_common])\n",
    "    else:\n",
    "        output = \"\\n\".join([f\"{word} ({count})\" for (word, count) in most_common])\n",
    "\n",
    "    print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 most common tokens in dfSpam (stopwords and short tokens removed) - non-Spam emails:\n",
      "\n",
      "language (15162)\n",
      "university (15135)\n",
      "subject (14113)\n",
      "http (12953)\n",
      "enron (12349)\n",
      "com (11705)\n",
      "one (9584)\n",
      "information (9021)\n",
      "mail (8764)\n",
      "www (8569)\n",
      "would (8195)\n",
      "new (8067)\n",
      "please (8060)\n",
      "linguistics (7967)\n",
      "ect (7466)\n",
      "conference (7023)\n",
      "also (6845)\n",
      "papers (6170)\n",
      "may (6157)\n",
      "english (5787)\n"
     ]
    }
   ],
   "source": [
    "print_most_common_n(d3.neg_word_counter, 20, \"dfSpam (stopwords and short tokens removed) - non-Spam emails\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 most common tokens in dfSpam (stopwords and short tokens removed) - Spam emails:\n",
      "\n",
      "subject (6630)\n",
      "com (6141)\n",
      "email (5354)\n",
      "http (4930)\n",
      "free (4511)\n",
      "000 (4127)\n",
      "mail (3959)\n",
      "please (3949)\n",
      "money (3929)\n",
      "get (3915)\n",
      "company (3898)\n",
      "information (3878)\n",
      "one (3670)\n",
      "business (3606)\n",
      "time (3371)\n",
      "report (3130)\n",
      "click (3111)\n",
      "www (3088)\n",
      "new (3087)\n",
      "order (2751)\n"
     ]
    }
   ],
   "source": [
    "print_most_common_n(d3.pos_word_counter, 20, \"dfSpam (stopwords and short tokens removed) - Spam emails\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='ppBody'>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAHzCAYAAAA+ZSfoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABfdklEQVR4nO3deVxO2eMH8M/TntKqRaQyGsq+DGXNMkLGGMYw1iEMspTdfMk6wzDJFmahmLGOYcZgKrKnLCWMJbvMUI2lQlTq/P7w6v56FDP13Kdcfd6v1329dO/tnHPL8/R5zj33HJUQQoCIiIhIQXTKugFERERExcUAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIqjV9YN0Ja8vDzcuXMHFStWhEqlKuvmEBER0X8ghMCjR4/g4OAAHZ1X97O8tQHmzp07cHR0LOtmEBERUQncvn0bVatWfeXxtzbAVKxYEcCLH4CZmVkZt4aIiIj+i4yMDDg6Okp/x1/lrQ0w+beNzMzMGGCIiIgU5t+Gf3AQLxERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpjl5ZN6AsOE/dXazzby7w0VJLiIiIqCTYA0NERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKo1fWDXhbOU/dXazzby7w0VJLiIiI3j7sgSEiIiLFYYAhIiIixWGAISIiIsUpdoD5+++/0b9/f1hbW8PY2Bh169bFqVOnpONCCAQGBqJy5cowNjZGhw4dcOXKFbUyHjx4gH79+sHMzAwWFhbw9fXF48eP1c45e/YsWrVqBSMjIzg6OmLhwoUlvEQiIiJ62xQrwDx8+BAtWrSAvr4+/vjjD1y4cAFBQUGwtLSUzlm4cCGWLVuG1atX4/jx4zAxMYG3tzeePXsmndOvXz+cP38ee/fuxa5du3D48GEMHz5cOp6RkYGOHTvCyckJcXFxWLRoEWbNmoXvvvtOhksmIiIipSvWU0hff/01HB0dERoaKu1zcXGR/i2EwJIlSzB9+nR8+OGHAID169fDzs4Ov/76K/r06YOLFy8iPDwcJ0+eRJMmTQAAy5cvR5cuXfDNN9/AwcEBGzZsQHZ2NtauXQsDAwPUrl0bCQkJWLx4sVrQISIiovKpWD0wO3fuRJMmTdCrVy/Y2tqiYcOG+P7776XjN27cQHJyMjp06CDtMzc3R7NmzRATEwMAiImJgYWFhRReAKBDhw7Q0dHB8ePHpXNat24NAwMD6Rxvb28kJibi4cOHRbYtKysLGRkZahsRERG9nYoVYK5fv45Vq1bB1dUVERERGDlyJMaOHYt169YBAJKTkwEAdnZ2at9nZ2cnHUtOToatra3acT09PVhZWamdU1QZBet42fz582Fubi5tjo6Oxbk0IiIiUpBiBZi8vDw0atQIX331FRo2bIjhw4dj2LBhWL16tbba959NmzYN6enp0nb79u2ybhIRERFpSbECTOXKleHu7q62z83NDUlJSQAAe3t7AEBKSoraOSkpKdIxe3t7pKamqh1//vw5Hjx4oHZOUWUUrONlhoaGMDMzU9uIiIjo7VSsANOiRQskJiaq7bt8+TKcnJwAvBjQa29vj6ioKOl4RkYGjh8/Dk9PTwCAp6cn0tLSEBcXJ52zf/9+5OXloVmzZtI5hw8fRk5OjnTO3r17UbNmTbUnnoiIiKh8KlaACQgIQGxsLL766itcvXoVGzduxHfffQc/Pz8AgEqlgr+/P+bNm4edO3fi3LlzGDhwIBwcHNC9e3cAL3psOnXqhGHDhuHEiROIjo7G6NGj0adPHzg4OAAA+vbtCwMDA/j6+uL8+fPYsmULli5divHjx8t79URERKRIxXqM+r333sOOHTswbdo0zJkzBy4uLliyZAn69esnnTN58mQ8efIEw4cPR1paGlq2bInw8HAYGRlJ52zYsAGjR49G+/btoaOjg549e2LZsmXScXNzc0RGRsLPzw+NGzdGpUqVEBgYyEeoiYiICACgEkKIsm6ENmRkZMDc3Bzp6emFxsOUxkrRXI2aiIio+F7397sgroVEREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKo1fWDaCSc566u1jn31zgo6WWEBERlS72wBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4hQrwMyaNQsqlUptq1WrlnT82bNn8PPzg7W1NUxNTdGzZ0+kpKSolZGUlAQfHx9UqFABtra2mDRpEp4/f652zsGDB9GoUSMYGhqiRo0aCAsLK/kVEhER0Vun2D0wtWvXxt27d6Xt6NGj0rGAgAD8/vvv+Pnnn3Ho0CHcuXMHPXr0kI7n5ubCx8cH2dnZOHbsGNatW4ewsDAEBgZK59y4cQM+Pj5o27YtEhIS4O/vj6FDhyIiIkLDSyUiIqK3hV6xv0FPD/b29oX2p6enY82aNdi4cSPatWsHAAgNDYWbmxtiY2Ph4eGByMhIXLhwAfv27YOdnR0aNGiAuXPnYsqUKZg1axYMDAywevVquLi4ICgoCADg5uaGo0ePIjg4GN7e3hpeLhEREb0Nit0Dc+XKFTg4OKB69ero168fkpKSAABxcXHIyclBhw4dpHNr1aqFatWqISYmBgAQExODunXrws7OTjrH29sbGRkZOH/+vHROwTLyz8kv41WysrKQkZGhthEREdHbqVgBplmzZggLC0N4eDhWrVqFGzduoFWrVnj06BGSk5NhYGAACwsLte+xs7NDcnIyACA5OVktvOQfzz/2unMyMjLw9OnTV7Zt/vz5MDc3lzZHR8fiXBoREREpSLFuIXXu3Fn6d7169dCsWTM4OTlh69atMDY2lr1xxTFt2jSMHz9e+jojI4MhhoiI6C2l0WPUFhYWePfdd3H16lXY29sjOzsbaWlpauekpKRIY2bs7e0LPZWU//W/nWNmZvbakGRoaAgzMzO1jYiIiN5OGgWYx48f49q1a6hcuTIaN24MfX19REVFSccTExORlJQET09PAICnpyfOnTuH1NRU6Zy9e/fCzMwM7u7u0jkFy8g/J78MIiIiomLdQpo4cSI++OADODk54c6dO5g5cyZ0dXXx6aefwtzcHL6+vhg/fjysrKxgZmaGMWPGwNPTEx4eHgCAjh07wt3dHQMGDMDChQuRnJyM6dOnw8/PD4aGhgCAESNGYMWKFZg8eTKGDBmC/fv3Y+vWrdi9e7f8V0+v5Ty1+D/zmwt8tNASIiIidcUKMH/99Rc+/fRT3L9/HzY2NmjZsiViY2NhY2MDAAgODoaOjg569uyJrKwseHt7Y+XKldL36+rqYteuXRg5ciQ8PT1hYmKCQYMGYc6cOdI5Li4u2L17NwICArB06VJUrVoVP/zwAx+hJiIiIkmxAszmzZtfe9zIyAghISEICQl55TlOTk7Ys2fPa8vx8vLC6dOni9M0IiIiKke4FhIREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpjl5ZN4DKN+epu4v9PTcX+GihJUREpCTsgSEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLF0SvrBhBpm/PU3cU6/+YCHy21hIiI5MIeGCIiIlIcBhgiIiJSHI0CzIIFC6BSqeDv7y/te/bsGfz8/GBtbQ1TU1P07NkTKSkpat+XlJQEHx8fVKhQAba2tpg0aRKeP3+uds7BgwfRqFEjGBoaokaNGggLC9OkqURERPQWKXGAOXnyJL799lvUq1dPbX9AQAB+//13/Pzzzzh06BDu3LmDHj16SMdzc3Ph4+OD7OxsHDt2DOvWrUNYWBgCAwOlc27cuAEfHx+0bdsWCQkJ8Pf3x9ChQxEREVHS5hIREdFbpEQB5vHjx+jXrx++//57WFpaSvvT09OxZs0aLF68GO3atUPjxo0RGhqKY8eOITY2FgAQGRmJCxcu4KeffkKDBg3QuXNnzJ07FyEhIcjOzgYArF69Gi4uLggKCoKbmxtGjx6Njz/+GMHBwTJcMhERESldiQKMn58ffHx80KFDB7X9cXFxyMnJUdtfq1YtVKtWDTExMQCAmJgY1K1bF3Z2dtI53t7eyMjIwPnz56VzXi7b29tbKqMoWVlZyMjIUNuIiIjo7VTsx6g3b96M+Ph4nDx5stCx5ORkGBgYwMLCQm2/nZ0dkpOTpXMKhpf84/nHXndORkYGnj59CmNj40J1z58/H7Nnzy7u5RAREZECFasH5vbt2xg3bhw2bNgAIyMjbbWpRKZNm4b09HRpu337dlk3iYiIiLSkWD0wcXFxSE1NRaNGjaR9ubm5OHz4MFasWIGIiAhkZ2cjLS1NrRcmJSUF9vb2AAB7e3ucOHFCrdz8p5QKnvPyk0spKSkwMzMrsvcFAAwNDWFoaFicyyGSDSfLIyIqXcUKMO3bt8e5c+fU9g0ePBi1atXClClT4OjoCH19fURFRaFnz54AgMTERCQlJcHT0xMA4OnpiS+//BKpqamwtbUFAOzduxdmZmZwd3eXztmzZ49aPXv37pXKICqPGJKIiP5fsQJMxYoVUadOHbV9JiYmsLa2lvb7+vpi/PjxsLKygpmZGcaMGQNPT094eHgAADp27Ah3d3cMGDAACxcuRHJyMqZPnw4/Pz+pB2XEiBFYsWIFJk+ejCFDhmD//v3YunUrdu8u3hs4Ef13xQ1IQPFDUmnUQUTlg+xrIQUHB0NHRwc9e/ZEVlYWvL29sXLlSum4rq4udu3ahZEjR8LT0xMmJiYYNGgQ5syZI53j4uKC3bt3IyAgAEuXLkXVqlXxww8/wNvbW+7mEtFbhj1VROWDxgHm4MGDal8bGRkhJCQEISEhr/weJyenQreIXubl5YXTp09r2jwiItkxJBGVPa6FRERERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIoj+1pIRESkOS5XQPR67IEhIiIixWGAISIiIsVhgCEiIiLF4RgYIqJyqLhjbACOs6E3C3tgiIiISHEYYIiIiEhxGGCIiIhIcRhgiIiISHEYYIiIiEhxGGCIiIhIcRhgiIiISHEYYIiIiEhxGGCIiIhIcRhgiIiISHEYYIiIiEhxuBYSERFpBddbIm1iDwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOF3MkIiLFKu6CkVws8u3BAENERPQaDElvJgYYIiKiMqbtkFTc8ktSR2kr1hiYVatWoV69ejAzM4OZmRk8PT3xxx9/SMefPXsGPz8/WFtbw9TUFD179kRKSopaGUlJSfDx8UGFChVga2uLSZMm4fnz52rnHDx4EI0aNYKhoSFq1KiBsLCwkl8hERERvXWKFWCqVq2KBQsWIC4uDqdOnUK7du3w4Ycf4vz58wCAgIAA/P777/j5559x6NAh3LlzBz169JC+Pzc3Fz4+PsjOzsaxY8ewbt06hIWFITAwUDrnxo0b8PHxQdu2bZGQkAB/f38MHToUERERMl0yERERKV2xbiF98MEHal9/+eWXWLVqFWJjY1G1alWsWbMGGzduRLt27QAAoaGhcHNzQ2xsLDw8PBAZGYkLFy5g3759sLOzQ4MGDTB37lxMmTIFs2bNgoGBAVavXg0XFxcEBQUBANzc3HD06FEEBwfD29tbpssmIiIiJSvxY9S5ubnYvHkznjx5Ak9PT8TFxSEnJwcdOnSQzqlVqxaqVauGmJgYAEBMTAzq1q0LOzs76Rxvb29kZGRIvTgxMTFqZeSfk1/Gq2RlZSEjI0NtIyIiordTsQPMuXPnYGpqCkNDQ4wYMQI7duyAu7s7kpOTYWBgAAsLC7Xz7ezskJycDABITk5WCy/5x/OPve6cjIwMPH369JXtmj9/PszNzaXN0dGxuJdGREREClHsAFOzZk0kJCTg+PHjGDlyJAYNGoQLFy5oo23FMm3aNKSnp0vb7du3y7pJREREpCXFfozawMAANWrUAAA0btwYJ0+exNKlS9G7d29kZ2cjLS1NrRcmJSUF9vb2AAB7e3ucOHFCrbz8p5QKnvPyk0spKSkwMzODsbHxK9tlaGgIQ0PD4l4OERERKZDGSwnk5eUhKysLjRs3hr6+PqKioqRjiYmJSEpKgqenJwDA09MT586dQ2pqqnTO3r17YWZmBnd3d+mcgmXkn5NfBhEREVGxemCmTZuGzp07o1q1anj06BE2btyIgwcPIiIiAubm5vD19cX48eNhZWUFMzMzjBkzBp6envDw8AAAdOzYEe7u7hgwYAAWLlyI5ORkTJ8+HX5+flLvyYgRI7BixQpMnjwZQ4YMwf79+7F161bs3l38SXiIiIjo7VSsAJOamoqBAwfi7t27MDc3R7169RAREYH3338fABAcHAwdHR307NkTWVlZ8Pb2xsqVK6Xv19XVxa5duzBy5Eh4enrCxMQEgwYNwpw5c6RzXFxcsHv3bgQEBGDp0qWoWrUqfvjhBz5CTURERJJiBZg1a9a89riRkRFCQkIQEhLyynOcnJywZ8+e15bj5eWF06dPF6dpREREVI5oPAaGiIiIqLQxwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4hR7KQEiIiKilzlPLf6EszcX+JS4PvbAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiMMAQERGR4jDAEBERkeIwwBAREZHiFCvAzJ8/H++99x4qVqwIW1tbdO/eHYmJiWrnPHv2DH5+frC2toapqSl69uyJlJQUtXOSkpLg4+ODChUqwNbWFpMmTcLz58/Vzjl48CAaNWoEQ0ND1KhRA2FhYSW7QiIiInrrFCvAHDp0CH5+foiNjcXevXuRk5ODjh074smTJ9I5AQEB+P333/Hzzz/j0KFDuHPnDnr06CEdz83NhY+PD7Kzs3Hs2DGsW7cOYWFhCAwMlM65ceMGfHx80LZtWyQkJMDf3x9Dhw5FRESEDJdMRERESqdXnJPDw8PVvg4LC4OtrS3i4uLQunVrpKenY82aNdi4cSPatWsHAAgNDYWbmxtiY2Ph4eGByMhIXLhwAfv27YOdnR0aNGiAuXPnYsqUKZg1axYMDAywevVquLi4ICgoCADg5uaGo0ePIjg4GN7e3jJdOhERESmVRmNg0tPTAQBWVlYAgLi4OOTk5KBDhw7SObVq1UK1atUQExMDAIiJiUHdunVhZ2cnnePt7Y2MjAycP39eOqdgGfnn5JdRlKysLGRkZKhtRERE9HYqcYDJy8uDv78/WrRogTp16gAAkpOTYWBgAAsLC7Vz7ezskJycLJ1TMLzkH88/9rpzMjIy8PTp0yLbM3/+fJibm0ubo6NjSS+NiIiI3nAlDjB+fn74888/sXnzZjnbU2LTpk1Denq6tN2+fbusm0RERERaUqwxMPlGjx6NXbt24fDhw6hataq0397eHtnZ2UhLS1PrhUlJSYG9vb10zokTJ9TKy39KqeA5Lz+5lJKSAjMzMxgbGxfZJkNDQxgaGpbkcoiIiEhhitUDI4TA6NGjsWPHDuzfvx8uLi5qxxs3bgx9fX1ERUVJ+xITE5GUlARPT08AgKenJ86dO4fU1FTpnL1798LMzAzu7u7SOQXLyD8nvwwiIiIq34rVA+Pn54eNGzfit99+Q8WKFaUxK+bm5jA2Noa5uTl8fX0xfvx4WFlZwczMDGPGjIGnpyc8PDwAAB07doS7uzsGDBiAhQsXIjk5GdOnT4efn5/UgzJixAisWLECkydPxpAhQ7B//35s3boVu3fvlvnyiYiISImK1QOzatUqpKenw8vLC5UrV5a2LVu2SOcEBweja9eu6NmzJ1q3bg17e3ts375dOq6rq4tdu3ZBV1cXnp6e6N+/PwYOHIg5c+ZI57i4uGD37t3Yu3cv6tevj6CgIPzwww98hJqIiIgAFLMHRgjxr+cYGRkhJCQEISEhrzzHyckJe/bseW05Xl5eOH36dHGaR0REROUE10IiIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFYYAhIiIixWGAISIiIsVhgCEiIiLFKXaAOXz4MD744AM4ODhApVLh119/VTsuhEBgYCAqV64MY2NjdOjQAVeuXFE758GDB+jXrx/MzMxgYWEBX19fPH78WO2cs2fPolWrVjAyMoKjoyMWLlxY/KsjIiKit1KxA8yTJ09Qv359hISEFHl84cKFWLZsGVavXo3jx4/DxMQE3t7eePbsmXROv379cP78eezduxe7du3C4cOHMXz4cOl4RkYGOnbsCCcnJ8TFxWHRokWYNWsWvvvuuxJcIhEREb1t9Ir7DZ07d0bnzp2LPCaEwJIlSzB9+nR8+OGHAID169fDzs4Ov/76K/r06YOLFy8iPDwcJ0+eRJMmTQAAy5cvR5cuXfDNN9/AwcEBGzZsQHZ2NtauXQsDAwPUrl0bCQkJWLx4sVrQISIiovJJ1jEwN27cQHJyMjp06CDtMzc3R7NmzRATEwMAiImJgYWFhRReAKBDhw7Q0dHB8ePHpXNat24NAwMD6Rxvb28kJibi4cOHRdadlZWFjIwMtY2IiIjeTrIGmOTkZACAnZ2d2n47OzvpWHJyMmxtbdWO6+npwcrKSu2cosooWMfL5s+fD3Nzc2lzdHTU/IKIiIjojfTWPIU0bdo0pKenS9vt27fLuklERESkJbIGGHt7ewBASkqK2v6UlBTpmL29PVJTU9WOP3/+HA8ePFA7p6gyCtbxMkNDQ5iZmaltRERE9HaSNcC4uLjA3t4eUVFR0r6MjAwcP34cnp6eAABPT0+kpaUhLi5OOmf//v3Iy8tDs2bNpHMOHz6MnJwc6Zy9e/eiZs2asLS0lLPJREREpEDFDjCPHz9GQkICEhISALwYuJuQkICkpCSoVCr4+/tj3rx52LlzJ86dO4eBAwfCwcEB3bt3BwC4ubmhU6dOGDZsGE6cOIHo6GiMHj0affr0gYODAwCgb9++MDAwgK+vL86fP48tW7Zg6dKlGD9+vGwXTkRERMpV7MeoT506hbZt20pf54eKQYMGISwsDJMnT8aTJ08wfPhwpKWloWXLlggPD4eRkZH0PRs2bMDo0aPRvn176OjooGfPnli2bJl03NzcHJGRkfDz80Pjxo1RqVIlBAYG8hFqIiIiAlCCAOPl5QUhxCuPq1QqzJkzB3PmzHnlOVZWVti4ceNr66lXrx6OHDlS3OYRERFROfDWPIVERERE5QcDDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpDgMMERERKQ4DDBERESkOAwwREREpzhsdYEJCQuDs7AwjIyM0a9YMJ06cKOsmERER0RvgjQ0wW7Zswfjx4zFz5kzEx8ejfv368Pb2Rmpqalk3jYiIiMrYGxtgFi9ejGHDhmHw4MFwd3fH6tWrUaFCBaxdu7asm0ZERERlTK+sG1CU7OxsxMXFYdq0adI+HR0ddOjQATExMUV+T1ZWFrKysqSv09PTAQAZGRmFzs3LyixWe4oq49+8DXUUt/y3pY438XdRGnW8ib+L0qjjTfxdlEYdb+LvojTqeBN/F6VRx5v4u3hVHfn7hBCv/2bxBvr7778FAHHs2DG1/ZMmTRJNmzYt8ntmzpwpAHDjxo0bN27c3oLt9u3br80Kb2QPTElMmzYN48ePl77Oy8vDgwcPYG1tDZVK9a/fn5GRAUdHR9y+fRtmZmZaaSPreDPKZx1vVh1vwzWwjjenfNbxZtVRkvKFEHj06BEcHBxee94bGWAqVaoEXV1dpKSkqO1PSUmBvb19kd9jaGgIQ0NDtX0WFhbFrtvMzExr/1FYx5tVPut4s+p4G66Bdbw55bOON6uO4pZvbm7+r+e8kYN4DQwM0LhxY0RFRUn78vLyEBUVBU9PzzJsGREREb0J3sgeGAAYP348Bg0ahCZNmqBp06ZYsmQJnjx5gsGDB5d104iIiKiMvbEBpnfv3vjnn38QGBiI5ORkNGjQAOHh4bCzs9NKfYaGhpg5c2ah21Cso/TreBuugXW8OeWzjjerjrfhGljHm1G+Soh/e06JiIiI6M3yRo6BISIiInodBhgiIiJSHAYYIiIiUhwGGCIiIlIcBhgiIiKShRACSUlJePbsmdbrYoAholI3Z84cZGYWXvjt6dOnmDNnjix1zJw5E7du3ZKlrDdFafxRoFfbtGnTK49NmjRJ4/LXrl2LGzduaFxOWRJCoEaNGrh9+7bW6yrXj1Hr6uri7t27sLW1Vdt///592NraIjc3V5Z67ty5g6NHjyI1NRV5eXlqx8aOHVuiMouzSqi2p6CWy5EjR/Dtt9/i2rVr2LZtG6pUqYIff/wRLi4uaNmypcblP3/+HOfPn0dycjIAwN7eHu7u7tDX19e4bCqe0njtNWjQAH/++SfatGkDX19f9OzZU6vzaWhLXl4evvzyS6xevRopKSm4fPkyqlevjhkzZsDZ2Rm+vr4a1/H06VMIIVChQgUAwK1bt7Bjxw64u7ujY8eOGpcPANeuXUNoaCiuXbuGpUuXwtbWFn/88QeqVauG2rVra1z+4cOH0bx5c+jpqU9v9vz5cxw7dgytW7fWuA4LCwts2rQJnTt3VtsfEBCAzZs34+7duxqV7+rqiuvXr6NKlSpo06YN2rRpAy8vL9SoUUOjcktb7dq1sWbNGnh4eGi3Io2XjlYwlUolUlJSCu3/+++/hZGRkSx1hIaGCgMDA2FqaiqcnJyEs7OztLm4uJS4XJVKJXR0dF675Z8jl0uXLgk/Pz/Rrl070a5dO+Hn5ycuXbokS9nbtm0TxsbGYujQocLQ0FBcu3ZNCCHE8uXLRefOnTUqOzc3V/zvf/8TFhYWQqVSqW0WFhZi+vTpIjc3t8TlBwQE/OdNLocPHxb9+vUTHh4e4q+//hJCCLF+/Xpx5MgRWcp3cnISs2fPFrdu3ZKlvJepVCqRmppaaH9UVJSoVKmSbPXEx8eLMWPGiEqVKgkLCwsxYsQIceLECdnKF+LF/6/ExERx5MgRcejQIbVNDrNnzxbVq1cXP/30kzA2NpZeG5s3bxYeHh6y1PH++++LVatWCSGEePjwobCzsxNVq1YVRkZGYuXKlRqXf/DgQWFsbCw6dOggDAwMpGuYP3++6Nmzp8blCyGEjo5Oke/n9+7dk+19cNeuXcLc3FztdTZ69Gjh4OAgLl68KEsdf/31l/jpp5/E8OHDRc2aNYWOjo6oUqWK6NevnyzlCyGEo6OjGDBggPjhhx/E1atXZSs3386dO0XLli3FuXPnZC+7oHIZYJYuXSqWLl0qdHR0xJdffil9vXTpUrF48WLRvXt30aBBA1nqqlq1qpg3b55GfyCLcvDgwf+8yWHbtm1CT09PeHh4SH+MPT09hZ6enti2bZvG5Tdo0ECsW7dOCCGEqamp9AYXHx8v7OzsNCp70qRJwsbGRqxevVrcuHFDZGZmiszMTHHjxg3x7bffCltbWzF58uQSl+/l5fWftrZt22p0Hfm0GfbyBQcHi/r16wtdXV3RoUMHsWnTJvHs2TONy7WwsBCWlpZCR0dH+nf+ZmZmJnR0dMSoUaNkuAJ12dnZ4pdffhFdu3YV+vr6om7dumLJkiUiLS1No3JjYmKEi4uL9IGh4CbXH8133nlH7Nu3Twih/tq4ePGisLCwkKUOa2tr8eeffwohhPj+++9FvXr1RG5urti6dauoVauWxuV7eHiIoKAgIYT6NRw/flxUqVJF4/KFeHUoTkxMFBUrVpSlDiGE2LBhg7C0tBSnTp0SI0eOFA4ODiIxMVG28vM9efJEhIeHi0GDBgk9PT2hq6srW9k//vijGDZsmHB1dRUqlUpUrVpV9OvXT3z33Xfi8uXLGpdvYWEhDAwMhI6OjjAyMlJ7nVtaWspwBS+Uy1tILi4uAF50k1atWhW6urrSMQMDAzg7O2POnDlo1qyZxnVZW1vjxIkTeOeddzQuqyy988476NevX6HxCTNnzsRPP/2Ea9euaVR+hQoVcOHCBTg7O6NixYo4c+YMqlevjuvXr8Pd3V2je//29vZYt24dvL29izweERGBgQMHFlr9/E3VsGFDBAQEYODAgWo/q9OnT6Nz587SLTI5xMfHIywsDJs2bUJubi769u2LIUOGoFGjRiUqb926dRBCYMiQIViyZInairP5rz1tLNianZ2NHTt2YO3atdi/fz+aN2+OO3fuICUlBd9//z169+5donIbNGiAd999F7Nnz0blypWhUqnUjv+XFXX/jbGxMS5dugQnJye13/eFCxfQtGlTPH78WOM6KlSogEuXLqFatWr45JNPULt2bcycORO3b99GzZo1ixyvVBympqY4d+4cXFxc1K7h5s2bqFWrlkav7x49egAAfvvtN3Tq1EntNmFubi7Onj2LmjVrIjw8XKNrKGjlypUYP348bGxscODAAdlu8URGRuLgwYM4ePAgTp8+DTc3N+k2UuvWrWFpaSlLPQXdvXsXhw4dwq5du7Blyxbk5eVpfAt33bp1rz0+aNAgjcrP98auhaRN+YOk2rZti+3bt2vlP0U+X19f/Pzzz5g6daqs5Z49exZ16tSBjo4Ozp49+9pz69Wrp3F9d+/excCBAwvt79+/PxYtWqRx+fb29rh69SqcnZ3V9h89ehTVq1fXqOxHjx7BwcHhlccrV66MJ0+eaFRHaUpMTCzyfr65uTnS0tJkratRo0Zo1KgRgoKCsHLlSkyZMgWrVq1C3bp1MXbsWAwePLjQH+3XyX/jcnFxQYsWLQqNV5BbXFwcQkNDsWnTJhgaGmLgwIEICQmR/uAsX74cY8eOLXGAuXLlCrZt26bVMQru7u44cuQInJyc1PZv27YNDRs2lKWOGjVq4Ndff8VHH32EiIgIBAQEAABSU1NlGUNnYWGBu3fvSh8e850+fRpVqlTRqOz8kCiEQMWKFWFsbCwdMzAwgIeHB4YNG1bi8sePH1/kfhsbGzRq1AgrV66U9i1evLjE9QBAp06dYGNjgwkTJmDPnj2wsLDQqLzXyczMxNGjR3Hw4EEcOHAAp0+fRp06deDl5aVx2XIFlH9TLgNMvgMHDmi9jvnz56Nr164IDw9H3bp1Cw0YLel/+AYNGiA5ORm2trZo0KABVCoViupMU6lUsgyI9PLywpEjRwq9UR89ehStWrXSuPxhw4Zh3LhxWLt2LVQqFe7cuYOYmBhMnDgRM2bM0KhsLy8vTJw4ERs2bEClSpXUjt27dw9TpkzR6EXbo0cPhIWFwczMTPo0+Crbt28vcT35tBn2XpaTk4MdO3YgNDQUe/fuhYeHB3x9ffHXX3/hiy++wL59+7Bx48Zil9umTRutD+qsW7cuLl26hI4dO2LNmjX44IMP1HpbAeDTTz/FuHHjSlxHs2bNcPXqVa0GmMDAQAwaNAh///038vLysH37diQmJmL9+vXYtWuXbHX07dsXAQEBaN++vdQLFhkZKUtI6tOnD6ZMmYKff/4ZKpUKeXl5iI6OxsSJE4v8YFQcoaGh0nvf8uXLYWpqqnF7Czp9+nSR+2vUqIGMjAzpeHGC/KssXrwYhw8fxsKFC7F06VKp98XLywvvvvuuxuXna968udTD4+XlhalTp8rew6Pt1zeA8j2It0ePHmLBggWF9n/99dfi448/lqWOuXPnCpVKJWrVqiXatGkj25iImzdviry8POnfr9vksGrVKmFjYyP8/PzEjz/+KH788Ufh5+cnbG1txapVq8Rvv/0mbSWRl5cn5s2bJ0xMTKQxBEZGRmL69Okatz0pKUnUqVNH6OnpiYYNG4pOnTqJTp06iYYNGwo9PT1Rr149kZSUVOLyP/vsM5GRkSH9+3WbHL766ivh7u4uYmNjRcWKFcWRI0fETz/9JGxsbMSyZctkqSMuLk6MHj1aWFtbCxsbGzFhwoRCgxTPnTtX4sHupTGoc86cOdIAZ23Zvn27cHd3F6GhoeLUqVPizJkzaptcDh8+LDp06CBsbGyEsbGxaNGihYiIiJCtfCGEuHv3roiPj1cbr3f8+HFZBqdmZWWJoUOHCj09PaFSqYS+vr7Q0dER/fv3F8+fP9e4/NzcXKGvry/L+I03xdmzZ8Xy5cvFRx99JPT19WUbKySEEJaWlsLa2lp8+umn4ttvv5V9DE9pvL6FKKeDePNVqlRJnD17ttD+s2fPCltbW1nqsLCwEKGhobKUVZZeHqD4qk3TgYtZWVni/Pnz4vjx4+LRo0cytf7FG9yePXtEYGCgGD58uBg+fLgIDAwUf/zxh+wDrLVNm2Evn46OjvD29hZbt24V2dnZRZ7z+PHjEoey0hjUWRpe9RqQ+wnA0paeni527NghLly4IGu5t27dErt37xZbtmyRPWy4u7uLmJgYWct8WVpamrh//36h/ffv3xfp6emy1JGXlyfi4uJEUFCQ6Nq1q7CwsBC6urqyPViSX8eZM2fE0qVLRY8ePUSlSpWEg4OD+PTTT8V3332ncfml9foul4N48xkbGyMhIQE1a9ZU23/p0iU0bNgQT58+1bgOe3t7HDlyBK6urhqX9W8uXLiApKQkZGdnq+3v1q2b1uuWy9WrV3Ht2jW0bt0axsbGEELI0jX7NsrOzsbVq1fx+PFjuLu7y9p1fuvWrUJjLuSkzUGd+XJzcxEWFoaoqKgi52Dav3+/xnX820R5cv4Ms7Ozi7yOatWqaVz2J598gtatW2P06NF4+vQp6tevj5s3b0IIgc2bN6Nnz54a16Ftv//+OxYuXIhVq1ahTp06Wqmjc+fO+OCDDzBq1Ci1/atXr8bOnTuxZ88ejcr/4IMPEB0djYyMDNSvXx9eXl5o06YNWrdurbXxMEIIxMXFYcWKFdiwYYMsg3hL4/UNlPMxMHXr1sWWLVsQGBiotn/z5s1wd3eXpY5x48Zh+fLlWLZsmSzlFeX69ev46KOPcO7cObWxMPl/+OUYA7N+/Xr07t270ERg2dnZ2Lx5s8b3se/fv49PPvkEBw4cgEqlwpUrV1C9enX4+vrC0tISQUFBGpUPACdOnEBMTIzaRHbNmzfHe++9p3HZBW3btg1bt24tMkzGx8fLVo+BgQHc3d2RkZGBffv2oWbNmnBzc5OlbCcnJ6SlpWHbtm24du0aJk2aBCsrK8THx8POzk7jgZfaHNSZb9y4cQgLC4OPjw/q1KmjlSCszZCX78qVKxgyZAiOHTumtj8/3Mvx+j58+DD+97//AQB27NgBIQTS0tKwbt06zJs3T+MAI4TAtm3bcODAgSJDmBxjwwYOHIjMzEzUr18fBgYGaoN5AeDBgwca13H8+PEixy16eXlJPz9N1KpVC59//jlatWolyxNsrxIfHy897XT06FE8evQIdevWxZgxY9CmTRuNyy+N1zdQzgPMjBkz0KNHD1y7dg3t2rUDAERFRWHTpk34+eefZanjxIkT2L9/P3bt2oXatWsXGsQrxwt33LhxcHFxQVRUFFxcXHDixAncv38fEyZMwDfffKNx+QAwePBgdOrUqdDMqY8ePcLgwYM1DjABAQHQ19dHUlKS2h/h3r17Y/z48RoFmNTUVPTs2RPR0dGoVq0a7OzsAAApKSkICAhAixYt8MsvvxS6tpJYtmwZ/ve//+Gzzz7Db7/9hsGDB+PatWs4efIk/Pz8NC4fKPxp+b333sONGzdk/bR89uxZtG/fHhYWFrh58yaGDRsGKysrbN++HUlJSVi/fr1G5WtzUGe+zZs3Y+vWrejSpYss5b3KtWvXsGTJEly8eBHAi6eGxo0bJ9vUCZ999hn09PSwa9euIh/VlkN6ejqsrKwAAOHh4ejZsycqVKgAHx8fWabI9/f3x7fffou2bdvCzs5OK9ewZMkS2ct8WVZWFp4/f15of05Ojiw99nPnzoWRkZHG5fybpk2bomHDhmjTpg2GDRuG1q1byxqYSuP1DaB8D+IV4sXMis2bNxcVKlQQ1tbWom3btrJN/iZE6QzqtLa2lgYMmpmZSbPjRkVFyXbf9FWTRCUkJMgyMZGdnZ1ISEgQQqjfM7127ZowMTHRqOyePXsKT0/PImcNvnTpkmjevLlsg7Zr1qwpNm7cKIRQv44ZM2YIPz8/Weoo+LPasGGDqFGjhnjy5IlYuXKlbL/vdu3aiUmTJgkh1K8jOjpaODk5aVx+UYM6VSqVbIM6hRCicuXKWplgrKDw8HBhYGAgmjZtKk3w2LRpU2FoaCgiIyNlqaNChQqyzfL6Kq6urmLLli3i8ePHwsbGRkRFRQkhXry+ra2tNS7f0tJS7N69W+NyypqXl5cYPXp0of2jRo0SLVu21Lh8Q0ND0apVKzF9+nSxb98+kZmZqXGZRZFrvM6raHvQdr5yH2DeBhYWFuL69etCCCGqV68u9u/fL4QQ4urVq8LY2Fijshs0aCAaNmwodHR0RN26dUXDhg2lrV69eqJixYqiV69eGl+DqampNKiv4B/MkydPCisrK43Ljo+Pf+XxU6dOCVNTU43qyGdsbCw9+WVjYyMFjcuXL2t8HfmMjIykp6YGDBggpkyZIoR4MUBS07CXz8zMTJpivODv4+bNm8LQ0FCWOoR48YSYtgZ1fvPNN2LUqFHS03ra0KBBA+nnX9CUKVNEw4YNZamjSZMmsi0R8SohISFCT09PWFhYSLPwCiHEsmXLhJeXl8blOzs7az2ECSHE8+fPxbZt28TcuXPF3Llzxfbt22X9g3n06FFhZGQkWrVqJWbNmiVmzZolWrVqJYyMjMThw4c1Lv/IkSPiyy+/FO+//74wMTERhoaGokWLFuKLL76QLRDne/jwofj+++/F1KlTpYHJcXFxsj65p81B20IIUa5vIQGQ7vNfv34dEydOlPU+f0H//PMPEhMTAQA1a9aEjY2NbGXXqVMHZ86cgYuLC5o1a4aFCxfCwMAA3333ncbzgnTv3h0AkJCQAG9vb7WBovkzp8pxy6JVq1ZYv3495s6dCwBSt+PChQvRtm1bjco2NDR87eKXjx49km2RP3t7ezx48ABOTk6oVq0aYmNjUb9+fekWjxwcHR0RExMDKysrhIeHY/PmzQCAhw8fytb9/Kqf2eXLl2X5v1vU5GCxsbFQqVQwMjJCjRo18OGHH0q3NUri6NGjOHDgAP744w+t3b69ePEitm7dWmh//kzDcvj6668xefJkfPXVV0XOJSXHRHOjRo1C06ZNcfv2bbz//vvQ0dEBAFSvXh3z5s3TuPxZs2Zh9uzZWLt2baGxKXK5evUqunTpgr///lt6MGP+/PlwdHTE7t27Zbml16JFC8TExGDRokXYunUrjI2NUa9ePaxZs0aWBzVatmyJli1b4osvvsDz589x8uRJfPvtt1i4cCEWLFgg2wLD2r5FnK9atWqyDDJ/lXL9FNLZs2fRoUMHmJub4+bNm0hMTET16tUxffp02X6JT548wZgxY7B+/Xpp4Jquri4GDhyI5cuXS6u/aiIiIgJPnjxBjx49cPXqVXTt2hWXL1+GtbU1tmzZIo3v0cS6devQp08fra3m++eff6J9+/Zo1KgR9u/fj27duuH8+fN48OABoqOjNXrz8fPzw+7duxEcHIz27dtLb/gZGRmIiorC+PHj0bVrVyxfvlzj6xg6dCgcHR0xc+ZMhISEYNKkSWjRogVOnTqFHj16YM2aNRrXsXLlSowbNw6mpqZwcnJCfHw8dHR0sHz5cmzfvl2WCRqHDh2K+/fvY+vWrbCyssLZs2ehq6uL7t27o3Xr1hr/cW7bti3i4+ORm5sr/bG5fPkydHV1UatWLSQmJkKlUuHo0aMlHlA/ePDg1x4PDQ0tUbkFOTo6YvHixejVq5fa/q1bt2LixIlISkrSuI78MAGoT5YmZBzEmy87Oxs3btzAO++8I+ssyU+fPsVHH32E6OhoODs7Fwphcgxu79KlC4QQ2LBhgxR879+/j/79+0NHRwe7d+/WuI7ScPnyZWmA7cGDB5GVlYXWrVvDy8tLo0kXC+rQoQMaNWqEhQsXqj0ldOzYMfTt2xc3b94sdpmvmrG4KJrOWJyvXAcYbfwSX/b5559j3759WLFiBVq0aAHgxSfDsWPH4v3338eqVas0rqMoDx48gKWlpWyD5apXr46TJ0/C2tpabX9aWhoaNWqE69eva1xHeno6VqxYgTNnzuDx48do1KgR/Pz8ULlyZY3KzcrKgr+/P9auXYvnz5/DwMAAwIs3az09Pfj6+iI4OFiWcJaXl4e8vDzpzX/Lli2Ijo6Gq6srRowYUeiNu6Ti4uKQlJSE999/X+oV2717NywsLKT/Z5pIT0/Hxx9/jFOnTklLMSQnJ8PT0xN79uyBiYmJRuUvWbIER44cQWhoqBQo09PTMXToULRs2RLDhg1D37598fTpU0RERGh8PdoyZ84cBAcHY+rUqWjevDkAIDo6Gl9//TXGjx+v8SzSAHDo0KHXHpfjqZHMzEyMGTNGWsPm8uXLqF69OsaMGYMqVapovBRK/hOGH3/8cZGDeGfOnKlR+QBgYmKC2NhY1K1bV23/mTNn0KJFixKvGZWRkaH2oed1NO0Nq1KlCp4+fSrNvtumTRvUq1dP9kHP5ubmiI+PxzvvvKP2t+/WrVuoWbNmiR5zfrmnPD4+Hs+fPy/0AaVx48ayTGEAoHwP4i2N+/zW1tbiwIEDhfbv379fVKpUSZY6SoNKpSpyqfrk5GRhYGCgUdnZ2dmiXbt2Wp9FMz09Xezfv19s3LhRbNy4Uezfv18rg9mePn0qjh8/Ln7//Xe1GYp37twpe13aduTIERESEiK+/vprsXfvXtnKdXBwEOfPny+0/88//xQODg5CiBf34+UYQKpNeXl5YvHixaJKlSrSRHZVqlQRS5YskXXszcOHD8U333wjfH19ha+vrwgKCtJ4Je2Cxo4dKxo3biyOHDkiTExMpPfCX3/9VZaB4RUqVND6OB5LS0sRHR1daP/Ro0c1etBAR0dHeu/Ln6Dw5U2uiQvr168vDA0Nhaenp5g2bZqIiIgQT5480bjcl9nY2EjjAgv+7YuMjBRVq1bVuPygoCDxwQcfiAcPHkj7Hjx4ID788EPxzTffaFx+vnI9Bkbb9/mBF59s8h/bLcjW1lbjFV7zPXv2DMuXL3/lHAuadM/u3LlT+ndERITao3a5ubmIiooqtCZPcenr6//rgpSaunfvHtauXVvkPDCfffaZbL/v8PBwDBgwAPfv3y90TM7u/r/++gs7d+4scq4Zubpngf+/Jy+39PR0pKamFro99M8//0ivSQsLi0LXVlzanpNHpVIhICAAAQEBePToEQCgYsWKGpdb0KlTp9CpUycYGRmhadOmAIDg4GB89dVXiIyMLPHK4AX9+uuv2LJlCzw8PNQ+7deuXVvjleaBF7fa5Bir8zpdu3bF8OHDsWbNGunndPz4cYwYMUKjyTz3798v3ZLS9vp5CQkJSEtLw+HDh3Ho0CF88cUXuHDhAho0aIC2bdviyy+/lKWebt26Yc6cOdL4LZVKhaSkJEyZMkWWMY1BQUGIjIxUW1vJ0tIS8+bNQ8eOHTFhwgSN6wBQvntgfH19Rffu3UV2drYwNTUV169fF7du3RINGzYU48aNk6WOdu3aiV69eomnT59K+zIzM0WvXr1E+/btZamjb9++olKlSmLEiBFi5syZ0uj4/E0TL0+PXnAzMDAQ7777rvj99981vgZ/f/8in+aQw4kTJ4SlpaWoUqWKGDRokJg8ebKYPHmyGDRokKhataqwsrISJ0+elKWuGjVqiFGjRonk5GRZyivKvn37RIUKFaT1nRo0aCAsLCyEubm5RutrFVXPtGnThK+vrxg8eLDapqm+ffsKFxcXsX37dnH79m1x+/ZtsX37dlG9enXRv39/IYQQmzZtEo0bNy5xHUuXLhWmpqZi9OjRwsDAQHz++eeiQ4cOwtzcXHzxxRcaX0Npadmypfjss89ETk6OtC8nJ0cMGjRItGrVSpY6jI2NpU/hBT+RJyQkCDMzM43L37Vrl/D29hY3btzQuKxXefjwoejWrZv03mRgYCB0dHRE9+7dZe2tKi337t0T27ZtEwMGDBB6enqyLk2RlpYmOnToIC1T4OjoKPT19UWrVq3E48ePNS7f1NT0lXce5HriU4hyvpSAtu/zA8C5c+fQqVMnZGVloX79+gBe3JM1NDREZGSkLKtympubY8+ePbKMfXgVFxcXnDx5stBqznLJH+js6uqKxo0bF/rZa9Kr4OHhgfr162P16tWF7iULITBixAicPXsWMTExJa4jn5mZGU6fPi3bJGZFadq0KTp37ozZs2dL969tbW3Rr18/dOrUCSNHjtS4jtmzZ2POnDlo0qRJkZOn7dixQ6PyHz9+jICAAKxfv16aGExPTw+DBg1CcHAwTExMkJCQAODFyuslUatWLcycOROffvqp2n3+wMBAPHjwACtWrChRuY0aNUJUVBQsLS3RsGHD145PkKOXx9jYGKdPn0atWrXU9l+4cAFNmjSRpSe3devW6NWrF8aMGYOKFSvi7NmzcHFxwZgxY3DlyhWEh4drVL6lpSUyMzPx/PlzVKhQodBYMDlmyc135coVXLx4ESqVCm5ubhqvFF6c3uF69eppVNf27dulwbsXLlyAlZUVWrZsKY2Hyf8bIpfo6Gi1MYcdOnSQpdyBAwfiyJEjCAoKUusNmzRpElq1aiWNtdJUuQ4w+Y4ePYqzZ8/K/kvMl5mZiQ0bNuDSpUsAADc3N/Tr10+2xwnd3d2xefNmjV88/yYqKuqV68qsXbu22OWdPXsWderUgY6OzmsflVapVBoN+nrVH4B8cq59NWTIELRo0QK+vr4al/UqFStWREJCAt555x1YWlri6NGjqF27Ns6cOYMPP/xQlsHnlStXxsKFCzFgwADNG/wajx8/lgaAV69eXdb1nCpUqICLFy/CyckJtra22Lt3L+rXr48rV67Aw8OjyNt8/8Xs2bMxadIkVKhQAbNmzXptgJFjcKqdnR1+/PFHdOzYUW1/REQEBg4ciJSUFI3rOHr0KDp37oz+/fsjLCwMn3/+OS5cuIBjx47h0KFDaNy4sUbl/9sfrEGDBmlU/svES8upaEJHR0dtiZZXkeMWsa2tLdq0aSNtLw9IlpPc7+cFZWZmYuLEiVi7di1ycnIAQHpgYtGiRbJ0DgDlfCmBfNq6zw+8mIfAzs4Ow4YNU9u/du1a/PPPP5gyZYrGdQQFBWHKlClYvXq11tZmmTNnDmbPnv3KT+Ql0bBhQ9y9exe2tra4detWkU85ycHe3h4nTpx4ZYA5ceJEkeOUSmLFihXo1asXjhw5UuScHWPHjtW4DhMTE2k8R+XKlXHt2jWpJ+/evXsalw+8eEIr/6kabTI1NdVa8NbWnDwFQ8msWbNkaOnr9e7dG76+vvjmm2/UnnSaNGkSPv30U1nqaNmyJRISErBgwQLUrVtXGlsTExMjyx9RuQPKq6xZswbBwcG4cuUKAMDV1RX+/v4YOnRoicu8ceOGXM37Vx07dkS7du3Qpk0brfbi/lsPqyZyc3Nx6tQpfPnll1i0aJE0huqdd96RLbhIZLsZpRBLly6VxqMsXbr0tduaNWtEbGysRvU5OTkVOTI+NjZWODs7a1R2vtTUVOHl5SV0dHSEqampsLS0VNvkYG9vL9avXy9LWfmsrKykn++rliqQw4oVK4ShoaEYO3as+O2330RsbKyIjY0Vv/32mxg7dqwwNjYWISEhstT1ww8/CD09PWFqaiqcnJyEs7OztLm4uMhSx4cffigteT9hwgRRo0YNMW/ePNGoUSPZxlVNnjxZzJkzR5ayyoqvr680BmzFihXC2NhYuu8/ZMgQWepwcXER9+7dK7T/4cOHsv2+s7KyxNixY6UxHTo6OsLQ0FD4+/uLZ8+eyVKHNhR8wi89Pf21mxxmzJghTExMxNSpU6Un/6ZOnSpMTU3FjBkzZKnjq6++EmvWrCm0f82aNWLBggUalz906FDh6uoqdHR0RNWqVUW/fv3E999/L/sTmtp4Py/I0NBQmh1em8rdLSQXFxecOnUK1tbWhVbKfFlWVhZSU1MREBCARYsWlag+IyMjXLx4sVBd169fh7u7uyzLinfo0AFJSUnw9fUtco4FOT79WFtb48SJE7J+Khg+fDjWr1+PypUrIykpCVWrVoWurm6R52o6z8yWLVsQHByMuLg4qZs3f06C8ePH45NPPtGo/Hz29vYYO3Yspk6dqjYBmZyuX7+Ox48fo169enjy5AkmTJiAY8eOwdXVFYsXL5alF27cuHFYv3496tWrh3r16hXqSZLzSSdteXlOns2bN0s/p88//1yaD0gTOjo6SE5OLrQQaEpKChwdHTV+iqqgzMxMtU+zckyCWVBeXh6uXr1a5C2F1q1bF7s8XV1dqYc1/zbMy4SMk/HZ2Nhg2bJlhXqlNm3ahDFjxsjSO+ns7IyNGzcW6p08fvw4+vTpI1tvzd9//y09iXTo0CFcvnwZlStXxl9//SVL+dp4Py+oSZMm+Prrr9G+fXutlJ+v3N1CKvgf7L/8Z9u7dy/69u1b4gDj6OiI6OjoQgEmOjoaDg4OJSrzZceOHUNMTIzsA7wKGjp0KDZu3CjLxFz5vvvuO2n24LFjx2LYsGGyP4Kar3fv3ujduzdycnKkN7JKlSrJNrFcvuzsbPTu3Vtr4QWA2vIQJiYmWL16tex1nD17Vho8++eff6od08ZKwtqgo6Oj9nvo06cP+vTpI0vZ/2V6gX/7gFRcFSpU0NqYiNjYWPTt2xe3bt0qdHutpAGj4OPHoaGhcHR0LPQBJS8vT5bZioEXK0I3adKk0P7GjRsXuYJ0SSQnJxc5saaNjQ3u3r0rSx3Ai0HP1tbWsLS0hIWFBfT09GRdfkYb7+cFzZs3DxMnTsTcuXOLfChDrkfqy10PTHE9ffoU3333XYmncF64cCEWLlyIRYsWSVP6R0VFYfLkyZgwYQKmTZumcRsbNWqElStXwsPDQ+OyCio4NXReXh7WrVuntU/kgwcPxrJly7QWYEpLQEAAbGxs8MUXX2i1nvw1vK5du4ZJkyZpbQ0vpXv48CHWrFmDixcvAngx4H3w4MEarbEE/P/0/kUN7tTX14ezszOCgoLQtWtXjeopLQ0aNMC7776L2bNnFzkmomBAK4mCvTEF3b9/H7a2trL0wIwZMwb6+vqF3osmTpyIp0+fIiQkROM6XF1dMXPmTPTv319t/48//oiZM2dq3FP8xRdf4ODBgzh9+jTc3NzQpk0beHl5oXXr1mpzqmhK2z2spbX8RbkPMFFRUQgODpbe4Nzc3ODv7y/bk0hCCEydOhXLli2TupONjIwwZcoUBAYGylJHZGQkZs+ejS+//FLWxd7+6yKKmj4l9DYZO3Ys1q9fj/r162vt1ktprOFVUH63ddWqVWUtV9sOHz6Mbt26wczMTPpkHhcXh7S0NPz+++8lui3yMm1PL1BaTExMcObMGY0fOX4VHR0dpKSkFOpFuHXrFtzd3fHkyRON68ifisHR0VH6MHf8+HEkJSVh4MCBaq/Fkr4Otf2BVEdHBzY2NggICECPHj3w7rvvalTeq2jzqU+gdJa/AMp5gMlfFO/jjz+Gp6cngBddqdu2bUNwcDD8/Pxkq+vx48e4ePEijI2N4erqKuuiiKW52Bu9nrbfGIDSWcMrLy8P8+bNQ1BQkLSGTMWKFTFhwgT873//0+otMrnUrVsXnp6eWLVqlXTrIjc3F6NGjcKxY8dw7ty5Mm7hm6Ndu3aYPHkyOnXqJGu5+b24S5cuxbBhw9TG7eTm5uL48ePQ1dVFdHS0xnWVxgcubX8gPXPmDA4dOoSDBw/iyJEjMDAwkHphvLy8tBZotCEtLa1Q76evr6/GvXkFlesAU7VqVUydOhWjR49W2x8SEoKvvvoKf//9dxm1rHhKK+3Sm0EbC7G9bNq0aVizZg1mz56ttgjprFmzMGzYMNmmNNcmY2NjJCQkSIvJ5UtMTESDBg1kmfcHeLHi/KFDh4pcrkCOx+ZLw44dOzB9+nRMmjSpyF7ckj7qnh8qDh06BE9PT7WB0wYGBnB2dsbEiRPh6upa8saXAW1+IC3ozJkzCA4OxoYNG5CXl6eYD6NFLX9x8uRJPH36VLblLwCUv8eoCzIxMRFXrlwptP/y5cvCxMSkDFpUcocPHxb9+vUTHh4e4q+//hJCCLF+/XqtL6BGpU/bC7EJIUTlypXFb7/9Vmj/r7/+Ki22+KZr3ry52LFjR6H9O3bsEM2aNZOljvj4eGFvby/MzMyErq6usLGxESqVSpiYmMj2GHVpeHmZkILLh8gxhf1nn32mlYVTi3LlyhURHh4uMjMzhRBC1kU1tS0vL0/ExcVJiyFaWloKXV1d0bBhQ+Hv71/WzfvPSmP5CyHK+WKO3bp1w44dOzBp0iS1/b/99ptiBt8BwC+//IIBAwagX79+OH36NLKysgC8WCrhq6++wp49e8q4hSQnbS/EBryY2r2oif9q1aol67Tv2jR27FiMGzcOV69elcZExMbGIiQkBAsWLFCbIr6kPQwBAQH44IMPsHr1apibmyM2Nhb6+vro379/iQf+lwVtT9YWGhqq1fKBFwOCP/nkExw4cAAqlQpXrlxB9erV4evrC0tLSwQFBWm9DZqysrLC48ePUb9+fbRp0wbDhg1Dq1atYGFhUdZNK5ZTp07h+++/l6YwAF7MxDt58uQinxQrMdmikEIUnKhu7ty5wtzcXHTp0kXMnTtXzJ07V/j4+AgLCwsxd+7csm7qf9agQQOxbt06IYT6J/L4+HhhZ2dXlk0jLXjVQmytW7eWZSE2IYRo2rSpGDNmTKH9o0ePlq33QtuK6lWQu4fB3NxcXLp0Sfr3hQsXhBAvJqqsWbOmLNdB/82AAQOEt7e3uH37ttr7YHh4uHB3dy/j1v03u3btKrWeKm2ytbUVERERhfaHh4cLW1tb2eopdz0wwcHBal9bWlriwoULuHDhgrTPwsICa9euxfTp00u7eSWSmJhY5BMV5ubmSEtLK/0GkVaZm5tj7969Wl3Da+HChfDx8cG+ffukAe4xMTFISkrCH3/8IVs92lQaU8Dr6+tLA5ptbW2RlJQENzc3mJub4/bt21qvXxM7d+5E586doa+vrzavTVG6detWSq0qucjISERERBR6Ws7V1RW3bt0qo1YVj4+PT1k3QRalsfwFUM4nsntb2Nvb4+rVq3B2dlbbf/ToUbVJz+jtos01vNq0aYPExESsWrVKeoqgR48eGDVqlGwTMGqbttYFK6hhw4Y4efIkXF1d0aZNGwQGBuLevXv48ccfUadOHa3Xr4nu3btLswh37979lecp5UnGJ0+eFDk78YMHD7Q2yJaK9s0330ClUmHgwIHSJIL6+voYOXIkFixYIFs95foppLfF/Pnz8dNPP2Ht2rV4//33sWfPHty6dQsBAQGYMWMGxowZU9ZNJA0tW7bsP58r15Mvz549w9mzZ4ucWl4Jn8gB4M6dOzh69GiR1yDHz+nUqVN49OgR2rZti9TUVAwcOFBarmDt2rVanR2b1HXp0gWNGzfG3LlzUbFiRZw9exZOTk7o06cP8vLysG3btrJuYrmj7eUvynWAGTJkyGuPa7qkeGkRQuCrr77C/PnzkZmZCQAwNDSUpnIm5fuv09KrVCqNZwMFgPDwcAwcOBD379+XbWr50hYWFiateWRtba02R5JcP6e3WVpamqIGj54/fx7t2rVDo0aNsH//fnTr1g3nz5/HgwcPEB0drdXVnalslOsA89FHH6l9nZOTgz///BNpaWlo164dtm/fXkYtK5ns7GxcvXoVjx8/hru7O0xNTcu6SaRl+S9fudcncnV1RceOHREYGAg7OztZyy4tjo6OGDFiBKZNm6aIiffK0tdffw1nZ2f07t0bANCrVy/88ssvqFy5Mvbs2fPG9yTl5OSgU6dOmD9/Pvbu3YszZ85IY8P8/PyKXL+IlK9cB5ii5OXlYeTIkXjnnXcwefLksm4OUZHWrFmD4OBgXLlyBcCLwOHv74+hQ4fKUr6ZmRlOnz6t6E+t2l5xF3jx6G5gYCAOHDhQ5G0qpTxy7uLigg0bNqB58+bYu3cvPvnkE2zZsgVbt25FUlISIiMjy7qJ/8rGxka6fUflAwNMERITE+Hl5SXr6qJEcgkMDMTixYsxZswYtSeEVqxYgYCAAMyZM0fjOoYMGYIWLVrA19dX47LKyuTJk2FlZYWpU6dqrY4uXbrg6tWr8PX1hZ2dXaGesEGDBmmtbjkZGxvj8uXLcHR0xLhx4/Ds2TN8++23uHz5Mpo1a4aHDx+WdRP/VUBAAAwNDWUdJEpvNgaYIuzZsweDBg3CP//8U9ZNISrExsYGy5YtK/Q44qZNmzBmzBjcu3dP4zoyMzPRq1cv2NjYFDm1vBKmyM/NzUXXrl3x9OnTIq9BjoU1K1asiKNHj77xt1j+jYODA7Zt24bmzZujZs2amDdvHnr16oXExES89957yMjIKOsm/qv8xRxdXV3RuHFjmJiYqB2X4/dNb5Zy9xh1QfkLjeUTQuDu3bvYvXu3Yj45UfmTk5NT5GyWjRs3lh5Z1NSmTZsQGRkJIyMjHDx4sNAAWCUEmPnz5yMiIkJaC+nla5BDrVq1ZFtTqSz16NEDffv2haurK+7fv4/OnTsDAE6fPq21Farl9ueff0pr7Fy+fFntmNxjxOjNUK57YF5evTR/KfN27dphyJAhatMgE70pxowZA319/UKfKCdOnIinT58iJCRE4zrs7e0xduxYTJ06VbEDYC0tLREcHIzPPvtMa3WcPHkSU6dORWBgIOrUqVOol8fMzExrdcspJycHS5cuxe3bt/HZZ5+hYcOGAF5M/FmxYkXZxlYRyalcB5jMzEwIIaSuxps3b+LXX3+Fm5sbvL29y7h1REXL7yp3dHSU1vg5fvw4kpKSMHDgQLU/oiXtNreyssLJkycVPYjX3t4eR44c0eqgzitXrqBv376Ij49X2y+EUMzj5kRKVa4DTMeOHdGjRw+MGDECaWlpqFWrFvT19XHv3j0sXrwYI0eOLOsmEhXycs/hq6hUKuzfv79EdQQEBMDGxgZffPFFib7/TTB//nzcvXu3WJMAFlfTpk2hp6eHcePGFTmIt02bNlqrW07r169/7fGBAweWUkuI/rtyHWAqVaqEQ4cOoXbt2vjhhx+wfPlynD59Gr/88gsCAwOlKdSJypuxY8di/fr1qF+/PurVq6eVAbDa9tFHH2H//v2wtrZG7dq1C12DHPM8VahQAadPn5bG2SiVpaWl2tc5OTnIzMyEgYEBKlSooJjHwal8KdeDPDIzM1GxYkUALxYC69GjB3R0dODh4aGYxb+ItOHcuXPSOIg///xT7ZhSBkRaWFigR48eWq2jSZMmuH37tuIDTFGPSV+5cgUjR47EpEmTyqBFRP+uXPfA1KtXD0OHDsVHH32EOnXqIDw8HJ6enoiLi4OPjw+Sk5PLuolE9Ab7+eefMWvWLEyaNKnIR7Xr1atXRi2Tx6lTp9C/f39cunSprJtCVEi5DjDbtm1D3759kZubi/bt20uzTc6fPx+HDx/GH3/8UcYtJCJN/fPPP0hMTAQA1KxZEzY2NrKVXdQTWiqV6q0ZxJuQkIDWrVsrYh4YKn/KdYABgOTkZNy9exf169eX3oxOnDgBMzMz1KpVq4xbR0Ql9eTJE+mJrfwp/nV1dTFw4EAsX75clpVx/+1Ws5OTk8Z1lIadO3eqfZ0/J9aKFSvg6OjID3P0Rir3AYaI3k6ff/459u3bhxUrVqBFixYAgKNHj2Ls2LF4//33sWrVKo3Kz8nJQa1atbBr1y64ubnJ0eQy83JPkkqlkubECgoK4mKI9EZigCGit1KlSpWwbds2eHl5qe0/cOAAPvnkE1mWCqlSpQr27dun+ABTUH5vlVInMKTyg/9DieitlJmZCTs7u0L7bW1tkZmZKUsdfn5++Prrr2VbwqEsrVmzBnXq1IGxsTGMjY1Rp04d/PDDD2XdLKJXYg8MEb2V2rdvD2tra6xfvx5GRkYAgKdPn2LQoEF48OAB9u3bp3EdH330EaKiomBqaoq6desWWkBQjrlmSkNprHBOJDcGGCJ6K507dw6dOnVCVlaWtFr0mTNnYGhoiMjISNSuXVvjOgYPHvza46GhoRrXURpKY4VzIrkxwBDRWyszMxMbNmyQ5jFxc3NDv379YGxsXMYte7NYWFjg5MmThdaNunz5Mpo2bYq0tLSyaRjRazDAENFbaf78+bCzs8OQIUPU9q9duxb//PMPpkyZIltd2pxrpjSUxgrnRHJjgCGit5KzszM2btyI5s2bq+0/fvw4+vTpgxs3bmhcR2nMNaMt48ePl/79/PlzhIWFoVq1akWucL58+fKyaibRKzHAENFbycjICBcvXoSLi4va/uvXr8Pd3R3Pnj3TuA5tzzWjTaWxqjmRNpXrxRyJ6O3l6OiI6OjoQgEmOjoaDg4OstTxyy+/FJprpkuXLjA2NsYnn3zyRgeYAwcOlHUTiDTCAENEb6Vhw4bB398fOTk5aNeuHQAgKioKkydPxoQJE2SpozTmmiGiovEWEhG9lYQQmDp1KpYtW4bs7GwAL24rTZkyBYGBgbLUURpzzRBR0RhgiOit9vjxY1y8eBHGxsZwdXWFoaGhbGW/aq4ZIyMjREREyDLXDBEVjQGGiEgDnGuGqGwwwBARFUOjRo0QFRUFS0tLzJkzBxMnTnyjH5cmelsxwBARFYOxsTGuXLmCqlWrQldXF3fv3oWtrW1ZN4uo3OFTSERExdCgQQMMHjwYLVu2hBAC33zzDUxNTYs8V67BwkRUGHtgiIiKITExETNnzsS1a9cQHx8Pd3d36OkV/iyoUqkQHx9fBi0kKh8YYIiISkhHRwfJycm8hURUBhhgiIiISHE4BoaISANXrlzBgQMHkJqaKi3omI9jYIi0hz0wREQl9P3332PkyJGoVKkS7O3toVKppGMcA0OkXQwwREQl5OTkhFGjRmHKlCll3RSicocBhoiohMzMzJCQkIDq1auXdVOIyh2dsm4AEZFS9erVC5GRkWXdDKJyiYN4iYhKqEaNGpgxYwZiY2NRt25d6Ovrqx0fO3ZsGbWM6O3HW0hERCXk4uLyymMqlQrXr18vxdYQlS8MMERERKQ4vIVERFQM48ePx9y5c2FiYoLx48e/8jyVSoWgoKBSbBlR+cIAQ0RUDKdPn0ZOTo7071cpOCcMEcmPt5CIiIhIcfgYNRERESkOAwwREREpDgMMERERKQ4DDBGVO87OzliyZElZN4OINMAAQ0RvJGdnZ6hUKqhUKujq6sLBwQG+vr54+PBhWTeNiN4ADDBE9MaaM2cO7t69i6SkJGzYsAGHDx/m9PxEBIABhoi0wMvLC6NHj8bo0aNhbm6OSpUqYcaMGciftcHZ2Rlz587Fp59+ChMTE1SpUgUhISGFyqlYsSLs7e1RpUoVtG3bFoMGDUJ8fLzaOb/88gtq164NQ0NDODs7F5o8LjU1FR988AGMjY3h4uKCDRs2qB0fMmQIunbtqrYvJycHtra2WLNmjRw/DiLSAgYYItKKdevWQU9PDydOnMDSpUuxePFi/PDDD9LxRYsWoX79+jh9+jSmTp2KcePGYe/eva8s7++//8bvv/+OZs2aSfvi4uLwySefoE+fPjh37hxmzZqFGTNmICwsTDrns88+w+3bt3HgwAFs27YNK1euRGpqqnR86NChCA8Px927d6V9u3btQmZmJnr37i3TT4OI5MaJ7IhIdl5eXkhNTcX58+elGWmnTp2KnTt34sKFC3B2doabmxv++OMP6Xv69OmDjIwM7NmzB8CLXpq7d+9CX18fubm5ePbsGZo1a4bw8HBYWFgAAPr164d//vkHkZGRUjmTJ0/G7t27cf78eVy+fBk1a9bEiRMn8N577wEALl26BDc3NwQHB8Pf3x8AULt2bQwaNAiTJ08GAHTr1g3W1tYIDQ3V9o+KiEqIPTBEpBUeHh5q0+l7enriypUryM3Nlb4uyNPTExcvXlTbN2nSJCQkJODs2bOIiooCAPj4+EhlXLx4ES1atFD7nhYtWkj1XLx4EXp6emjcuLF0vFatWlIAyjd06FAprKSkpOCPP/7AkCFDNLh6ItI2BhgiemNVqlQJNWrUgKurK9q1a4clS5bg2LFjOHDggKz1DBw4ENevX0dMTAx++uknuLi4oFWrVrLWQUTyYoAhIq04fvy42texsbFwdXWFrq6u9PXLx93c3F5bZv73Pn36FADg5uaG6OhotXOio6Px7rvvQldXF7Vq1cLz588RFxcnHU9MTERaWpra91hbW6N79+4IDQ1FWFgYBg8e/N8vlIjKBFejJiKtSEpKwvjx4/H5558jPj4ey5cvV3tCKDo6GgsXLkT37t2xd+9e/Pzzz9i9e7daGY8ePUJycjKEELh9+zYmT54MGxsbNG/eHAAwYcIEvPfee5g7dy569+6NmJgYrFixAitXrgQA1KxZE506dcLnn3+OVatWQU9PD/7+/jA2Ni7U3qFDh6Jr167Izc3FoEGDtPiTISJZCCIimbVp00aMGjVKjBgxQpiZmQlLS0vxxRdfiLy8PCGEEE5OTmL27NmiV69eokKFCsLe3l4sXbpUrQwnJycBQNpsbGxEly5dxOnTp9XO27Ztm3B3dxf6+vqiWrVqYtGiRWrH7969K3x8fIShoaGoVq2aWL9+vXBychLBwcFq5+Xl5QknJyfRpUsX2X8eRCQ/PoVERLLz8vJCgwYNXjldv7OzM/z9/aWngN4Ejx8/RpUqVRAaGooePXqUdXOI6F/wFhIRlWt5eXm4d+8egoKCYGFhgW7dupV1k4joP2CAIaJyLSkpCS4uLqhatSrCwsKgp8e3RSIl4C0kIiIiUhw+Rk1ERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrDAENERESKwwBDREREisMAQ0RERIrzf+UHZF3a37wHAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "d3.pos_word_counter.head(20).plot.bar()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a Naive-Bayes based Sentiment-Analysis Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import GaussianNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PhraseId</th>\n",
       "      <th>SentenceId</th>\n",
       "      <th>Phrase</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>A series of escapades demonstrating the adage ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>A series of escapades demonstrating the adage ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>A series</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>series</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PhraseId  SentenceId                                             Phrase  \\\n",
       "0         1           1  A series of escapades demonstrating the adage ...   \n",
       "1         2           1  A series of escapades demonstrating the adage ...   \n",
       "2         3           1                                           A series   \n",
       "3         4           1                                                  A   \n",
       "4         5           1                                             series   \n",
       "\n",
       "   Sentiment  \n",
       "0          1  \n",
       "1          2  \n",
       "2          2  \n",
       "3          2  \n",
       "4          2  "
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent_raw = pd.read_csv(\"./kaggle-datasets/sentiment-analysis-on-movie-reviews/train.tsv\", sep='\\t')\n",
    "\n",
    "#TODO: worth re-mapping the data to 3 classes?\n",
    "\n",
    "sent_raw.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'id'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "File \u001b[1;32mc:\\Users\\theco\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\pandas\\core\\indexes\\base.py:3803\u001b[0m, in \u001b[0;36mIndex.get_loc\u001b[1;34m(self, key, method, tolerance)\u001b[0m\n\u001b[0;32m   3802\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m-> 3803\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_engine\u001b[39m.\u001b[39;49mget_loc(casted_key)\n\u001b[0;32m   3804\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mKeyError\u001b[39;00m \u001b[39mas\u001b[39;00m err:\n",
      "File \u001b[1;32mc:\\Users\\theco\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\pandas\\_libs\\index.pyx:138\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[1;34m()\u001b[0m\n",
      "File \u001b[1;32mc:\\Users\\theco\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\pandas\\_libs\\index.pyx:165\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[1;34m()\u001b[0m\n",
      "File \u001b[1;32mpandas\\_libs\\hashtable_class_helper.pxi:5745\u001b[0m, in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[1;34m()\u001b[0m\n",
      "File \u001b[1;32mpandas\\_libs\\hashtable_class_helper.pxi:5753\u001b[0m, in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: 'id'",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32md:\\Documents\\Phishing-ML\\classif.ipynb Cell 46\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> <a href='vscode-notebook-cell:/d%3A/Documents/Phishing-ML/classif.ipynb#Y243sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m sent_dataset \u001b[39m=\u001b[39m Dataset(sent_raw, preprocessing\u001b[39m=\u001b[39;49m[\u001b[39m\"\u001b[39;49m\u001b[39mlower\u001b[39;49m\u001b[39m\"\u001b[39;49m,\u001b[39m\"\u001b[39;49m\u001b[39mpunc\u001b[39;49m\u001b[39m\"\u001b[39;49m], feature_selection\u001b[39m=\u001b[39;49m[\u001b[39m\"\u001b[39;49m\u001b[39malltokens\u001b[39;49m\u001b[39m\"\u001b[39;49m,\u001b[39m\"\u001b[39;49m\u001b[39mcustom_tokenize\u001b[39;49m\u001b[39m\"\u001b[39;49m], bodyColumn\u001b[39m=\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39mPhrase\u001b[39;49m\u001b[39m'\u001b[39;49m)\n",
      "\u001b[1;32md:\\Documents\\Phishing-ML\\classif.ipynb Cell 46\u001b[0m in \u001b[0;36mDataset.__init__\u001b[1;34m(self, df, preprocessing, feature_selection, retList, bodyColumn)\u001b[0m\n\u001b[0;32m     <a href='vscode-notebook-cell:/d%3A/Documents/Phishing-ML/classif.ipynb#Y243sZmlsZQ%3D%3D?line=43'>44</a>\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mNLTK_ENGLISH_STOP \u001b[39m=\u001b[39m \u001b[39mset\u001b[39m(NLTK_STOP\u001b[39m.\u001b[39mwords(\u001b[39m'\u001b[39m\u001b[39menglish\u001b[39m\u001b[39m'\u001b[39m))\n\u001b[0;32m     <a href='vscode-notebook-cell:/d%3A/Documents/Phishing-ML/classif.ipynb#Y243sZmlsZQ%3D%3D?line=45'>46</a>\u001b[0m \u001b[39m# === apply preprocessing and feature selection ===\u001b[39;00m\n\u001b[1;32m---> <a href='vscode-notebook-cell:/d%3A/Documents/Phishing-ML/classif.ipynb#Y243sZmlsZQ%3D%3D?line=47'>48</a>\u001b[0m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mprocess_phrases()\n",
      "\u001b[1;32md:\\Documents\\Phishing-ML\\classif.ipynb Cell 46\u001b[0m in \u001b[0;36mDataset.process_phrases\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    <a href='vscode-notebook-cell:/d%3A/Documents/Phishing-ML/classif.ipynb#Y243sZmlsZQ%3D%3D?line=238'>239</a>\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mpreprocessed_phrases \u001b[39m=\u001b[39m [\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mpreprocess_phrase(phrase) \u001b[39mfor\u001b[39;00m phrase \u001b[39min\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mbodies]\n\u001b[0;32m    <a href='vscode-notebook-cell:/d%3A/Documents/Phishing-ML/classif.ipynb#Y243sZmlsZQ%3D%3D?line=239'>240</a>\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mfeatures \u001b[39m=\u001b[39m [\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mextract_features_from_phrase(phrase) \u001b[39mfor\u001b[39;00m phrase \u001b[39min\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mpreprocessed_phrases]\n\u001b[1;32m--> <a href='vscode-notebook-cell:/d%3A/Documents/Phishing-ML/classif.ipynb#Y243sZmlsZQ%3D%3D?line=241'>242</a>\u001b[0m data \u001b[39m=\u001b[39m {\u001b[39m'\u001b[39m\u001b[39mid\u001b[39m\u001b[39m'\u001b[39m: \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mdata[\u001b[39m'\u001b[39;49m\u001b[39mid\u001b[39;49m\u001b[39m'\u001b[39;49m], \u001b[39m'\u001b[39m\u001b[39mppBody\u001b[39m\u001b[39m'\u001b[39m: \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mfeatures, \u001b[39m'\u001b[39m\u001b[39mlabel\u001b[39m\u001b[39m'\u001b[39m: \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mdata[\u001b[39m'\u001b[39m\u001b[39mLabel\u001b[39m\u001b[39m'\u001b[39m]}\n\u001b[0;32m    <a href='vscode-notebook-cell:/d%3A/Documents/Phishing-ML/classif.ipynb#Y243sZmlsZQ%3D%3D?line=242'>243</a>\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mpp_df \u001b[39m=\u001b[39m pd\u001b[39m.\u001b[39mDataFrame(data\u001b[39m=\u001b[39mdata)\n",
      "File \u001b[1;32mc:\\Users\\theco\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\pandas\\core\\frame.py:3805\u001b[0m, in \u001b[0;36mDataFrame.__getitem__\u001b[1;34m(self, key)\u001b[0m\n\u001b[0;32m   3803\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcolumns\u001b[39m.\u001b[39mnlevels \u001b[39m>\u001b[39m \u001b[39m1\u001b[39m:\n\u001b[0;32m   3804\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_getitem_multilevel(key)\n\u001b[1;32m-> 3805\u001b[0m indexer \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mcolumns\u001b[39m.\u001b[39;49mget_loc(key)\n\u001b[0;32m   3806\u001b[0m \u001b[39mif\u001b[39;00m is_integer(indexer):\n\u001b[0;32m   3807\u001b[0m     indexer \u001b[39m=\u001b[39m [indexer]\n",
      "File \u001b[1;32mc:\\Users\\theco\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\pandas\\core\\indexes\\base.py:3805\u001b[0m, in \u001b[0;36mIndex.get_loc\u001b[1;34m(self, key, method, tolerance)\u001b[0m\n\u001b[0;32m   3803\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_engine\u001b[39m.\u001b[39mget_loc(casted_key)\n\u001b[0;32m   3804\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mKeyError\u001b[39;00m \u001b[39mas\u001b[39;00m err:\n\u001b[1;32m-> 3805\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mKeyError\u001b[39;00m(key) \u001b[39mfrom\u001b[39;00m \u001b[39merr\u001b[39;00m\n\u001b[0;32m   3806\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mTypeError\u001b[39;00m:\n\u001b[0;32m   3807\u001b[0m     \u001b[39m# If we have a listlike key, _check_indexing_error will raise\u001b[39;00m\n\u001b[0;32m   3808\u001b[0m     \u001b[39m#  InvalidIndexError. Otherwise we fall through and re-raise\u001b[39;00m\n\u001b[0;32m   3809\u001b[0m     \u001b[39m#  the TypeError.\u001b[39;00m\n\u001b[0;32m   3810\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_check_indexing_error(key)\n",
      "\u001b[1;31mKeyError\u001b[0m: 'id'"
     ]
    }
   ],
   "source": [
    "sent_dataset = Dataset(sent_raw, preprocessing=[\"lower\",\"punc\"], feature_selection=[\"alltokens\",\"custom_tokenize\"], bodyColumn='Phrase')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Here we use a standard labelled dataset to train the NB model. We use the whole dataset for training and validation, and do not set aside a testing dataset, as we will test against the email datasets\n",
    "\n",
    "Once we have defined a custom Sentiment measure, we can re-implement Naive Bayes with these 'custom' classes (for urgency, authority, scarcity, etc). We may use a naive dictionary-based system for labelling, and train based on this. However, we need to be careful with this approach as dictionary-based approaches struggle with things like sarcasm, negation, etc.\n",
    "\n",
    "More 'objective' features, such as a count of the number of imperative words, may be a better metric here (saying \"click here\" in an email is unlikely to be sarcastic, and \"do not click this link\" is still potentially fishy - unless it is part of an advisory telling employees not to interact with phishing emails...).\n",
    "\n",
    "Or, we may have to manually label a small training dataset."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Persuasion\n",
    "\n",
    "One of our metrics will be focused around detecting persuasive language. This paper is useful for this: https://convokit.cornell.edu/documentation/persuasionforgood.html"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Email Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import ShuffleSplit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = d.pp_df.iloc[:, 0:2]\n",
    "y = d.pp_df.iloc[:, 2]\n",
    "\n",
    "ss = ShuffleSplit(n_splits=1, test_size=0.25, random_state=10)\n",
    "indexes = list(ss.split(X, y))\n",
    "train_set  = indexes[0][0]\n",
    "test_set  = indexes[0][1]\n",
    "Xtrain = X.iloc[train_set, :]\n",
    "ytrain = y.iloc[train_set]\n",
    "Xtest = X.iloc[test_set, :]\n",
    "ytest = y.iloc[test_set]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>ppBody</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>save up to 70 on life insurance why spend more...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2.0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1 fight the risk of cancer http www adclick ws...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3.0</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1 fight the risk of cancer http www adclick ws...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4.0</th>\n",
       "      <td>3.0</td>\n",
       "      <td>adult club offers free membership instant acce...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5.0</th>\n",
       "      <td>4.0</td>\n",
       "      <td>i thought you might like these 1 slim down gua...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2601.0</th>\n",
       "      <td>2600.0</td>\n",
       "      <td>subject computationally intensive methods in q...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2602.0</th>\n",
       "      <td>2601.0</td>\n",
       "      <td>subject books a survey of american linguistics...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2603.0</th>\n",
       "      <td>2602.0</td>\n",
       "      <td>subject wecol 98 western conference on linguis...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2604.0</th>\n",
       "      <td>2603.0</td>\n",
       "      <td>subject euralex 98 revised programme euralex 9...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2605.0</th>\n",
       "      <td>2604.0</td>\n",
       "      <td>body label 0 subject great part time or summer...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>18651 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            id                                             ppBody label\n",
       "1.0        0.0  save up to 70 on life insurance why spend more...     1\n",
       "2.0        1.0  1 fight the risk of cancer http www adclick ws...     1\n",
       "3.0        2.0  1 fight the risk of cancer http www adclick ws...     1\n",
       "4.0        3.0  adult club offers free membership instant acce...     1\n",
       "5.0        4.0  i thought you might like these 1 slim down gua...     1\n",
       "...        ...                                                ...   ...\n",
       "2601.0  2600.0  subject computationally intensive methods in q...     0\n",
       "2602.0  2601.0  subject books a survey of american linguistics...     0\n",
       "2603.0  2602.0  subject wecol 98 western conference on linguis...     0\n",
       "2604.0  2603.0  subject euralex 98 revised programme euralex 9...     0\n",
       "2605.0  2604.0  body label 0 subject great part time or summer...     0\n",
       "\n",
       "[18651 rows x 3 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d.pp_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0       True\n",
       "2.0       True\n",
       "3.0       True\n",
       "4.0       True\n",
       "5.0       True\n",
       "          ... \n",
       "2601.0    True\n",
       "2602.0    True\n",
       "2603.0    True\n",
       "2604.0    True\n",
       "2605.0    True\n",
       "Name: label, Length: 18651, dtype: bool"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d.pp_df.iloc[:, -1] == y"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word2Vec\n",
    "\n",
    "Classify with word2vec (see https://towardsdatascience.com/text-classification-with-nlp-tf-idf-vs-word2vec-vs-bert-41ff868d1794)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT\n",
    "\n",
    "Classify with BERT (also using TDS article methods)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest\n",
    "\n",
    "Use RF/ensemble method for some of the classifiers above to see which features work best and which preprocessing methods are good - exploring Phishing-Dataset-for-Machine-Learning features first may be helpful"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Phishing-Dataset-for-Machine-Learning\n",
    "\n",
    "Inspect this dataset - it is for webpages (https://www.sciencedirect.com/science/article/pii/S0020025519300763?via%3Dihub) but some of the features may be applicable to email classification\n",
    "\n",
    "Other repos that use this dataset to take inspiration from:\n",
    "- https://github.com/andpalmier/MLWithPhishing\n",
    "- https://github.com/rewanthtammana/Detect-phishing-websites-using-ML\n",
    "- https://www.kaggle.com/code/fadilparves/pishing-detection-using-machine-learning\n",
    "\n",
    "Random forests used to discover best features - we could apply similar to the Email-Spam-Dataset"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Feature Extraction\n",
    "\n",
    "Can we apply the features used in this dataset to Email-Spam-Dataset?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transfer Learning\n",
    "\n",
    "Use a pretrained word representation and replace the classification layer, then fine-tune the model."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comments\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Todo\n",
    "\n",
    "- [x] Remove NaN id from merged `dfSpam`\n",
    "- [ ] Tokenise body data\n",
    "- [ ] Set seed method\n",
    "- [ ] Stratified Sample?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "26de051ba29f2982a8de78e945f0abaf191376122a1563185a90213a26c5da77"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}