ToxicComment_S13_Group8_Supplementary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "496f3024",
   "metadata": {},
   "source": [
    "# You're Toxic, I'm Slippin' Under: Toxic Comment Classification Challenge\n",
    "\n",
    "#### STINTSY S13 Group 8\n",
    "- VICENTE, Francheska Josefa\n",
    "- VISTA, Sophia Danielle S."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d0c582a",
   "metadata": {},
   "source": [
    "## Import Libraries\n",
    "Before starting, the relevant libraries and files in building and training the model should be loaded into the notebook first.\n",
    "\n",
    "#### Basic Libraries \n",
    "- `numpy` contains a large collection of mathematical functions\n",
    "- `pandas` contains functions that are designed for data manipulation and data analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7797c6f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6255319",
   "metadata": {},
   "source": [
    "#### Natural Language Processing Libraries \n",
    "- `re` is a module that allows the use of regular expressions\n",
    "- `nltk` provides functions for processing text data\n",
    "- `stopwords` is a corpus from NLTK, which includes a compiled list of stopwords\n",
    "- `Counter` is from Python's `collections` module, which is helpful for tokenization\n",
    "- `string` contains functions for string operations\n",
    "- `TFidfVectorizer` converts the given text documents into a matrix, which has TF-IDF features \n",
    "- `CountVectorizer` converts the given text documents into a matrix, which has the counts of the tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ce303532",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import nltk\n",
    "import string\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "from collections import Counter\n",
    "from gensim.models import Word2Vec\n",
    "from gensim.models import Doc2Vec\n",
    "from nltk.tokenize.casual import TweetTokenizer\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c181e3",
   "metadata": {},
   "source": [
    "#### Machine Learning Libraries\n",
    "The following code block can be used to install **scikit-multilearn** without restarting Jupyter Notebook. The `sys` module is used to access the *executable* function of the interpreter, which would run the installation of scikit-multilearn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "850d2434",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: scikit-multilearn in c:\\users\\user\\anaconda3\\lib\\site-packages (0.2.0)\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install scikit-multilearn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f60a154c",
   "metadata": {},
   "source": [
    "The following libraries are multi-label classification modules that would allow the usage of one model that can classify one instance as more than one class.\n",
    "- `ClassifierChain` chains binary classifiers in a way that its predictions are dependent on the earlier classes\n",
    "- `BinaryRelevance` uses binary classifiers to classify the classes independently\n",
    "- `MultiOutputClassifier` fits one classifier per target class \n",
    "- `OneVsRestClassifier` fits one class against the other classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ea567ba9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from skmultilearn.problem_transform import ClassifierChain\n",
    "from skmultilearn.problem_transform import BinaryRelevance\n",
    "from sklearn.multioutput import MultiOutputClassifier\n",
    "from sklearn.multiclass import OneVsRestClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5da4f7f3",
   "metadata": {},
   "source": [
    "The following classes are classifiers that implement different methods of classification.\n",
    "- `RandomForestClassifier` is a class under the ensemble module that trains by fitting using a number of decision trees\n",
    "- `GradientBoostingClassifier` is a class under the ensemble module that optimizes arbitrary differentiable loss functions\n",
    "- `AdaBoostClassifier` is a class under the ensemble module that implements AdaBoost-SAMME\n",
    "- `MultinomialNB` is a class under the Naive Bayes module that allows the classification of discrete features\n",
    "- `LogisticRegression` is a class under the linear models module that implements regularized logistic regression\n",
    "- `SGDClassifier` is a class under the linear models module that implements regularized linear models with stochastic gradient descent (SGD) learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "836013db",
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.linear_model import SGDClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d3e6642",
   "metadata": {},
   "source": [
    "Meanwhile, the following classes are used for hyperparameter tuning.\n",
    "- `ParameterGrid` is a class that allows the iteration over different combinations of parameter values \n",
    "- `GridSearchCV` is a cross-validation class that allows the exhaustive search over all possible combinations of hyperparameter values\n",
    "- `RandomizedSearchCV` is a cross-validation class that allows a random search over some possible combinations of hyperparameter values\n",
    "- `train_test_split` divides the dataset into two subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "54f9c18e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ParameterGrid\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf7a0b4",
   "metadata": {},
   "source": [
    "And lastly, these classes computes different scores about how well a model works.\n",
    "- `log_loss` computes the Logistic loss given the true values and the predicted values\n",
    "- `f1_score` computes the balanced F-score by comparing the actual classes and the predicted classes\n",
    "- `accuracy_score` computes the accuracy by determining how many classes were correctly predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "93333f06",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import log_loss\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c878a641",
   "metadata": {},
   "source": [
    "The warnings module is used to ignore any ConvergenceWarnings that might appear when doing hyperparameter tuning. As these models will not be chosen due to low accuracy scores, the warnings would only clutter the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "23c3827f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "from sklearn.exceptions import ConvergenceWarning\n",
    "warnings.filterwarnings(action='ignore', category = ConvergenceWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbe4ba0e",
   "metadata": {},
   "source": [
    "### Load Files\n",
    "The csv files to be loaded here contains the datasets that have already gone through the data cleaning and preprocessing techniques discussed in the main notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8424525e",
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv('cleaned_data/cleaned_train.csv')\n",
    "test = pd.read_csv('cleaned_data/cleaned_test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe1ca0f",
   "metadata": {},
   "source": [
    "## Initialize Datasets\n",
    "Before using these datasets, we would need to convert the values in the `comment_text` column into either \"str, unicode or file objects\", according to the documentation of TF-IDF vectorizer and Count vectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d03b0b3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "test ['comment_text'] = test ['comment_text'].apply(lambda x: np.str_(x))\n",
    "train ['comment_text'] = train ['comment_text'].apply(lambda x: np.str_(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d7c6cd",
   "metadata": {},
   "source": [
    "Then, we would be declaring our **X_train**, **y_train**, and **X_test**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "84367b5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train ['comment_text']\n",
    "y_train = train.loc [ : , 'toxic' : ]\n",
    "\n",
    "X_test = test ['comment_text']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed343812",
   "metadata": {},
   "source": [
    "Afterwards, we would be declaring the different classes that our model would need to predict. This can be found in the **train** data's column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c232fbad",
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = train.columns [2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3df21c9",
   "metadata": {},
   "source": [
    "## Vectorizing Data\n",
    "As explained in the **Feature Engineering** part of the main notebook, three types of vectorizers would be used: (1) Count Vectorizer, (2) TF-IDF Vectorizer, and (3) Average Word2Vec Vectors.\n",
    "\n",
    "Two types of CountVectorizer and TF-IDF Vectorizers were made in consideration of the more complex estimators: one with no **max_features** parameter, and one with a **max_features** parameter that is equal to 5000. Limiting the number of max features would lessen the time and space complexity from training the estimators; this would lessen the burden on our machines."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec18f9b5",
   "metadata": {},
   "source": [
    "#### Count Vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4b6d1b3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "count_vectorizer = CountVectorizer()                     # creating the Vectorizer with no max features\n",
    "count_train = count_vectorizer.fit_transform(X_train)    # fitting the vectorizer according to the train data, and then\n",
    "                                                         # returning the transformed train data\n",
    "count_test = count_vectorizer.transform(X_test)          # returning the transformed test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4c9749e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "count_vectorizer_5000 = CountVectorizer(max_features = 5000)     # creating the Vectorizer with max features = 5000\n",
    "count_train_5000 = count_vectorizer_5000.fit_transform(X_train)  # fitting the vectorizer according to the train data, and then\n",
    "                                                                 # returning the transformed train data\n",
    "count_test_5000 = count_vectorizer_5000.transform(X_test)        # returning the transformed test data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cec4160",
   "metadata": {},
   "source": [
    "#### TF-IDF Vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "60850d4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_vectorizer = TfidfVectorizer()                    # creating the Vectorizer with no max features\n",
    "tfidf_train = tfidf_vectorizer.fit_transform(X_train)   # fitting the vectorizer according to the train data, and then\n",
    "                                                        # returning the transformed train data\n",
    "tfidf_test = tfidf_vectorizer.transform(X_test)         # returning the transformed test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4cdca5f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_vectorizer_5000 = TfidfVectorizer(max_features = 5000)    # creating the Vectorizer with max features = 5000\n",
    "tfidf_train_5000 = tfidf_vectorizer_5000.fit_transform(X_train) # fitting the vectorizer according to the train data, and then\n",
    "                                                                # returning the transformed train data\n",
    "tfidf_test_5000 = tfidf_vectorizer_5000.transform(X_test)       # returning the transformed test data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "411c4764",
   "metadata": {},
   "source": [
    "#### Average Word2Vec Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f2c6a2",
   "metadata": {},
   "source": [
    "Before building a Word2Vec model, the data must be tokenized to produce a list of lists of tokens as indicated in Gensim's documentation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d874e588",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(data):\n",
    "    t = TweetTokenizer()                     # initialize tokenizer\n",
    "    tokens_list = []                         # initialize empty list\n",
    "    \n",
    "    for text in data:\n",
    "        tokens_list += [t.tokenize(text)]    # add tokenized sentence to list\n",
    "        \n",
    "    return tokens_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e795188c",
   "metadata": {},
   "source": [
    "The train set is tokenized using NLTK's `TweetTokenizer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c1d9007c",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_train = tokenize(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bd82688",
   "metadata": {},
   "source": [
    "The Word2Vec model is trained using this tokenized list, which would transform these words into word vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "98e63d59",
   "metadata": {},
   "outputs": [],
   "source": [
    "wrd2v_model = Word2Vec(tokens_train, epochs=30, sg=0, workers=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55fb824f",
   "metadata": {},
   "source": [
    "To transform the word vectors into usable features, these vectors are averaged for all words in the model's vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a1fe9b72",
   "metadata": {},
   "outputs": [],
   "source": [
    "def vectorize_word2vec(model, tokens_list):\n",
    "    vectors = []\n",
    "    \n",
    "    for tokens in tokens_list:                    # iterate through each sentence\n",
    "        feat = np.zeros(100)                      # initializes a list that will hold the vectors\n",
    "        count = 0                                 # initializes the word count for a sentence\n",
    "        \n",
    "        for token in tokens:                      # iterate through each word in the sentence\n",
    "            if token in model.wv.index_to_key:    # if the word is in the model's vocabulary...\n",
    "                feat += model.wv[token]           # ...add word vectors to list and...\n",
    "                count += 1                        # ...update the word count\n",
    "        \n",
    "        if count > 1:                             # if sentence contains more than 1 word in the model...\n",
    "            feat /= count                         # ...divide word vectors by word count to get the average\n",
    "            \n",
    "        vectors.append(feat)                      # add the averaged vectors to the list\n",
    "        \n",
    "    return vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1827eb0a",
   "metadata": {},
   "source": [
    "Using the defined function, the train data can be vectorized using the word vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0d6de79c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "wrd2v_train = vectorize_word2vec(wrd2v_model, tokens_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ac794f0",
   "metadata": {},
   "source": [
    "The test data is also vectorized as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f63f8433",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_test = tokenize(X_test)\n",
    "wrd2v_test = vectorize_word2vec(wrd2v_model, tokens_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54ac6bbd",
   "metadata": {},
   "source": [
    "## Training and Tuning Different Models <a class=\"anchor\" id=\"toc\"></a>\n",
    "As an experiment to find the model with the highest ROC AUC score in the test set, we would be training and predicting using different models. Two approaches were tested: creating an array of models (i.e., fitting one classifier for each of the labels), and utilizing SKLearn's pre-made multi-label classifiers to create one classifier for the whole task. \n",
    "\n",
    "Mostly, Logistic Regression, Multinomial Naive Bayes, and Random Forest Classifiers are used as base models, as these are the common models used for text classification. The experiment was expanded further by testing how the type of feature fed on the model affects the accuracy score. This was done by creating two classifiers each for the models, wherein one utilized a TF-IDF vector as its input, and another used a Count vector. \n",
    "\n",
    "#### Variable and Function Declarations\n",
    "* [**Helper Functions**](#functs)\n",
    "* [**Hyperparameters**](#params)\n",
    "\n",
    "### Six Single-Label Classifiers\n",
    "* [**Logistic Regression**](#lr)\n",
    "* [**Multinomial Naive Bayes**](#mn)\n",
    "* [**Random Forest Classifier**](#rf)\n",
    "* [**Gradient Boosting Classifier**](#gbc)\n",
    "* [**eXtreme Gradient Boosting Classifier**](#xgb)\n",
    "* [**AdaBoostClassifier Boosting Classifier**](#adb)\n",
    "* [**Stochastic Gradient Descent Classifier**](#sgd)\n",
    "\n",
    "### Multi-Label Classifiers\n",
    "* [**OneVsRest Classifier: Logistic Regression**](#oc_lr)\n",
    "* [**OneVsRest Classifier: Multinomial Naive Bayes**](#oc_mn)\n",
    "* [**MultiOutput Classifier: Logistic Regression**](#mo_lr)\n",
    "* [**MultiOutput Classifier: Multinomial Naive Bayes**](#mo_mn)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15478728",
   "metadata": {},
   "source": [
    "### Declaring Helper Functions <a class=\"anchor\" id=\"functs\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>\n",
    "Helper functions that would be repeatedly used throughout the notebook, will be declared and discussed here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3a6101e",
   "metadata": {},
   "source": [
    "#### Submission Template Functions\n",
    "The following `to_submission_csv` functions are used to create CSV files with the correct submission template. The first function is used by almost all models, while a modified version is used for the MultiOutput Classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4f28762a",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_submission = pd.read_csv('data/sample_submission.csv')\n",
    "sample_submission ['id'] = test ['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5a3d8620",
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_submission_csv(predictions, filename):\n",
    "    for i in range (6):\n",
    "        sample_submission[classes [i]] = predictions[:, i : i + 1]\n",
    "\n",
    "    sample_submission.to_csv(f'results/' + filename + '.csv', index = False) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c1aeb6cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_submission_csv_multiclass(predictions, filename):\n",
    "    for i in range (6):\n",
    "        temp = list(zip(*predictions[i]))\n",
    "        sample_submission[classes [i]] = temp[1]\n",
    "\n",
    "    sample_submission.to_csv(f'results/' + filename + '.csv', index = False)     "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04a4b117",
   "metadata": {},
   "source": [
    "#### Display Functions\n",
    "The `format_results` function is used to compute for the final test accuracy and to display the final results as a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1d9eb531",
   "metadata": {},
   "outputs": [],
   "source": [
    "index = ['model', 'vector', 'tuned', 'private', 'public', 'test accuracy']\n",
    "all_results = pd.DataFrame(index=index)\n",
    "\n",
    "def update_results(all_results, results):\n",
    "    for result in results:\n",
    "        # private score accounts for 90% of the test data, while the remaining 10% is the public score\n",
    "        results[result] += [round((results[result][3]*9 + results[result][4]) / 10, 5)]\n",
    "\n",
    "    return pd.concat([pd.DataFrame(results, index=index), all_results], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba67bfb5",
   "metadata": {},
   "source": [
    "#### Training and Tuning Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65432a0a",
   "metadata": {},
   "source": [
    "As the task requires us to give predictions for six classes, the `train_models` function would train multiple classifiers that will give predictions for a given class. The function would fit each classifier using the passed train set, compute for the training accuracies for each class, then predict the classes of the passed test set.\n",
    "\n",
    "The function will return the trained models and their predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5ecee997",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_models(model, X_train, X_test):\n",
    "    \"\"\"Trains six models using a given train and test set.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : estimator object\n",
    "        the type of estimator to be trained \n",
    "    X_train : \n",
    "        the data used in fitting the model\n",
    "    X_test : \n",
    "        the data to be predicted\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    models\n",
    "        a list of fitted estimator objects\n",
    "    test_predictions\n",
    "        a list of prediction probabilities by the fitted model\n",
    "    \"\"\"\n",
    "    \n",
    "    test_predictions = np.zeros((len(test), len(classes)))                  # initialize empty list for predictions\n",
    "    models = []                                                             # initialize empty list for models\n",
    "    train_accuracy = []\n",
    "    \n",
    "    \n",
    "    print('Fitting', str(model) + '...')\n",
    "    \n",
    "    for i in range(6):                                                      # loop for each of six classes\n",
    "        \n",
    "        model.fit(X_train, y_train[classes[i]])                             # fit the model\n",
    "        \n",
    "        train_predictions = model.predict(X_train)                          # predict using train data\n",
    "        accuracy = accuracy_score(train_predictions, y_train[classes[i]])   # get training accuracy \n",
    "        print(classes[i] + ':', accuracy)\n",
    "        \n",
    "        test_predictions[:,i] = model.predict_proba(X_test)[:,1]            # predict using test data\n",
    "        \n",
    "        models += [model]\n",
    "        train_accuracy += [accuracy]\n",
    "    \n",
    "    print('\\nOverall training accuracy:', np.mean(train_accuracy))\n",
    "    \n",
    "    return models, test_predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54662d22",
   "metadata": {},
   "source": [
    "Similarly, the `tune_and_train_models` function will train multiple classifiers with the addition of hyperparameter tuning to achieve a better training accuracy. Hyperparameter tuning will be done using a `GridSearchCV` for a more comprehensive search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "887dbaed",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tune_and_train_models(model, hyperparameters, X_train, X_test, scoring='accuracy', cv=2):\n",
    "    \"\"\"Tunes six models using a given train and test set.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : estimator object\n",
    "        the type of estimator to be trained \n",
    "    hyperparameters : estimator object\n",
    "        the hyperparameters used for tuning the model  \n",
    "    X_train : \n",
    "        the data used in fitting the model\n",
    "    X_test : \n",
    "        the data to be predicted\n",
    "    scoring : \n",
    "        the metric for deciding the best combination of parameters\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    models\n",
    "        a list of fitted estimator objects\n",
    "    test_predictions\n",
    "        a list of prediction probabilities by the fitted model\n",
    "    \"\"\"\n",
    "    \n",
    "    test_predictions = np.zeros((len(test), len(classes)))                  # initialize empty list for predictions\n",
    "    models = []                                                             # initialize empty list for models\n",
    "    train_accuracy = []\n",
    "    \n",
    "    print('Tuning', str(model) + '...')\n",
    "    \n",
    "    for i in range(6):                                                              # loop for each of six classes\n",
    "        model_cv = GridSearchCV(model, hyperparameters, \n",
    "                                cv=cv, scoring=scoring)\n",
    "        model_cv.fit(X_train, y_train[classes[i]])\n",
    "        \n",
    "        train_predictions = model_cv.predict(X_train)                               # predict using train data\n",
    "        accuracy = accuracy_score(train_predictions, y_train[classes[i]])           # get training accuracy \n",
    "        print(classes[i] + ':', accuracy, model_cv.best_params_)\n",
    "        \n",
    "        test_predictions[:,i] = model_cv.predict_proba(X_test)[:,1]                 # predict using test data\n",
    "        \n",
    "        models += [model_cv.best_estimator_]\n",
    "        train_accuracy += [accuracy]\n",
    "    \n",
    "    print('\\nOverall training', scoring + ':', np.mean(train_accuracy))\n",
    "    \n",
    "    return models, test_predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "629a521b",
   "metadata": {},
   "source": [
    "Multi-label classifiers would follow the same pipeline of training the model, getting the training predictions, and predicting the classes of the test data. As such, the `train_model` and `tune_and_train_model` functions will forgo the loop and proceed to train and/or tune the model using the whole **y_train**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7702b49b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, X_train, X_test, dense=False):\n",
    "    \"\"\"Trains a model using a given train and test set.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : estimator object\n",
    "        the type of estimator to be trained \n",
    "    X_train : \n",
    "        the data used in fitting the model\n",
    "    X_test : \n",
    "        the data to be predicted\n",
    "    dense : \n",
    "        a boolean value indicating if the predictions need to be converted to dense\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    model\n",
    "        a fitted estimator object\n",
    "    test_predictions\n",
    "        a list of prediction probabilities by the fitted model\n",
    "    \"\"\"\n",
    "    \n",
    "    print('Fitting', str(model) + '...')\n",
    "    \n",
    "    model.fit(X_train, y_train)                                               # fit the model\n",
    "    train_predictions = model.predict(X_train)                                # predict using train data\n",
    "    \n",
    "    if dense:                                           \n",
    "        train_predictions = train_predictions.todense()                       # convert predictions to dense\n",
    "        \n",
    "    accuracy = accuracy_score(train_predictions, y_train)                     # get training accuracy \n",
    "    print(accuracy)                                                        \n",
    "    \n",
    "    test_predictions = model.predict_proba(X_test)                            # predict using test data\n",
    "    \n",
    "    return model, test_predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ea53f9",
   "metadata": {},
   "source": [
    "As with the previous functions, the `tune_and_train_model` function will tune a single multi-label classifier using a `GridSearchCV` to increase the training accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "1f419c91",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tune_and_train_model(model, hyperparameters, X_train, X_test, scoring='accuracy', dense=False, cv=2):\n",
    "    \"\"\"Tunes a model using a given train and test set.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : estimator object\n",
    "        the type of estimator to be trained \n",
    "    hyperparameters : estimator object\n",
    "        the hyperparameters used for tuning the model  \n",
    "    X_train : \n",
    "        the data used in fitting the model\n",
    "    X_test : \n",
    "        the data to be predicted\n",
    "    scoring : \n",
    "        the metric for deciding the best combination of parameters\n",
    "    dense : \n",
    "        a boolean value indicating if the predictions need to be converted to dense\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    model\n",
    "        a  fitted estimator object\n",
    "    test_predictions\n",
    "        a list of prediction probabilities by the fitted model\n",
    "    \"\"\"\n",
    "    \n",
    "    print('Tuning', str(model) + '...')\n",
    "\n",
    "    model_cv = GridSearchCV(model, hyperparameters, \n",
    "                            cv=cv, scoring=scoring)\n",
    "    model_cv.fit(X_train, y_train)\n",
    "\n",
    "    train_predictions = model_cv.predict(X_train)                                # predict using train data\n",
    "    \n",
    "    if dense:                                           \n",
    "        train_predictions = train_predictions.todense()                          # convert predictions to dense\n",
    "        \n",
    "    accuracy = accuracy_score(train_predictions, y_train)                        # get training accuracy \n",
    "    print(accuracy, model_cv.best_params_)                                                        \n",
    "    \n",
    "    test_predictions = model_cv.predict_proba(X_test)                            # predict using test data\n",
    "    \n",
    "    return model_cv, test_predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ce26671",
   "metadata": {},
   "source": [
    "### Declaring Hyperparameter Values <a class=\"anchor\" id=\"params\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>\n",
    "As hyperparameters for each base estimator will remain constant, these will be declared here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "194e83d3",
   "metadata": {},
   "source": [
    "#### Logistic Regression Hyperparameters <a class=\"anchor\" id=\"param_lr\"></a>\n",
    "Tuning Logistic Regression models mostly involve altering the C, which controls the regularization strength, and the maximum number of iterations. \n",
    "\n",
    "For the C value, different powers of the default value (1) were tested to see if a stronger or weaker regularization strength can affect the results.\n",
    "\n",
    "During earlier testing stages, it was determined that the default number of max iterations (100) resulted in a ConvergenceWarning. With this, higher values were considered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c75638bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_lr = [{\n",
    "    'C' : [0.01, 0.1, 1, 10],\n",
    "    'max_iter' : [50, 100, 300, 600, 900] \n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76ce74e5",
   "metadata": {},
   "source": [
    "As the OneVsRest Classifier and MultiOutput Classifier require a slightly altered format, this was declared as a different variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "153902f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_lr_mo = [{\n",
    "    'estimator__C': [0.01, 0.1, 1, 10],           \n",
    "    'estimator__max_iter': [50, 100, 300, 600, 900] \n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baf27334",
   "metadata": {},
   "source": [
    "The Binary Relevance classifier also requires its own separate format, as seen below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c2442034",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_lr_multi = [{\n",
    "    'classifier': [LogisticRegression()],\n",
    "    'classifier__C': [0.01, 0.1, 1, 10],            \n",
    "    'classifier__max_iter': [50, 100, 300, 600, 900] \n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b95e23d",
   "metadata": {},
   "source": [
    "#### Multinomial Naive Bayes Hyperparameters <a class=\"anchor\" id=\"param_mnb\"></a>\n",
    "For Multinomial Naive Bayes hyperparameters, we would be tuning the alpha and fit_prior hyperparameters. The value of alpha indicates the value that would be used as the additive smoothing, while the fit_prior determines if the class prior probabilities would be learned. \n",
    "\n",
    "As we would be experimenting with an n-classifier and 1-classifier approach, we would need to declare different sets of hyperparameters to take into account the needs of the classifiers. First, we would be declaring the hyperparameters that would be used by the n-classifier approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ee4b0a57",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_mnb = [{\n",
    "    'alpha' : [0.0001, 0.001, 0.1, 1, 10, 100, 1000],\n",
    "    'fit_prior' : [True, False]\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8f4323",
   "metadata": {},
   "source": [
    "Next, we would be declaring the hyperparameters to be used by the `MultiOutputClassifier` and `OneVsRestClassifier`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "da924612",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_mn_mo = [{\n",
    "    'estimator__alpha': [0.0001, 0.001, 0.1, 1, 10, 100, 1000], \n",
    "    'estimator__fit_prior': [True, False]\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "003e10be",
   "metadata": {},
   "source": [
    "Last is the hyperparameters used by the `ClassifierChain` and `BinaryRelevance` classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "0a58b0d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_mn_multi = [{\n",
    "    'classifier': [MultinomialNB()],\n",
    "    'classifier__alpha': [0.0001, 0.001, 0.1, 1, 10, 100, 1000],  \n",
    "    'classifier__fit_prior': [True, False]\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ddf8952",
   "metadata": {},
   "source": [
    "#### Random Forest Classifier Hyperparameters <a class=\"anchor\" id=\"param_rf\"></a>\n",
    "For Random Forest Classifier hyperparameters, the `n_estimators` parameter refers to the number of trees in the forest while `max_features` is the size of the random subsets of features to consider when splitting a node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "id": "865abbaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_rf = [{\n",
    "    'n_estimators' : [100, 200],\n",
    "    'max_features' : ['sqrt', 'log2'], \n",
    "    'max_depth' : [1000],\n",
    "    'max_leaf_nodes' : [100]\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aa7f119",
   "metadata": {},
   "source": [
    "#### Gradient Boosting Classifier Hyperparameters <a class=\"anchor\" id=\"param_gbc\"></a>\n",
    "For  Gradient Boosting Classifier hyperparameters, the learning rate and the number of estimators are tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2ebe92c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_gbc = [{\n",
    "    'n_estimators' : [50, 100, 250],\n",
    "    'learning_rate' : [0.001, 0.01, 0.1, 1, 1.2],\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0db7abec",
   "metadata": {},
   "source": [
    "#### XGBoost Classifier Hyperparameters <a class=\"anchor\" id=\"param_xgb\"></a>\n",
    "For XGBoost Classifier hyperparameters, only the learning rate was tuned to lessen the time complexity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ca0f7176",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_xgb = [{\n",
    "    'learning_rate' : [0.001, 0.01, 0.1, 1, 1.2],\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d3753a8",
   "metadata": {},
   "source": [
    "#### Adaboost Classifier Hyperparameters <a class=\"anchor\" id=\"param_adb\"></a>\n",
    "For Adaboost Classifier hyperparameters, the learning rate and the number of estimators are tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "id": "2c28c698",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_adb = {\n",
    "    'n_estimators' : [10, 25, 50, 100],\n",
    "    'learning_rate' : [0.01, 0.1, 1, 1.2]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66fcd8d",
   "metadata": {},
   "source": [
    "#### SGDClassifier Hyperparameters <a class=\"anchor\" id=\"param_sgd\"></a>\n",
    "For SGDClassifier hyperparameters, the loss function and the alpha are tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "id": "29d29c93",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters_sgd = [{\n",
    "    'loss' : ['log', 'modified_huber'],\n",
    "    'alpha' : [0.0001, 0.001, 0.01, 0.1, 1, 10, 100]\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a025af9f",
   "metadata": {},
   "source": [
    "## Model Experimentation\n",
    "It should be noted that all models used below will have these constant values for parameters, if applicable:\n",
    "* `n_jobs = -1`, which would ensure that all CPU cores will be used for faster processing,\n",
    "* `class_weight='balanced'`, which would take into account the imbalance between the classes, and\n",
    "* `random_state=8`, which would ensure that the output can be reproduced"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae47d6f2",
   "metadata": {},
   "source": [
    "### Logistic Regression <a class=\"anchor\" id=\"lr\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39db2b05",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "A `LogisticRegression()` object is first initialized to be used as the base classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "51162851",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression(n_jobs=-1, class_weight='balanced')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01452841",
   "metadata": {},
   "source": [
    "The model is then trained using the Count vectorized train data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "id": "78aad487",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting LogisticRegression(class_weight='balanced', n_jobs=-1)...\n",
      "toxic: 0.9616847672822756\n",
      "severe_toxic: 0.9756534708687669\n",
      "obscene: 0.9744627783243822\n",
      "threat: 0.9962963195066773\n",
      "insult: 0.9615468976192416\n",
      "identity_hate: 0.9769820330761855\n",
      "\n",
      "Overall training accuracy: 0.9744377111129215\n",
      "Wall time: 1min 2s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lr_models_count, predictions_lr_count = train_models(lr, count_train, count_test)\n",
    "to_submission_csv(predictions_lr_count, 'submission_lr_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0bc5451",
   "metadata": {},
   "source": [
    "Next, the model will be trained using the TF-IDF vectorized data as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "id": "72d587df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting LogisticRegression(class_weight='balanced', n_jobs=-1)...\n",
      "toxic: 0.9574233413339517\n",
      "severe_toxic: 0.9794260861936066\n",
      "obscene: 0.9807421147952949\n",
      "threat: 0.9937519975434133\n",
      "insult: 0.9681019734162223\n",
      "identity_hate: 0.9810241209242281\n",
      "\n",
      "Overall training accuracy: 0.9767449390344529\n",
      "Wall time: 52.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lr_models_tfidf, predictions_lr_tfidf = train_models(lr, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_lr_tfidf, 'submission_lr_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75aab4e3",
   "metadata": {},
   "source": [
    "Lastly, the Word2Vec vectorized data will be used to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "id": "7e9ece5f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting LogisticRegression(class_weight='balanced', n_jobs=-1)...\n",
      "toxic: 0.8984088587525302\n",
      "severe_toxic: 0.948117139079156\n",
      "obscene: 0.92264258543219\n",
      "threat: 0.9272236183266382\n",
      "insult: 0.9160687092266139\n",
      "identity_hate: 0.9084670773511477\n",
      "\n",
      "Overall training accuracy: 0.9201546646947126\n",
      "Wall time: 4min 6s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lr_models_wrd2v, predictions_lr_wrd2v = train_models(lr, wrd2v_train, wrd2v_test)\n",
    "to_submission_csv(predictions_lr_wrd2v, 'submission_lr_wrd2v')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c0ebed0",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "A `LogisticRegression()` object with default parameters will serve as the base estimator, which will be tuned using the [`parameters_lr`](#param_lr) hyperparameters. This will first be tuned using Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "ea6b02bd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning LogisticRegression(class_weight='balanced', n_jobs=-1)...\n",
      "toxic: 0.9836123105075484 {'C': 10, 'max_iter': 300}\n",
      "severe_toxic: 0.9899292477956521 {'C': 10, 'max_iter': 600}\n",
      "obscene: 0.9879802721045804 {'C': 1, 'max_iter': 600}\n",
      "threat: 0.9976499489255567 {'C': 10, 'max_iter': 100}\n",
      "insult: 0.9784860657638292 {'C': 1, 'max_iter': 600}\n",
      "identity_hate: 0.9898540461612699 {'C': 10, 'max_iter': 300}\n",
      "\n",
      "Overall training accuracy: 0.9879186485430728\n",
      "Wall time: 1h 17min 13s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lr = LogisticRegression(n_jobs=-1, class_weight='balanced')\n",
    "lr_models_count_tuned, predictions_lr_count_tuned = tune_and_train_models(lr, parameters_lr, count_train, count_test)\n",
    "to_submission_csv(predictions_lr_count_tuned, 'submission_lr_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d3f7a6",
   "metadata": {},
   "source": [
    "Then, TF-IDF vectors will be used in training and tuning the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "6a6d9acc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning LogisticRegression(class_weight='balanced', n_jobs=-1)...\n",
      "toxic: 0.9787116706669758 {'C': 10, 'max_iter': 100}\n",
      "severe_toxic: 0.989340168326325 {'C': 10, 'max_iter': 300}\n",
      "obscene: 0.990167386304529 {'C': 10, 'max_iter': 100}\n",
      "threat: 0.998145026351906 {'C': 10, 'max_iter': 100}\n",
      "insult: 0.9829918970238953 {'C': 10, 'max_iter': 300}\n",
      "identity_hate: 0.9926177062248153 {'C': 10, 'max_iter': 100}\n",
      "\n",
      "Overall training accuracy: 0.9886623091497412\n",
      "Wall time: 28min 23s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lr = LogisticRegression(n_jobs=-1, class_weight='balanced')\n",
    "lr_models_tfidf_tuned, predictions_lr_tfidf_tuned = tune_and_train_models(lr, parameters_lr, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_lr_tfidf_tuned, 'submission_lr_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55fa65e6",
   "metadata": {},
   "source": [
    "Lastly, the model using Word2Vec vectors will be tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "6ed33d18",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning LogisticRegression(class_weight='balanced', n_jobs=-1)...\n",
      "toxic: 0.8991358078848913 {'C': 0.01, 'max_iter': 100}\n",
      "severe_toxic: 0.9492012959748325 {'C': 0.01, 'max_iter': 100}\n",
      "obscene: 0.9227365874751678 {'C': 10, 'max_iter': 100}\n",
      "threat: 0.9283579096452362 {'C': 10, 'max_iter': 300}\n",
      "insult: 0.9174724730684147 {'C': 0.01, 'max_iter': 100}\n",
      "identity_hate: 0.9094948330210376 {'C': 0.01, 'max_iter': 100}\n",
      "\n",
      "Overall training accuracy: 0.9210664845115968\n",
      "Wall time: 1h 14min 5s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lr = LogisticRegression(n_jobs=-1, class_weight='balanced')\n",
    "lr_models_wrd2v_tuned, predictions_lr_wrd2v_tuned = tune_and_train_models(lr, parameters_lr, wrd2v_train, wrd2v_test)\n",
    "to_submission_csv(predictions_lr_wrd2v_tuned, 'submission_lr_wrd2v_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b79b557",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "As seen from the scores returned by Kaggle, the models performed quite accurately as all models reached at least 90% accuracy. Of the Logistic Regression models, the untuned model trained with TF-IDF vectors gave the highest training accuracy rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "id": "507cf514",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_lr_tfidf</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_tfidf_tuned</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.97135</td>\n",
       "      <td>0.97227</td>\n",
       "      <td>0.97144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_wrd2v_tuned</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.9524</td>\n",
       "      <td>0.94993</td>\n",
       "      <td>0.95215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_wrd2v</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.95233</td>\n",
       "      <td>0.94982</td>\n",
       "      <td>0.95208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_count</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.94845</td>\n",
       "      <td>0.94248</td>\n",
       "      <td>0.94785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_count_tuned</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.9145</td>\n",
       "      <td>0.91797</td>\n",
       "      <td>0.91485</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         model            vector      tuned  \\\n",
       "submission_lr_tfidf        Logistic Regression    TF-IDF Vectors  Not Tuned   \n",
       "submission_lr_tfidf_tuned  Logistic Regression    TF-IDF Vectors      Tuned   \n",
       "submission_lr_wrd2v_tuned  Logistic Regression  Word2Vec Vectors      Tuned   \n",
       "submission_lr_wrd2v        Logistic Regression  Word2Vec Vectors  Not Tuned   \n",
       "submission_lr_count        Logistic Regression     Count Vectors  Not Tuned   \n",
       "submission_lr_count_tuned  Logistic Regression     Count Vectors      Tuned   \n",
       "\n",
       "                           private   public test accuracy  \n",
       "submission_lr_tfidf        0.97558  0.97621       0.97564  \n",
       "submission_lr_tfidf_tuned  0.97135  0.97227       0.97144  \n",
       "submission_lr_wrd2v_tuned   0.9524  0.94993       0.95215  \n",
       "submission_lr_wrd2v        0.95233  0.94982       0.95208  \n",
       "submission_lr_count        0.94845  0.94248       0.94785  \n",
       "submission_lr_count_tuned   0.9145  0.91797       0.91485  "
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_lr_count\": ['Logistic Regression', 'Count Vectors', 'Not Tuned', 0.94845, 0.94248],\n",
    "    \"submission_lr_tfidf\": ['Logistic Regression', 'TF-IDF Vectors', 'Not Tuned', 0.97558, 0.97621], \n",
    "    \"submission_lr_wrd2v\": ['Logistic Regression', 'Word2Vec Vectors', 'Not Tuned', 0.95233, 0.94982],\n",
    "    \"submission_lr_count_tuned\": ['Logistic Regression', 'Count Vectors', 'Tuned', 0.91450, 0.91797],\n",
    "    \"submission_lr_tfidf_tuned\": ['Logistic Regression', 'TF-IDF Vectors', 'Tuned', 0.97135, 0.97227], \n",
    "    \"submission_lr_wrd2v_tuned\": ['Logistic Regression', 'Word2Vec Vectors', 'Tuned', 0.95240, 0.94993]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "727d5546",
   "metadata": {},
   "source": [
    "### Multinomial Naive Bayes <a class=\"anchor\" id=\"mn\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8a9afeb",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "A `MultinomialNB()` object with default parameters is initialized to serve as the base classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "id": "d909b1fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "mn = MultinomialNB()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f4f22c5",
   "metadata": {},
   "source": [
    "The model will first be trained using the Count vectorized train data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "id": "d6b0d90b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting MultinomialNB()...\n",
      "toxic: 0.9513696097661856\n",
      "severe_toxic: 0.98641983819115\n",
      "obscene: 0.9670867513520627\n",
      "threat: 0.9955505699657206\n",
      "insult: 0.9646301646289113\n",
      "identity_hate: 0.9877233331871079\n",
      "\n",
      "Overall training accuracy: 0.975463377848523\n",
      "Wall time: 3.81 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "mn_models_count, predictions_mn_count = train_models(mn, count_train, count_test)\n",
    "to_submission_csv(predictions_mn_count, 'submission_mn_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95ce6858",
   "metadata": {},
   "source": [
    "Then, the model will be trained using the TF-IDF vectorized data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "id": "ba44ba9d",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting MultinomialNB()...\n",
      "toxic: 0.9236828747078103\n",
      "severe_toxic: 0.9899104473870566\n",
      "obscene: 0.9538449968979326\n",
      "threat: 0.996973134216117\n",
      "insult: 0.9535629907689994\n",
      "identity_hate: 0.9911074067343063\n",
      "\n",
      "Overall training accuracy: 0.968180308452037\n",
      "Wall time: 3.37 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "mn_models_tfidf, predictions_mn_tfidf = train_models(mn, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_mn_tfidf, 'submission_mn_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38cdb55a",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "A `MultinomialNB()` object with default parameters is declared for use as the base estimator, to be tuned by the defined [`parameters_mnb`](#param_mn) hyperparameters.\n",
    "\n",
    "The first model to be tuned will be trained on Count vectors as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "2ee76844",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning MultinomialNB()...\n",
      "toxic: 0.9513696097661856 {'alpha': 1, 'fit_prior': True}\n",
      "severe_toxic: 0.9901297854873379 {'alpha': 1000, 'fit_prior': True}\n",
      "obscene: 0.9670867513520627 {'alpha': 1, 'fit_prior': True}\n",
      "threat: 0.9970044682304429 {'alpha': 1000, 'fit_prior': True}\n",
      "insult: 0.9646301646289113 {'alpha': 1, 'fit_prior': True}\n",
      "identity_hate: 0.99123274279161 {'alpha': 1000, 'fit_prior': True}\n",
      "\n",
      "Overall training accuracy: 0.9769089203760917\n",
      "Wall time: 36.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "mn = MultinomialNB()\n",
    "mn_models_count_tuned, predictions_mn_count_tuned = tune_and_train_models(mn, parameters_mnb, count_train, count_test)\n",
    "to_submission_csv(predictions_mn_count_tuned, 'submission_mn_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe1a83a0",
   "metadata": {},
   "source": [
    "Lastly, the model will be tuned and trained using TF-IDF vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "ecf109d1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning MultinomialNB()...\n",
      "toxic: 0.9617975697338489 {'alpha': 0.1, 'fit_prior': True}\n",
      "severe_toxic: 0.9900044494300343 {'alpha': 10, 'fit_prior': True}\n",
      "obscene: 0.9767501613701738 {'alpha': 0.1, 'fit_prior': True}\n",
      "threat: 0.9970044682304429 {'alpha': 10, 'fit_prior': True}\n",
      "insult: 0.9738611652493248 {'alpha': 0.1, 'fit_prior': True}\n",
      "identity_hate: 0.9911951419744189 {'alpha': 10, 'fit_prior': True}\n",
      "\n",
      "Overall training accuracy: 0.9817688259980407\n",
      "Wall time: 23.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "mn = MultinomialNB()\n",
    "mn_models_tfidf_tuned, predictions_mn_tfidf_tuned = tune_and_train_models(mn, parameters_mnb, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_mn_tfidf_tuned, 'submission_mn_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6be91b0a",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "As seen from the scores returned by Kaggle, the  Naive Bayes models yielded relatively lower accuracy scores compared to Logistic Regression models. Of the four models, the untuned Count vector model scored the best at 0.84654."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "id": "4b875569",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_mn_count</th>\n",
       "      <td>Multinomial Naive Bayes</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.84551</td>\n",
       "      <td>0.85581</td>\n",
       "      <td>0.84654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mn_tfidf_tuned</th>\n",
       "      <td>Multinomial Naive Bayes</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.8293</td>\n",
       "      <td>0.83952</td>\n",
       "      <td>0.83032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mn_tfidf</th>\n",
       "      <td>Multinomial Naive Bayes</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.8251</td>\n",
       "      <td>0.83586</td>\n",
       "      <td>0.82618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mn_count_tuned</th>\n",
       "      <td>Multinomial Naive Bayes</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.75966</td>\n",
       "      <td>0.76208</td>\n",
       "      <td>0.7599</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             model          vector  \\\n",
       "submission_mn_count        Multinomial Naive Bayes   Count Vectors   \n",
       "submission_mn_tfidf_tuned  Multinomial Naive Bayes  TF-IDF Vectors   \n",
       "submission_mn_tfidf        Multinomial Naive Bayes  TF-IDF Vectors   \n",
       "submission_mn_count_tuned  Multinomial Naive Bayes   Count Vectors   \n",
       "\n",
       "                                tuned  private   public test accuracy  \n",
       "submission_mn_count         Not Tuned  0.84551  0.85581       0.84654  \n",
       "submission_mn_tfidf_tuned       Tuned   0.8293  0.83952       0.83032  \n",
       "submission_mn_tfidf         Not Tuned   0.8251  0.83586       0.82618  \n",
       "submission_mn_count_tuned       Tuned  0.75966  0.76208        0.7599  "
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_mn_count\": ['Multinomial Naive Bayes', 'Count Vectors', ' Not Tuned', 0.84551, 0.85581],\n",
    "    \"submission_mn_tfidf\": ['Multinomial Naive Bayes', 'TF-IDF Vectors', 'Not Tuned', 0.82510, 0.83586],\n",
    "    \"submission_mn_count_tuned\": ['Multinomial Naive Bayes', 'Count Vectors', 'Tuned', 0.75966, 0.76208],\n",
    "    \"submission_mn_tfidf_tuned\": ['Multinomial Naive Bayes', 'TF-IDF Vectors', 'Tuned', 0.82930, 0.83952]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b008b957",
   "metadata": {},
   "source": [
    "### RandomForestClassifier <a class=\"anchor\" id=\"rf\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc9f95b7",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "For the `RandomForestClassifier()` object, default parameters were used aside from the `max_depth` which is set to 1000, and the `max_leaf_nodes` which is set to 100. This was done in consideration of the lengthy execution time when training this type of classifier, especially when conducting hyperparameter tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "id": "ca166526",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(max_depth=1000, max_leaf_nodes=100, n_jobs=-1, class_weight='balanced', random_state=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88bdbaca",
   "metadata": {},
   "source": [
    "The models will first be trained using Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "id": "109a6cdf",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting RandomForestClassifier(class_weight='balanced', max_depth=1000,\n",
      "                       max_leaf_nodes=100, n_jobs=-1, random_state=8)...\n",
      "toxic: 0.7749403087027091\n",
      "severe_toxic: 0.8696442336013436\n",
      "obscene: 0.7857192096308226\n",
      "threat: 0.9530303125254589\n",
      "insult: 0.7856565416021708\n",
      "identity_hate: 0.8484373727055669\n",
      "\n",
      "Overall training accuracy: 0.8362379964613452\n",
      "Wall time: 1min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf_models_count, predictions_rf_count = train_models(rf, count_train, count_test)\n",
    "to_submission_csv(predictions_rf_count, 'submission_rf_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b786abb",
   "metadata": {},
   "source": [
    "Afterwards, the models will be trained using TF-IDF vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "id": "a44162ee",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting RandomForestClassifier(class_weight='balanced', max_depth=1000,\n",
      "                       max_leaf_nodes=100, n_jobs=-1, random_state=8)...\n",
      "toxic: 0.7776726347519286\n",
      "severe_toxic: 0.881933434019966\n",
      "obscene: 0.7912715969693741\n",
      "threat: 0.9599049952685639\n",
      "insult: 0.7929824341515689\n",
      "identity_hate: 0.8813944889735603\n",
      "\n",
      "Overall training accuracy: 0.8475265973558269\n",
      "Wall time: 1min 12s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf_models_tfidf, predictions_rf_tfidf = train_models(rf, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_rf_tfidf, 'submission_rf_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "621b1408",
   "metadata": {},
   "source": [
    "Lastly, the Word2Vec vectors will also be used to train the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "id": "9bc8682b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting RandomForestClassifier(class_weight='balanced', max_depth=1000,\n",
      "                       max_leaf_nodes=100, n_jobs=-1, random_state=8)...\n",
      "toxic: 0.8827293179838441\n",
      "severe_toxic: 0.965889792004813\n",
      "obscene: 0.9096264358812065\n",
      "threat: 0.9884440155166039\n",
      "insult: 0.9103345846049721\n",
      "identity_hate: 0.9449336032236434\n",
      "\n",
      "Overall training accuracy: 0.9336596248691804\n",
      "Wall time: 5min 38s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf_models_wrd2v, predictions_rf_wrd2v = train_models(rf, wrd2v_train, wrd2v_test)\n",
    "to_submission_csv(predictions_rf_wrd2v, 'submission_rf_wrd2v')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dbd252b",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "A RandomForestClassifier() object with a `max_depth` of 1000 and a `max_leaf_nodes` of 100 is initialized as the base estimator, to be tuned by the `parameters_rf` hyperparameters. This model will be tuned using Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "id": "22f77194",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning RandomForestClassifier(class_weight='balanced', max_depth=1000,\n",
      "                       max_leaf_nodes=100, n_jobs=-1, random_state=8)...\n",
      "toxic: 0.7749403087027091 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "severe_toxic: 0.8686352783400493 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "obscene: 0.7857192096308226 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "threat: 0.9531305813713018 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "insult: 0.7856565416021708 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "identity_hate: 0.8497346008986595 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "\n",
      "Overall training accuracy: 0.8363027534242855\n",
      "Wall time: 26min 23s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier(max_depth=1000, max_leaf_nodes=100, n_jobs=-1, class_weight='balanced', random_state=8)\n",
    "rf_models_count_tuned, predictions_rf_count_tuned = tune_and_train_models(rf, parameters_rf, count_train, count_test)\n",
    "to_submission_csv(predictions_rf_count_tuned, 'submission_rf_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6160aef8",
   "metadata": {},
   "source": [
    "After this, the model using TF-IDF vectors will be tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "id": "9c7b1eea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning RandomForestClassifier(class_weight='balanced', max_depth=1000,\n",
      "                       max_leaf_nodes=100, n_jobs=-1, random_state=8)...\n",
      "toxic: 0.7771399565083881 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "severe_toxic: 0.8836004035821046 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "obscene: 0.7912715969693741 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "threat: 0.9599049952685639 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "insult: 0.7929824341515689 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "identity_hate: 0.8796335173684442 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "\n",
      "Overall training accuracy: 0.8474221506414072\n",
      "Wall time: 26min 53s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier(max_depth=1000, max_leaf_nodes=100, n_jobs=-1, class_weight='balanced', random_state=8)\n",
    "rf_models_tfidf_tuned, predictions_rf_tfidf_tuned = tune_and_train_models(rf, parameters_rf, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_rf_tfidf_tuned, 'submission_rf_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a179825",
   "metadata": {},
   "source": [
    "Lastly, the Word2Vec-trained model will be tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "id": "0aa0fb38",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning RandomForestClassifier(class_weight='balanced', max_depth=1000,\n",
      "                       max_leaf_nodes=100, n_jobs=-1, random_state=8)...\n",
      "toxic: 0.8827293179838441 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "severe_toxic: 0.965889792004813 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "obscene: 0.9096264358812065 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 100}\n",
      "threat: 0.9886132191939638 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "insult: 0.9104285866479498 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "identity_hate: 0.9453284118041498 {'max_depth': 1000, 'max_features': 'sqrt', 'max_leaf_nodes': 100, 'n_estimators': 200}\n",
      "\n",
      "Overall training accuracy: 0.9337692939193212\n",
      "Wall time: 28min 52s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier(max_depth=1000, max_leaf_nodes=100, n_jobs=-1, class_weight='balanced', random_state=8)\n",
    "rf_models_wrd2v_tuned, predictions_rf_wrd2v_tuned = tune_and_train_models(rf, parameters_rf, wrd2v_train, wrd2v_test)\n",
    "to_submission_csv(predictions_rf_wrd2v_tuned, 'submission_rf_wrd2v_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19310dc6",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "As seen from the scores returned by Kaggle, the Random Forest models returned better results than Naive Bayes, but are still inferior to Logistic Regression models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "id": "ae086a1c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_rf_wrd2v_tuned</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.94065</td>\n",
       "      <td>0.94117</td>\n",
       "      <td>0.9407</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_wrd2v</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.9387</td>\n",
       "      <td>0.94112</td>\n",
       "      <td>0.93894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_count_tuned</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.90325</td>\n",
       "      <td>0.90111</td>\n",
       "      <td>0.90304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_count</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.90227</td>\n",
       "      <td>0.90004</td>\n",
       "      <td>0.90205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_tfidf_tuned</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.89725</td>\n",
       "      <td>0.89771</td>\n",
       "      <td>0.8973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_tfidf</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.89711</td>\n",
       "      <td>0.89752</td>\n",
       "      <td>0.89715</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             model            vector  \\\n",
       "submission_rf_wrd2v_tuned  Random Forest Classifer  Word2Vec Vectors   \n",
       "submission_rf_wrd2v        Random Forest Classifer  Word2Vec Vectors   \n",
       "submission_rf_count_tuned  Random Forest Classifer     Count Vectors   \n",
       "submission_rf_count        Random Forest Classifer     Count Vectors   \n",
       "submission_rf_tfidf_tuned  Random Forest Classifer    TF-IDF Vectors   \n",
       "submission_rf_tfidf        Random Forest Classifer    TF-IDF Vectors   \n",
       "\n",
       "                               tuned  private   public test accuracy  \n",
       "submission_rf_wrd2v_tuned      Tuned  0.94065  0.94117        0.9407  \n",
       "submission_rf_wrd2v        Not Tuned   0.9387  0.94112       0.93894  \n",
       "submission_rf_count_tuned      Tuned  0.90325  0.90111       0.90304  \n",
       "submission_rf_count        Not Tuned  0.90227  0.90004       0.90205  \n",
       "submission_rf_tfidf_tuned      Tuned  0.89725  0.89771        0.8973  \n",
       "submission_rf_tfidf        Not Tuned  0.89711  0.89752       0.89715  "
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_rf_count\": ['Random Forest Classifer', 'Count Vectors', 'Not Tuned', 0.90227, 0.90004],\n",
    "    \"submission_rf_tfidf\": ['Random Forest Classifer', 'TF-IDF Vectors', 'Not Tuned', 0.89711, 0.89752],\n",
    "    \"submission_rf_wrd2v\": ['Random Forest Classifer', 'Word2Vec Vectors', 'Not Tuned', 0.93870, 0.94112],\n",
    "    \"submission_rf_count_tuned\": ['Random Forest Classifer', 'Count Vectors', 'Tuned', 0.90325, 0.90111],\n",
    "    \"submission_rf_tfidf_tuned\": ['Random Forest Classifer', 'TF-IDF Vectors', 'Tuned', 0.89725, 0.89771],\n",
    "    \"submission_rf_wrd2v_tuned\": ['Random Forest Classifer', 'Word2Vec Vectors', 'Tuned', 0.94065, 0.94117]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9a6d61c",
   "metadata": {},
   "source": [
    "### GradientBoostingClassifier <a class=\"anchor\" id=\"gbc\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1686af7",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "The `GradientBoostingClassifier()` object was initialized with default parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "id": "b74c9b56",
   "metadata": {},
   "outputs": [],
   "source": [
    "gbc = GradientBoostingClassifier(random_state=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a258616b",
   "metadata": {},
   "source": [
    "The model will first be trained using the Count vectorized train data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "id": "ffc659fd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting GradientBoostingClassifier(random_state=8)...\n",
      "toxic: 0.943373169310213\n",
      "severe_toxic: 0.9916902194007683\n",
      "obscene: 0.9760106786320822\n",
      "threat: 0.9974118104166797\n",
      "insult: 0.968609584448302\n",
      "identity_hate: 0.9942094741525715\n",
      "\n",
      "Overall training accuracy: 0.9785508227267693\n",
      "Wall time: 17min 33s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gbc_models_count, predictions_gbc_count = train_models(gbc, count_train, count_test)\n",
    "to_submission_csv(predictions_gbc_count, 'submission_gbc_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8e21078",
   "metadata": {},
   "source": [
    "Next, the model will be trained using the TF-IDF vectorized data as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "id": "36ca0411",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting GradientBoostingClassifier(random_state=8)...\n",
      "toxic: 0.9443382569514511\n",
      "severe_toxic: 0.9923106328844213\n",
      "obscene: 0.9768378966102863\n",
      "threat: 0.9978191526029165\n",
      "insult: 0.9696498737239223\n",
      "identity_hate: 0.9950366921307756\n",
      "\n",
      "Overall training accuracy: 0.9793320841506289\n",
      "Wall time: 38min 23s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gbc_models_tfidf, predictions_gbc_tfidf = train_models(gbc, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_gbc_tfidf, 'submission_gbc_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dfe9431",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "As seen from the scores returned by Kaggle, the GradientBoostingClassifier models performed decently well, though inferior to some of the previous models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "9446f990",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_gbc_count</th>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.92562</td>\n",
       "      <td>0.93158</td>\n",
       "      <td>0.92622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_gbc_tfidf</th>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.9049</td>\n",
       "      <td>0.91988</td>\n",
       "      <td>0.9064</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           model          vector      tuned  \\\n",
       "submission_gbc_count  GradientBoostingClassifier   Count Vectors  Not Tuned   \n",
       "submission_gbc_tfidf  GradientBoostingClassifier  TF-IDF Vectors  Not Tuned   \n",
       "\n",
       "                      private   public test accuracy  \n",
       "submission_gbc_count  0.92562  0.93158       0.92622  \n",
       "submission_gbc_tfidf   0.9049  0.91988        0.9064  "
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_gbc_count\": ['GradientBoostingClassifier', 'Count Vectors', 'Not Tuned', 0.92562, 0.93158],\n",
    "    \"submission_gbc_tfidf\": ['GradientBoostingClassifier', 'TF-IDF Vectors', 'Not Tuned', 0.90490, 0.91988]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d06f4fd",
   "metadata": {},
   "source": [
    "### XGBClassifier <a class=\"anchor\" id=\"xgb\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0250de73",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "For the XGBClassifier, the `objective` parameter was set to binary:logistic, which is a learning objective for binary classification that uses logistic regression, and the `eval_metric` was set to auc which refers to the Area Under the ROC Convex Hull. This model was first trained on the Count-vectorized dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "id": "cf5261ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting XGBClassifier(base_score=None, booster=None, colsample_bylevel=None,\n",
      "              colsample_bynode=None, colsample_bytree=None,\n",
      "              enable_categorical=False, eval_metric='auc', gamma=None,\n",
      "              gpu_id=None, importance_type=None, interaction_constraints=None,\n",
      "              learning_rate=None, max_delta_step=None, max_depth=None,\n",
      "              min_child_weight=None, missing=nan, monotone_constraints=None,\n",
      "              n_estimators=100, n_jobs=None, num_parallel_tree=None,\n",
      "              predictor=None, random_state=None, reg_alpha=None,\n",
      "              reg_lambda=None, scale_pos_weight=None, subsample=None,\n",
      "              tree_method=None, use_label_encoder=False,\n",
      "              validate_parameters=None, verbosity=0)...\n",
      "toxic: 0.9643732257114388\n",
      "severe_toxic: 0.9943536106184708\n",
      "obscene: 0.9863947709796893\n",
      "threat: 0.9989910447387057\n",
      "insult: 0.9798772959998997\n",
      "identity_hate: 0.9955568367685858\n",
      "\n",
      "Overall training accuracy: 0.9865911308027983\n",
      "Wall time: 2min 33s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "xgb = xgboost.XGBClassifier(objective=\"binary:logistic\", eval_metric='auc', verbosity=0, use_label_encoder=False)\n",
    "xgb_models_count, predictions_xgb_count = train_models(xgb, count_train, count_test)\n",
    "to_submission_csv(predictions_xgb_count, 'submission_xgb_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d403eb1",
   "metadata": {},
   "source": [
    "Next, this was trained on the TF-IDF vectorized data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "id": "c2260e57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting XGBClassifier(base_score=None, booster=None, colsample_bylevel=None,\n",
      "              colsample_bynode=None, colsample_bytree=None,\n",
      "              enable_categorical=False, eval_metric='auc', gamma=None,\n",
      "              gpu_id=None, importance_type=None, interaction_constraints=None,\n",
      "              learning_rate=None, max_delta_step=None, max_depth=None,\n",
      "              min_child_weight=None, missing=nan, monotone_constraints=None,\n",
      "              n_estimators=100, n_jobs=None, num_parallel_tree=None,\n",
      "              predictor=None, random_state=None, reg_alpha=None,\n",
      "              reg_lambda=None, scale_pos_weight=None, subsample=None,\n",
      "              tree_method=None, use_label_encoder=False,\n",
      "              validate_parameters=None, verbosity=0)...\n",
      "toxic: 0.9667358103916125\n",
      "severe_toxic: 0.9954127003026866\n",
      "obscene: 0.9876167975383998\n",
      "threat: 0.9993482525020211\n",
      "insult: 0.9813186606588916\n",
      "identity_hate: 0.9959391117433619\n",
      "\n",
      "Overall training accuracy: 0.9877285555228289\n",
      "Wall time: 8min 9s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "xgb = xgboost.XGBClassifier(objective=\"binary:logistic\", eval_metric='auc', verbosity=0, use_label_encoder=False)\n",
    "xgb_models_tfidf, predictions_xgb_tfidf = train_models(xgb, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_xgb_tfidf, 'submission_xgb_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47aebec2",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "A XGBClassifier() object with the parameters stated earlier was initialized to be tuned with the `parameters_xgb` hyperparameters. As with the previous models, the model will first be tuned and trained with the Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92486643",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "xgb = xgboost.XGBClassifier(objective=\"binary:logistic\", eval_metric='auc', verbosity=0, use_label_encoder=False)\n",
    "xgb_models_count_tuned, predictions_xgb_count_tuned = tune_and_train_models(xgb, parameters_xgb, count_train, count_test)\n",
    "to_submission_csv(predictions_xgb_count_tuned, 'submission_xgb_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0979994a",
   "metadata": {},
   "source": [
    "Then lastly, the model will be tuned and trained on the TF-IDF vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6d71ab1",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "xgb = xgboost.XGBClassifier(objective=\"binary:logistic\", eval_metric='auc', verbosity=0, use_label_encoder=False)\n",
    "xgb_models_tfidf_tuned, predictions_xgb_tfidf_tuned = tune_and_train_models(xgb, parameters_xgb, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_xgb_tfidf_tuned, 'submission_xgb_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a05d50a",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "As seen from the scores returned by Kaggle, the XGBClassifier models yielded high accuracy scores. While not the model with the highest test accuracy score, this model has the potential to perform better if a more thorough hyperparameter tuning is conducted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "id": "26641385",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_xgb_tfidf</th>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.96502</td>\n",
       "      <td>0.96803</td>\n",
       "      <td>0.96532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_xgb_count</th>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.96468</td>\n",
       "      <td>0.96783</td>\n",
       "      <td>0.96499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_xgb_tfidf_tuned</th>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.96396</td>\n",
       "      <td>0.96693</td>\n",
       "      <td>0.96426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_xgb_count_tuned</th>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.96282</td>\n",
       "      <td>0.96673</td>\n",
       "      <td>0.96321</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    model          vector      tuned  private  \\\n",
       "submission_xgb_tfidf        XGBClassifier  TF-IDF Vectors  Not Tuned  0.96502   \n",
       "submission_xgb_count        XGBClassifier   Count Vectors  Not Tuned  0.96468   \n",
       "submission_xgb_tfidf_tuned  XGBClassifier  TF-IDF Vectors      Tuned  0.96396   \n",
       "submission_xgb_count_tuned  XGBClassifier   Count Vectors      Tuned  0.96282   \n",
       "\n",
       "                             public test accuracy  \n",
       "submission_xgb_tfidf        0.96803       0.96532  \n",
       "submission_xgb_count        0.96783       0.96499  \n",
       "submission_xgb_tfidf_tuned  0.96693       0.96426  \n",
       "submission_xgb_count_tuned  0.96673       0.96321  "
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_xgb_count\": ['XGBClassifier', 'Count Vectors', 'Not Tuned', 0.96468, 0.96783],\n",
    "    \"submission_xgb_tfidf\": ['XGBClassifier', 'TF-IDF Vectors', 'Not Tuned', 0.96502, 0.96803],\n",
    "    \"submission_xgb_count_tuned\": ['XGBClassifier', 'Count Vectors', 'Tuned', 0.96282, 0.96673],\n",
    "    \"submission_xgb_tfidf_tuned\": ['XGBClassifier', 'TF-IDF Vectors', 'Tuned', 0.96396, 0.96693],\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "096ea054",
   "metadata": {},
   "source": [
    "### AdaBoostClassifier <a class=\"anchor\" id=\"adb\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41db3139",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "An AdaBoostClassifier is initialized with the default parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "b44e1000",
   "metadata": {},
   "outputs": [],
   "source": [
    "adb = AdaBoostClassifier(random_state=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "620d8d00",
   "metadata": {},
   "source": [
    "This model is trained with the Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "id": "d0253f5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting AdaBoostClassifier(random_state=8)...\n",
      "toxic: 0.9454349474528579\n",
      "severe_toxic: 0.9896159076523929\n",
      "obscene: 0.9711977740316223\n",
      "threat: 0.9969480670046562\n",
      "insult: 0.9659211260191388\n",
      "identity_hate: 0.9915084821176781\n",
      "\n",
      "Overall training accuracy: 0.9767710507130577\n",
      "Wall time: 4min 43s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "adb_models_count, predictions_adb_count = train_models(adb, count_train, count_test)\n",
    "to_submission_csv(predictions_adb_count, 'submission_adb_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "394055a7",
   "metadata": {},
   "source": [
    "Then, the model is trained on the TF-IDF vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "id": "85a88422",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting AdaBoostClassifier(random_state=8)...\n",
      "toxic: 0.9480544710505041\n",
      "severe_toxic: 0.9899981826271691\n",
      "obscene: 0.9749139881306754\n",
      "threat: 0.9969606006103866\n",
      "insult: 0.9679327697388623\n",
      "identity_hate: 0.9917278202179594\n",
      "\n",
      "Overall training accuracy: 0.9782646387292595\n",
      "Wall time: 9min 50s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "adb_models_tfidf, predictions_adb_tfidf = train_models(adb, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_adb_tfidf, 'submission_adb_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e69c9a1d",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "As seen from the scores returned by Kaggle, the AdaBoostClassifier models yielded accuracy scores of about 0.93, which makes these models one of the more decently-performing models in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "id": "969543b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_adb_tfidf</th>\n",
       "      <td>AdaBoostClassifier</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.9383</td>\n",
       "      <td>0.94145</td>\n",
       "      <td>0.93862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_adb_count</th>\n",
       "      <td>AdaBoostClassifier</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.93539</td>\n",
       "      <td>0.94218</td>\n",
       "      <td>0.93607</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   model          vector      tuned  private  \\\n",
       "submission_adb_tfidf  AdaBoostClassifier  TF-IDF Vectors  Not Tuned   0.9383   \n",
       "submission_adb_count  AdaBoostClassifier   Count Vectors  Not Tuned  0.93539   \n",
       "\n",
       "                       public test accuracy  \n",
       "submission_adb_tfidf  0.94145       0.93862  \n",
       "submission_adb_count  0.94218       0.93607  "
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_adb_count\": ['AdaBoostClassifier', 'Count Vectors', 'Not Tuned', 0.93539, 0.94218],\n",
    "    \"submission_adb_tfidf\": ['AdaBoostClassifier', 'TF-IDF Vectors', 'Not Tuned', 0.93830, 0.94145],\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9803b2a5",
   "metadata": {},
   "source": [
    "### Stochastic Gradient Descent Classifier <a class=\"anchor\" id=\"sgd\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "004b3421",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "An SGDClassifier was initialized with a modified_huber loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "be11a3af",
   "metadata": {},
   "outputs": [],
   "source": [
    "sgd = SGDClassifier(loss='modified_huber', class_weight='balanced', n_jobs=-1, random_state=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce75afa0",
   "metadata": {},
   "source": [
    "The model will first be trained with Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "335d0c51",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting SGDClassifier(class_weight='balanced', loss='modified_huber', n_jobs=-1,\n",
      "              random_state=8)...\n",
      "toxic: 0.9740930369553364\n",
      "severe_toxic: 0.9773141736280402\n",
      "obscene: 0.9366551566387377\n",
      "threat: 0.9831673675041204\n",
      "insult: 0.8818895663999098\n",
      "identity_hate: 0.961321292716095\n",
      "\n",
      "Overall training accuracy: 0.9524067656403732\n",
      "Wall time: 11 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "sgd_models_count, predictions_sgd_count = train_models(sgd, count_train, count_test)\n",
    "to_submission_csv(predictions_sgd_count, 'submission_sgd_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed6f5a89",
   "metadata": {},
   "source": [
    "Then, TF-IDF vectors will be used as so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "cf693ba9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting SGDClassifier(class_weight='balanced', loss='modified_huber', n_jobs=-1,\n",
      "              random_state=8)...\n",
      "toxic: 0.9546032800446196\n",
      "severe_toxic: 0.9773705748538268\n",
      "obscene: 0.9798020943655176\n",
      "threat: 0.9951808285966748\n",
      "insult: 0.9667295435887473\n",
      "identity_hate: 0.9792819497277074\n",
      "\n",
      "Overall training accuracy: 0.9754947118628489\n",
      "Wall time: 5.58 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "sgd_models_tfidf, predictions_sgd_tfidf = train_models(sgd, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_sgd_tfidf, 'submission_sgd_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26b2ce66",
   "metadata": {},
   "source": [
    "Lastly, Word2Vec vectors will be used in training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "b2378521",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting SGDClassifier(class_weight='balanced', loss='modified_huber', n_jobs=-1,\n",
      "              random_state=8)...\n",
      "toxic: 0.9183560922724053\n",
      "severe_toxic: 0.9547912841305751\n",
      "obscene: 0.9135494544748106\n",
      "threat: 0.974863853707754\n",
      "insult: 0.9419694054684122\n",
      "identity_hate: 0.9325002663391218\n",
      "\n",
      "Overall training accuracy: 0.9393383927321798\n",
      "Wall time: 39.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "sgd_models_wrd2v, predictions_sgd_wrd2v = train_models(sgd, wrd2v_train, wrd2v_test)\n",
    "to_submission_csv(predictions_sgd_wrd2v, 'submission_sgd_wrd2v')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "689fd2aa",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "The model would be tuned using a GridSearchCV with `parameters_sgd`. The model will first be tuned with Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "id": "5c1c0175",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning SGDClassifier(class_weight='balanced', loss='modified_huber', n_jobs=-1,\n",
      "              random_state=8)...\n",
      "toxic: 0.9524600334647273 {'alpha': 0.0001, 'loss': 'log'}\n",
      "severe_toxic: 0.9767438945673086 {'alpha': 0.0001, 'loss': 'log'}\n",
      "obscene: 0.9366551566387377 {'alpha': 0.0001, 'loss': 'modified_huber'}\n",
      "threat: 0.9797770271540568 {'alpha': 0.0001, 'loss': 'log'}\n",
      "insult: 0.8818895663999098 {'alpha': 0.0001, 'loss': 'modified_huber'}\n",
      "identity_hate: 0.9657769895532397 {'alpha': 100, 'loss': 'log'}\n",
      "\n",
      "Overall training accuracy: 0.9488837779629966\n",
      "Wall time: 3min 56s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "sgd = SGDClassifier(loss='modified_huber', class_weight='balanced', n_jobs=-1, random_state=8)\n",
    "sgd_models_count_tuned, predictions_sgd_count_tuned = tune_and_train_models(sgd, parameters_sgd, count_train, count_test)\n",
    "to_submission_csv(predictions_sgd_count_tuned, 'submission_sgd_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87fa206d",
   "metadata": {},
   "source": [
    "Next, it will be tuned with TF-IDF vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "id": "17f763d8",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning SGDClassifier(class_weight='balanced', loss='modified_huber', n_jobs=-1,\n",
      "              random_state=8)...\n",
      "toxic: 0.9546032800446196 {'alpha': 0.0001, 'loss': 'modified_huber'}\n",
      "severe_toxic: 0.9773705748538268 {'alpha': 0.0001, 'loss': 'modified_huber'}\n",
      "obscene: 0.9798020943655176 {'alpha': 0.0001, 'loss': 'modified_huber'}\n",
      "threat: 0.9962775190980817 {'alpha': 1, 'loss': 'modified_huber'}\n",
      "insult: 0.9667295435887473 {'alpha': 0.0001, 'loss': 'modified_huber'}\n",
      "identity_hate: 0.9911951419744189 {'alpha': 100, 'loss': 'log'}\n",
      "\n",
      "Overall training accuracy: 0.977663025654202\n",
      "Wall time: 1min 25s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "sgd = SGDClassifier(loss='modified_huber', class_weight='balanced', n_jobs=-1, random_state=8)\n",
    "sgd_models_tfidf_tuned, predictions_sgd_tfidf_tuned = tune_and_train_models(sgd, parameters_sgd, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_sgd_tfidf_tuned, 'submission_sgd_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7be991f",
   "metadata": {},
   "source": [
    "Lastly, Word2Vec vectors were also tried."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "id": "0336ecc8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning SGDClassifier(class_weight='balanced', loss='modified_huber', n_jobs=-1,\n",
      "              random_state=8)...\n",
      "toxic: 0.8969298932763472 {'alpha': 0.001, 'loss': 'modified_huber'}\n",
      "severe_toxic: 0.9563642516497358 {'alpha': 0.0001, 'loss': 'log'}\n",
      "obscene: 0.9164259169899293 {'alpha': 0.01, 'loss': 'modified_huber'}\n",
      "threat: 0.9961396494350477 {'alpha': 100, 'loss': 'log'}\n",
      "insult: 0.9419694054684122 {'alpha': 0.0001, 'loss': 'modified_huber'}\n",
      "identity_hate: 0.9910572723113849 {'alpha': 100, 'loss': 'log'}\n",
      "\n",
      "Overall training accuracy: 0.9498143981884762\n",
      "Wall time: 8min 35s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "sgd = SGDClassifier(loss='modified_huber', class_weight='balanced', n_jobs=-1, random_state=8)\n",
    "sgd_models_wrd2v_tuned, predictions_sgd_wrd2v_tuned = tune_and_train_models(sgd, parameters_sgd, wrd2v_train, wrd2v_test)\n",
    "to_submission_csv(predictions_sgd_wrd2v_tuned, 'submission_sgd_wrd2v_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20213e5e",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "As seen from the scores returned by Kaggle, the predictions made by the SGDClassifer models resulted in one of the best-performing models so far, with the TF-IDF vectors at 0.96901."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "id": "78681a0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_sgd_tfidf</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.96854</td>\n",
       "      <td>0.97321</td>\n",
       "      <td>0.96901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_tfidf_tuned</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.96312</td>\n",
       "      <td>0.96252</td>\n",
       "      <td>0.96306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_wrd2v_tuned</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.93585</td>\n",
       "      <td>0.93342</td>\n",
       "      <td>0.93561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_wrd2v</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.91181</td>\n",
       "      <td>0.9156</td>\n",
       "      <td>0.91219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_count_tuned</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.89659</td>\n",
       "      <td>0.8979</td>\n",
       "      <td>0.89672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_count</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.88323</td>\n",
       "      <td>0.88505</td>\n",
       "      <td>0.88341</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   model            vector      tuned  \\\n",
       "submission_sgd_tfidf        SGDClassifer    TF-IDF Vectors  Not Tuned   \n",
       "submission_sgd_tfidf_tuned  SGDClassifer    TF-IDF Vectors      Tuned   \n",
       "submission_sgd_wrd2v_tuned  SGDClassifer  Word2Vec Vectors      Tuned   \n",
       "submission_sgd_wrd2v        SGDClassifer  Word2Vec Vectors  Not Tuned   \n",
       "submission_sgd_count_tuned  SGDClassifer     Count Vectors      Tuned   \n",
       "submission_sgd_count        SGDClassifer     Count Vectors  Not Tuned   \n",
       "\n",
       "                            private   public test accuracy  \n",
       "submission_sgd_tfidf        0.96854  0.97321       0.96901  \n",
       "submission_sgd_tfidf_tuned  0.96312  0.96252       0.96306  \n",
       "submission_sgd_wrd2v_tuned  0.93585  0.93342       0.93561  \n",
       "submission_sgd_wrd2v        0.91181   0.9156       0.91219  \n",
       "submission_sgd_count_tuned  0.89659   0.8979       0.89672  \n",
       "submission_sgd_count        0.88323  0.88505       0.88341  "
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_sgd_count\": ['SGDClassifer', 'Count Vectors', 'Not Tuned', 0.88323, 0.88505],\n",
    "    \"submission_sgd_tfidf\": ['SGDClassifer', 'TF-IDF Vectors', 'Not Tuned', 0.96854, 0.97321],\n",
    "    \"submission_sgd_wrd2v\": ['SGDClassifer', 'Word2Vec Vectors', 'Not Tuned', 0.91181, 0.91560],\n",
    "    \"submission_sgd_count_tuned\": ['SGDClassifer', 'Count Vectors', 'Tuned', 0.89659, 0.89790],\n",
    "    \"submission_sgd_tfidf_tuned\": ['SGDClassifer', 'TF-IDF Vectors', 'Tuned', 0.96312, 0.96252],\n",
    "    \"submission_sgd_wrd2v_tuned\": ['SGDClassifer', 'Word2Vec Vectors', 'Tuned', 0.93585, 0.93342]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325f8f56",
   "metadata": {},
   "source": [
    "### OneVsRest Classifier: Logistic Regression <a class=\"anchor\" id=\"oc_lr\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "571180e0",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "First, the OneVsRestClassifier is initialized with the base estimator set as a Logistic Regression model, which used the default values of the parameters, except for the `class_weight`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "id": "590aedce",
   "metadata": {},
   "outputs": [],
   "source": [
    "oc_lr = OneVsRestClassifier(LogisticRegression(class_weight='balanced'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30b7456f",
   "metadata": {},
   "source": [
    "The classifier is then trained using Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "id": "512f4a9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting OneVsRestClassifier(estimator=LogisticRegression(class_weight='balanced'))...\n",
      "0.8936210213635309\n",
      "Wall time: 54.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oc_lr_model_count, predictions_oc_lr_count = train_model(oc_lr, count_train, count_test)\n",
    "to_submission_csv(predictions_oc_lr_count, 'submission_oc_lr_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff448103",
   "metadata": {},
   "source": [
    "After this, the classifier will be trained with TF-IDF vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "id": "548874d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting OneVsRestClassifier(estimator=LogisticRegression(class_weight='balanced'))...\n",
      "0.9042307186142845\n",
      "Wall time: 48.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oc_lr_model_tfidf, predictions_oc_lr_tfidf = train_model(oc_lr, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_oc_lr_tfidf, 'submission_oc_lr_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e9a00c0",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "The model would be tuned using a `GridSearchCV` and the parameter grid **parameters_lr_mo**. Like in the previous models, we would be tuning the model with Count vector as input first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "id": "a6c5bac8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning OneVsRestClassifier(estimator=LogisticRegression(class_weight='balanced'))...\n",
      "0.9377894479573357 {'estimator__C': 1, 'estimator__max_iter': 600}\n",
      "Wall time: 1h 46min 5s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oc_lr = OneVsRestClassifier(LogisticRegression(class_weight='balanced'))\n",
    "oc_lr_model_count_tuned, predictions_oc_lr_count_tuned = tune_and_train_model(oc_lr, \n",
    "                                                                              parameters_lr_mo, \n",
    "                                                                              count_train, \n",
    "                                                                              count_test)\n",
    "to_submission_csv(predictions_oc_lr_count_tuned, 'submission_oc_lr_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c596454",
   "metadata": {},
   "source": [
    "After the Count vector, we would be tuning and predicting using the model that used a TF-IDF vector as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "id": "0fa3fd3b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning OneVsRestClassifier(estimator=LogisticRegression(class_weight='balanced'))...\n",
      "0.9488816890287082 {'estimator__C': 10, 'estimator__max_iter': 300}\n",
      "Wall time: 17min 32s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oc_lr = OneVsRestClassifier(LogisticRegression(class_weight='balanced'))\n",
    "oc_lr_model_tfidf_tuned, predictions_oc_lr_tfidf_tuned = tune_and_train_model(oc_lr, \n",
    "                                                                              parameters_lr_mo, \n",
    "                                                                              tfidf_train, \n",
    "                                                                              tfidf_test)\n",
    "to_submission_csv(predictions_oc_lr_tfidf_tuned, 'submission_oc_lr_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5b8cd5c",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "From the ROC AUC scores of the Kaggle competition, the tuned `OneVsRestClassifier` with a `LogisticRegression` base estimator and a TF-IDF vector input did not improve from its untuned model as the best parameters found was the default values of the `LogisticRegression` model. However, for the Count vectors version, the tuned model actually received a lower score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "id": "1b86c5e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_oc_lr_tfidf</th>\n",
       "      <td>OneVsRestClassifier (Logistic Regression)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_lr_tfidf_tuned</th>\n",
       "      <td>OneVsRestClassifier (Logistic Regression)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_lr_count</th>\n",
       "      <td>OneVsRestClassifier (Logistic Regression)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.93996</td>\n",
       "      <td>0.9441</td>\n",
       "      <td>0.94037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_lr_count_tuned</th>\n",
       "      <td>OneVsRestClassifier (Logistic Regression)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.91488</td>\n",
       "      <td>0.91866</td>\n",
       "      <td>0.91526</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                  model  \\\n",
       "submission_oc_lr_tfidf        OneVsRestClassifier (Logistic Regression)   \n",
       "submission_oc_lr_tfidf_tuned  OneVsRestClassifier (Logistic Regression)   \n",
       "submission_oc_lr_count        OneVsRestClassifier (Logistic Regression)   \n",
       "submission_oc_lr_count_tuned  OneVsRestClassifier (Logistic Regression)   \n",
       "\n",
       "                                      vector      tuned  private   public  \\\n",
       "submission_oc_lr_tfidf        TF-IDF Vectors  Not Tuned  0.97558  0.97621   \n",
       "submission_oc_lr_tfidf_tuned  TF-IDF Vectors      Tuned  0.97558  0.97621   \n",
       "submission_oc_lr_count         Count Vectors  Not Tuned  0.93996   0.9441   \n",
       "submission_oc_lr_count_tuned   Count Vectors      Tuned  0.91488  0.91866   \n",
       "\n",
       "                             test accuracy  \n",
       "submission_oc_lr_tfidf             0.97564  \n",
       "submission_oc_lr_tfidf_tuned       0.97564  \n",
       "submission_oc_lr_count             0.94037  \n",
       "submission_oc_lr_count_tuned       0.91526  "
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_oc_lr_count\": ['OneVsRestClassifier (Logistic Regression)', 'Count Vectors', 'Not Tuned', 0.93996, 0.94410],\n",
    "    \"submission_oc_lr_tfidf\": ['OneVsRestClassifier (Logistic Regression)', 'TF-IDF Vectors', 'Not Tuned', 0.97558, 0.97621],\n",
    "    \"submission_oc_lr_count_tuned\": ['OneVsRestClassifier (Logistic Regression)', 'Count Vectors', 'Tuned', 0.91488, 0.91866],\n",
    "    \"submission_oc_lr_tfidf_tuned\": ['OneVsRestClassifier (Logistic Regression)', 'TF-IDF Vectors', 'Tuned', 0.97558, 0.97621]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06d7f230",
   "metadata": {},
   "source": [
    "### OneVsRest Classifier: Multinomial Naive Bayes <a class=\"anchor\" id=\"oc_mn\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "850ac935",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "To start with training the `MultinomialNB` version of the `OneVsRestClassifier`, we would be create a `MultinomialNB` with the default parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "id": "48168411",
   "metadata": {},
   "outputs": [],
   "source": [
    "oc_mn = OneVsRestClassifier(MultinomialNB())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e240b3e3",
   "metadata": {},
   "source": [
    "This would be first used to train a version of this model using Count vectors as its feature input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "id": "11375a52",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting OneVsRestClassifier(estimator=MultinomialNB())...\n",
      "0.9063050303626599\n",
      "Wall time: 3.57 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oc_mn_model_count, predictions_oc_mn_count = train_model(oc_mn, count_train, count_test)\n",
    "to_submission_csv(predictions_oc_mn_count, 'submission_oc_mn_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2455275f",
   "metadata": {},
   "source": [
    "Afterwards, the a copy of the same `OneVsRestClassifier` object would be trained with a TF-IDF vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "id": "4c26be2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting OneVsRestClassifier(estimator=MultinomialNB())...\n",
      "0.9006335737696699\n",
      "Wall time: 3.24 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oc_mn_model_tfidf, predictions_oc_mn_tfidf = train_model(oc_mn, tfidf_train, tfidf_test)\n",
    "to_submission_csv(predictions_oc_mn_tfidf, 'submission_oc_mn_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de286172",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "These two models of `OneVsRestClassifier` with `MultinomialNB` would be tuned using the hyperparameter values found in **parameters_mn_mo**. We would be starting with the model with Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "id": "6df3099b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning OneVsRestClassifier(estimator=MultinomialNB())...\n",
      "0.9024133457833817 {'estimator__alpha': 10, 'estimator__fit_prior': True}\n",
      "Wall time: 21.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oc_mn = OneVsRestClassifier(MultinomialNB())\n",
    "oc_mn_model_count_tuned, predictions_oc_mn_count_tuned = tune_and_train_model(oc_mn, \n",
    "                                                                              parameters_mn_mo, \n",
    "                                                                              count_train, \n",
    "                                                                              count_test)\n",
    "to_submission_csv(predictions_oc_mn_count_tuned, 'submission_oc_mn_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdf9cbaa",
   "metadata": {},
   "source": [
    "This is followed by the model that used TF-IDF vectors as features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "id": "f4426017",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning OneVsRestClassifier(estimator=MultinomialNB())...\n",
      "0.9281573719535504 {'estimator__alpha': 0.1, 'estimator__fit_prior': True}\n",
      "Wall time: 20.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oc_mn = OneVsRestClassifier(MultinomialNB())\n",
    "oc_mn_model_tfidf_tuned, predictions_oc_mn_tfidf_tuned = tune_and_train_model(oc_mn, \n",
    "                                                                              parameters_mn_mo, \n",
    "                                                                              tfidf_train, \n",
    "                                                                              tfidf_test)\n",
    "to_submission_csv(predictions_oc_mn_tfidf_tuned, 'submission_oc_mn_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb92c18f",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "Analyzing the results of the predictions on the test data, we could see that tuning the `MultinomialNB` of the TF-IDF vector greatly improved its score. Meanwhile, for the model that utilized the Count vector as its input, we could see that it yielded the same score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "id": "e1624f2f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_oc_mn_tfidf_tuned</th>\n",
       "      <td>OneVsRestClassifier (Multinomial NB)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.90597</td>\n",
       "      <td>0.91409</td>\n",
       "      <td>0.90678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_mn_count</th>\n",
       "      <td>OneVsRestClassifier (Multinomial NB)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.84551</td>\n",
       "      <td>0.85581</td>\n",
       "      <td>0.84654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_mn_count_tuned</th>\n",
       "      <td>OneVsRestClassifier (Multinomial NB)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.84551</td>\n",
       "      <td>0.85581</td>\n",
       "      <td>0.84654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_mn_tfidf</th>\n",
       "      <td>OneVsRestClassifier (Multinomial NB)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.8251</td>\n",
       "      <td>0.83586</td>\n",
       "      <td>0.82618</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                             model  \\\n",
       "submission_oc_mn_tfidf_tuned  OneVsRestClassifier (Multinomial NB)   \n",
       "submission_oc_mn_count        OneVsRestClassifier (Multinomial NB)   \n",
       "submission_oc_mn_count_tuned  OneVsRestClassifier (Multinomial NB)   \n",
       "submission_oc_mn_tfidf        OneVsRestClassifier (Multinomial NB)   \n",
       "\n",
       "                                      vector      tuned  private   public  \\\n",
       "submission_oc_mn_tfidf_tuned  TF-IDF Vectors      Tuned  0.90597  0.91409   \n",
       "submission_oc_mn_count         Count Vectors  Not Tuned  0.84551  0.85581   \n",
       "submission_oc_mn_count_tuned   Count Vectors      Tuned  0.84551  0.85581   \n",
       "submission_oc_mn_tfidf        TF-IDF Vectors  Not Tuned   0.8251  0.83586   \n",
       "\n",
       "                             test accuracy  \n",
       "submission_oc_mn_tfidf_tuned       0.90678  \n",
       "submission_oc_mn_count             0.84654  \n",
       "submission_oc_mn_count_tuned       0.84654  \n",
       "submission_oc_mn_tfidf             0.82618  "
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = {\n",
    "    \"submission_oc_mn_count\": ['OneVsRestClassifier (Multinomial NB)', 'Count Vectors', 'Not Tuned', 0.84551, 0.85581],\n",
    "    \"submission_oc_mn_tfidf\": ['OneVsRestClassifier (Multinomial NB)', 'TF-IDF Vectors', 'Not Tuned', 0.82510, 0.83586],\n",
    "    \"submission_oc_mn_count_tuned\": ['OneVsRestClassifier (Multinomial NB)', 'Count Vectors', 'Tuned', 0.84551, 0.85581],\n",
    "    \"submission_oc_mn_tfidf_tuned\": ['OneVsRestClassifier (Multinomial NB)', 'TF-IDF Vectors', 'Tuned', 0.90597, 0.91409]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eec9910",
   "metadata": {},
   "source": [
    "### MultiOutput Classifier: Logistic Regression <a class=\"anchor\" id=\"mo_lr\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "743ea998",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "To start with experimenting with `MultiOutputClassifier`, we will first declare a `MultiOutputClassifier` object with `LogisticRegression` as its base estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "id": "df88d7da",
   "metadata": {},
   "outputs": [],
   "source": [
    "mo_lr = MultiOutputClassifier(LogisticRegression(n_jobs=-1, class_weight='balanced'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bda84047",
   "metadata": {},
   "source": [
    "As we now have this object, we can now create a version of this model that will be trained using an input of Count vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "id": "22f42e90",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting MultiOutputClassifier(estimator=LogisticRegression(class_weight='balanced',\n",
      "                                                   n_jobs=-1))...\n",
      "0.8932199459801593\n",
      "Wall time: 1min 9s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "mo_lr_model_count, predictions_mo_lr_count = train_model(mo_lr, count_train, count_test)\n",
    "to_submission_csv_multiclass(predictions_mo_lr_count, 'submission_mo_lr_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc10c240",
   "metadata": {},
   "source": [
    "Then, another copy of this model would be trained using a TF-IDF vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "id": "f2b3368d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting MultiOutputClassifier(estimator=LogisticRegression(class_weight='balanced',\n",
      "                                                   n_jobs=-1))...\n",
      "0.9042369854171497\n",
      "Wall time: 54.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "mo_lr_model_tfidf, predictions_mo_lr_tfidf = train_model(mo_lr, tfidf_train, tfidf_test)\n",
    "to_submission_csv_multiclass(predictions_mo_lr_tfidf, 'submission_mo_lr_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7a6871d",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "These two models would undergo hyperparameter tuning using **parameters_lr_mo** as their parameter grid and `GridSearchCV`. We will start with tuning the model that used Count vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "id": "ed2728eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning MultiOutputClassifier(estimator=LogisticRegression(class_weight='balanced',\n",
      "                                                   n_jobs=-1))...\n",
      "0.938353460215202 {'estimator__C': 1, 'estimator__max_iter': 600}\n",
      "Wall time: 1h 21min 19s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "mo_lr = MultiOutputClassifier(LogisticRegression(n_jobs=-1, class_weight='balanced'))\n",
    "mo_lr_model_count_tuned, predictions_mo_lr_count_tuned = tune_and_train_model(mo_lr, \n",
    "                                                                              parameters_lr_mo, \n",
    "                                                                              count_train, \n",
    "                                                                              count_test)\n",
    "to_submission_csv_multiclass(predictions_mo_lr_count_tuned, 'submission_mo_lr_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9015339",
   "metadata": {},
   "source": [
    "This is followed by tuning the TF-IDF version of this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b56cf82",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning MultiOutputClassifier(estimator=LogisticRegression(class_weight='balanced',\n",
      "                                                   n_jobs=-1))...\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "mo_lr = MultiOutputClassifier(LogisticRegression(n_jobs=-1, class_weight='balanced'))\n",
    "mo_lr_model_tfidf_tuned, predictions_mo_lr_tfidf_tuned = tune_and_train_model(mo_lr, \n",
    "                                                                              parameters_lr_mo, \n",
    "                                                                              tfidf_train, \n",
    "                                                                              tfidf_test)\n",
    "to_submission_csv_multiclass(predictions_mo_lr_tfidf_tuned, 'submission_mo_lr_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ee7ca61",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "With these results, we can see that the scores between all four classifiers are near each other (with a range of 95 to 97). Although, the difference between the two models was that tuning the TF-IDF version made the score lower, while tuning the Count version increased the score by almost one point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46e78c9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = {\n",
    "    \"submission_mo_lr_count\": ['MultiOutputClassifier (Logistic Regression)', 'Count Vectors', 'Not Tuned', 0.94866, 0.95065],\n",
    "    \"submission_mo_lr_tfidf\": ['MultiOutputClassifier (Logistic Regression)', 'TF-IDF Vectors', 'Not Tuned', 0.97558, 0.97621],\n",
    "    \"submission_mo_lr_count_tuned\": ['MultiOutputClassifier (Logistic Regression)', 'Count Vectors', 'Tuned', 0.95113, 0.95159],\n",
    "    \"submission_mo_lr_tfidf_tuned\": ['MultiOutputClassifier (Logistic Regression)', 'TF-IDF Vectors', 'Tuned', 0.97558, 0.97621]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5b03d3a",
   "metadata": {},
   "source": [
    "### MultiOutput Classifier: Multinomial Naive Bayes <a class=\"anchor\" id=\"mo_mn\"></a><a style=\"float:right; font-size:11px\" href=\"#toc\">Back to Models List</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33eb4b87",
   "metadata": {},
   "source": [
    "#### Model Training\n",
    "For our last approach, we would be creating a `MultiOutputClassifier` that utilized `MultinomialNB` as its base estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e4a0a6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "mo_mn = MultiOutputClassifier(MultinomialNB(), n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e090db2",
   "metadata": {},
   "source": [
    "With this object, we will be training this meta-estimator using a Count vector as its training input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3fb14f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "mo_mn_model_count, predictions_mo_mn_count = train_model(mo_mn, count_train, count_test)\n",
    "to_submission_csv_multiclass(predictions_mo_mn_count, 'submission_mo_mn_count')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d82ff6ff",
   "metadata": {},
   "source": [
    "On the other hand, we will also be training another copy of this meta-estimator, but with the input of a TF-IDF vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "314a0ad5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "mo_mn_model_tfidf, predictions_mo_mn_tfidf = train_model(mo_mn, tfidf_train, tfidf_test)\n",
    "to_submission_csv_multiclass(predictions_mo_mn_tfidf, 'submission_mo_mn_tfidf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "487fd6d2",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "Like in the previous model of `MultiOutputClassifier` (i.e., the one that utilized `LogisticRegression`), we will be tuning the model that used Count vector using `GridSearchCV` and the parameter grid `parameters_mn_mo`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13109edb",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "mo_mn = MultiOutputClassifier(MultinomialNB(), n_jobs=-1)\n",
    "mo_mn_model_count_tuned, predictions_mo_mn_count_tuned = tune_and_train_model(mo_mn, \n",
    "                                                                              parameters_mn_mo, \n",
    "                                                                              count_train, \n",
    "                                                                              count_test)\n",
    "to_submission_csv_multiclass(predictions_mo_mn_count_tuned, 'submission_mo_mn_count_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b530d884",
   "metadata": {},
   "source": [
    "Likewise, we would also be tuning the TF-IDF vector copy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f58b6046",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "mo_mn = MultiOutputClassifier(MultinomialNB(), n_jobs=-1)\n",
    "mo_mn_model_tfidf_tuned, predictions_mo_mn_tfidf_tuned = tune_and_train_model(mo_mn, \n",
    "                                                                              parameters_mn_mo, \n",
    "                                                                              tfidf_train, \n",
    "                                                                              tfidf_test)\n",
    "to_submission_csv_multiclass(predictions_mo_mn_tfidf_tuned, 'submission_mo_mn_tfidf_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7232d634",
   "metadata": {},
   "source": [
    "#### Test Accuracy Score\n",
    "From the results below, we can determine that the not tuned version of the TF-IDF was better than the tuned version. Meanwhile, this is the opposite for the model that used the Count vector. \n",
    "\n",
    "However, comparing the results of the `MultiOutputClassifier`, we can see the `LogisticRegression` performed better for all versions of the models. Although, they follow the same pattern with regards the vectors that they used as inputs (i.e., models that used untuned TF-IDF had a higher score, while tuned model that utilized Count vector had a higher score)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a15df76",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "results = {\n",
    "    \"submission_mo_mn_count\": ['MultiOutputClassifier (Multinomial NB)', 'Count Vectors', 'Not Tuned', 0.84551, 0.85581],\n",
    "    \"submission_mo_mn_tfidf\": ['MultiOutputClassifier (Multinomial NB)', 'TF-IDF Vectors', 'Not Tuned', 0.90867, 0.91595],\n",
    "    \"submission_mo_mn_count_tuned\": ['MultiOutputClassifier (Multinomial NB)', 'Count Vectors', 'Tuned', 0.87456, 0.88221],\n",
    "    \"submission_mo_mn_tfidf_tuned\": ['MultiOutputClassifier (Multinomial NB)', 'TF-IDF Vectors', 'Tuned', 0.89620, 0.90115]\n",
    "}\n",
    "\n",
    "all_results = update_results(all_results, results)\n",
    "pd.DataFrame(results, index=index).T.sort_values('test accuracy', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ce6d45",
   "metadata": {},
   "source": [
    "## All Scores\n",
    "The following is a tabulated list of all the models trained with their respective scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "id": "6a6c8860",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>vector</th>\n",
       "      <th>tuned</th>\n",
       "      <th>private</th>\n",
       "      <th>public</th>\n",
       "      <th>test accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>submission_lr_tfidf</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mo_lr_tfidf</th>\n",
       "      <td>MultiOutputClassifier (Logistic Regression)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_tfidf_tuned</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mo_lr_tfidf_tuned</th>\n",
       "      <td>MultiOutputClassifier (Logistic Regression)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_lr_tfidf</th>\n",
       "      <td>OneVsRestClassifier (Logistic Regression)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_lr_tfidf_tuned</th>\n",
       "      <td>OneVsRestClassifier (Logistic Regression)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.97558</td>\n",
       "      <td>0.97621</td>\n",
       "      <td>0.97564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_tfidf</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.96854</td>\n",
       "      <td>0.97321</td>\n",
       "      <td>0.96901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_xgb_tfidf</th>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.96502</td>\n",
       "      <td>0.96803</td>\n",
       "      <td>0.96532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_xgb_count</th>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.96468</td>\n",
       "      <td>0.96783</td>\n",
       "      <td>0.96499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_xgb_tfidf_tuned</th>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.96396</td>\n",
       "      <td>0.96693</td>\n",
       "      <td>0.96426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_xgb_count_tuned</th>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.96282</td>\n",
       "      <td>0.96673</td>\n",
       "      <td>0.96321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_tfidf_tuned</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.96312</td>\n",
       "      <td>0.96252</td>\n",
       "      <td>0.96306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_wrd2v_tuned</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.9524</td>\n",
       "      <td>0.94993</td>\n",
       "      <td>0.95215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_wrd2v</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.95233</td>\n",
       "      <td>0.94982</td>\n",
       "      <td>0.95208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mo_lr_count_tuned</th>\n",
       "      <td>MultiOutputClassifier (Logistic Regression)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.95113</td>\n",
       "      <td>0.95159</td>\n",
       "      <td>0.95118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mo_lr_count</th>\n",
       "      <td>MultiOutputClassifier (Logistic Regression)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.94866</td>\n",
       "      <td>0.95065</td>\n",
       "      <td>0.94886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_count_tuned</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.94845</td>\n",
       "      <td>0.94248</td>\n",
       "      <td>0.94785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_lr_count</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.94845</td>\n",
       "      <td>0.94248</td>\n",
       "      <td>0.94785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_count</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.94066</td>\n",
       "      <td>0.94585</td>\n",
       "      <td>0.94118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_wrd2v_tuned</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.94065</td>\n",
       "      <td>0.94117</td>\n",
       "      <td>0.9407</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_lr_count</th>\n",
       "      <td>OneVsRestClassifier (Logistic Regression)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.93996</td>\n",
       "      <td>0.9441</td>\n",
       "      <td>0.94037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_wrd2v</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.9387</td>\n",
       "      <td>0.94112</td>\n",
       "      <td>0.93894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_adb_tfidf</th>\n",
       "      <td>AdaBoostClassifier</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.9383</td>\n",
       "      <td>0.94145</td>\n",
       "      <td>0.93862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_tfidf</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.9374</td>\n",
       "      <td>0.93986</td>\n",
       "      <td>0.93765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_adb_count</th>\n",
       "      <td>AdaBoostClassifier</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.93539</td>\n",
       "      <td>0.94218</td>\n",
       "      <td>0.93607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_wrd2v_tuned</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.93585</td>\n",
       "      <td>0.93342</td>\n",
       "      <td>0.93561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_gbc_count</th>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.92562</td>\n",
       "      <td>0.93158</td>\n",
       "      <td>0.92622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_lr_count_tuned</th>\n",
       "      <td>OneVsRestClassifier (Logistic Regression)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.91488</td>\n",
       "      <td>0.91866</td>\n",
       "      <td>0.91526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_wrd2v</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>Word2Vec Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.91181</td>\n",
       "      <td>0.9156</td>\n",
       "      <td>0.91219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mo_mn_tfidf</th>\n",
       "      <td>MultiOutputClassifier (Multinomial NB)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.90867</td>\n",
       "      <td>0.91595</td>\n",
       "      <td>0.9094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_mn_tfidf_tuned</th>\n",
       "      <td>OneVsRestClassifier (Multinomial NB)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.90597</td>\n",
       "      <td>0.91409</td>\n",
       "      <td>0.90678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_gbc_tfidf</th>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.9049</td>\n",
       "      <td>0.91988</td>\n",
       "      <td>0.9064</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_count_tuned</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.90325</td>\n",
       "      <td>0.90111</td>\n",
       "      <td>0.90304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_rf_tfidf_tuned</th>\n",
       "      <td>Random Forest Classifer</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.89725</td>\n",
       "      <td>0.89771</td>\n",
       "      <td>0.8973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_count_tuned</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.89659</td>\n",
       "      <td>0.8979</td>\n",
       "      <td>0.89672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mo_mn_tfidf_tuned</th>\n",
       "      <td>MultiOutputClassifier (Multinomial NB)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.8962</td>\n",
       "      <td>0.90115</td>\n",
       "      <td>0.89669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_sgd_count</th>\n",
       "      <td>SGDClassifer</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.88323</td>\n",
       "      <td>0.88505</td>\n",
       "      <td>0.88341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mo_mn_count_tuned</th>\n",
       "      <td>MultiOutputClassifier (Multinomial NB)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.87456</td>\n",
       "      <td>0.88221</td>\n",
       "      <td>0.87532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mn_count</th>\n",
       "      <td>Multinomial Naive Bayes</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.84551</td>\n",
       "      <td>0.85581</td>\n",
       "      <td>0.84654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_mn_count_tuned</th>\n",
       "      <td>OneVsRestClassifier (Multinomial NB)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.84551</td>\n",
       "      <td>0.85581</td>\n",
       "      <td>0.84654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_mn_count</th>\n",
       "      <td>OneVsRestClassifier (Multinomial NB)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.84551</td>\n",
       "      <td>0.85581</td>\n",
       "      <td>0.84654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mo_mn_count</th>\n",
       "      <td>MultiOutputClassifier (Multinomial NB)</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.84551</td>\n",
       "      <td>0.85581</td>\n",
       "      <td>0.84654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mn_tfidf</th>\n",
       "      <td>Multinomial Naive Bayes</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.8293</td>\n",
       "      <td>0.83952</td>\n",
       "      <td>0.83032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_oc_mn_tfidf</th>\n",
       "      <td>OneVsRestClassifier (Multinomial NB)</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.8251</td>\n",
       "      <td>0.83586</td>\n",
       "      <td>0.82618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mn_tfidf</th>\n",
       "      <td>Multinomial Naive Bayes</td>\n",
       "      <td>TF-IDF Vectors</td>\n",
       "      <td>Not Tuned</td>\n",
       "      <td>0.8251</td>\n",
       "      <td>0.83586</td>\n",
       "      <td>0.82618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>submission_mn_count</th>\n",
       "      <td>Multinomial Naive Bayes</td>\n",
       "      <td>Count Vectors</td>\n",
       "      <td>Tuned</td>\n",
       "      <td>0.75966</td>\n",
       "      <td>0.76208</td>\n",
       "      <td>0.7599</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    model  \\\n",
       "submission_lr_tfidf                                   Logistic Regression   \n",
       "submission_mo_lr_tfidf        MultiOutputClassifier (Logistic Regression)   \n",
       "submission_lr_tfidf_tuned                             Logistic Regression   \n",
       "submission_mo_lr_tfidf_tuned  MultiOutputClassifier (Logistic Regression)   \n",
       "submission_oc_lr_tfidf          OneVsRestClassifier (Logistic Regression)   \n",
       "submission_oc_lr_tfidf_tuned    OneVsRestClassifier (Logistic Regression)   \n",
       "submission_sgd_tfidf                                         SGDClassifer   \n",
       "submission_xgb_tfidf                                        XGBClassifier   \n",
       "submission_xgb_count                                        XGBClassifier   \n",
       "submission_xgb_tfidf_tuned                                  XGBClassifier   \n",
       "submission_xgb_count_tuned                                  XGBClassifier   \n",
       "submission_sgd_tfidf_tuned                                   SGDClassifer   \n",
       "submission_lr_wrd2v_tuned                             Logistic Regression   \n",
       "submission_lr_wrd2v                                   Logistic Regression   \n",
       "submission_mo_lr_count_tuned  MultiOutputClassifier (Logistic Regression)   \n",
       "submission_mo_lr_count        MultiOutputClassifier (Logistic Regression)   \n",
       "submission_lr_count_tuned                             Logistic Regression   \n",
       "submission_lr_count                                   Logistic Regression   \n",
       "submission_rf_count                               Random Forest Classifer   \n",
       "submission_rf_wrd2v_tuned                         Random Forest Classifer   \n",
       "submission_oc_lr_count          OneVsRestClassifier (Logistic Regression)   \n",
       "submission_rf_wrd2v                               Random Forest Classifer   \n",
       "submission_adb_tfidf                                   AdaBoostClassifier   \n",
       "submission_rf_tfidf                               Random Forest Classifer   \n",
       "submission_adb_count                                   AdaBoostClassifier   \n",
       "submission_sgd_wrd2v_tuned                                   SGDClassifer   \n",
       "submission_gbc_count                           GradientBoostingClassifier   \n",
       "submission_oc_lr_count_tuned    OneVsRestClassifier (Logistic Regression)   \n",
       "submission_sgd_wrd2v                                         SGDClassifer   \n",
       "submission_mo_mn_tfidf             MultiOutputClassifier (Multinomial NB)   \n",
       "submission_oc_mn_tfidf_tuned         OneVsRestClassifier (Multinomial NB)   \n",
       "submission_gbc_tfidf                           GradientBoostingClassifier   \n",
       "submission_rf_count_tuned                         Random Forest Classifer   \n",
       "submission_rf_tfidf_tuned                         Random Forest Classifer   \n",
       "submission_sgd_count_tuned                                   SGDClassifer   \n",
       "submission_mo_mn_tfidf_tuned       MultiOutputClassifier (Multinomial NB)   \n",
       "submission_sgd_count                                         SGDClassifer   \n",
       "submission_mo_mn_count_tuned       MultiOutputClassifier (Multinomial NB)   \n",
       "submission_mn_count                               Multinomial Naive Bayes   \n",
       "submission_oc_mn_count_tuned         OneVsRestClassifier (Multinomial NB)   \n",
       "submission_oc_mn_count               OneVsRestClassifier (Multinomial NB)   \n",
       "submission_mo_mn_count             MultiOutputClassifier (Multinomial NB)   \n",
       "submission_mn_tfidf                               Multinomial Naive Bayes   \n",
       "submission_oc_mn_tfidf               OneVsRestClassifier (Multinomial NB)   \n",
       "submission_mn_tfidf                               Multinomial Naive Bayes   \n",
       "submission_mn_count                               Multinomial Naive Bayes   \n",
       "\n",
       "                                        vector       tuned  private   public  \\\n",
       "submission_lr_tfidf             TF-IDF Vectors   Not Tuned  0.97558  0.97621   \n",
       "submission_mo_lr_tfidf          TF-IDF Vectors   Not Tuned  0.97558  0.97621   \n",
       "submission_lr_tfidf_tuned       TF-IDF Vectors       Tuned  0.97558  0.97621   \n",
       "submission_mo_lr_tfidf_tuned    TF-IDF Vectors       Tuned  0.97558  0.97621   \n",
       "submission_oc_lr_tfidf          TF-IDF Vectors   Not Tuned  0.97558  0.97621   \n",
       "submission_oc_lr_tfidf_tuned    TF-IDF Vectors       Tuned  0.97558  0.97621   \n",
       "submission_sgd_tfidf            TF-IDF Vectors   Not Tuned  0.96854  0.97321   \n",
       "submission_xgb_tfidf            TF-IDF Vectors   Not Tuned  0.96502  0.96803   \n",
       "submission_xgb_count             Count Vectors   Not Tuned  0.96468  0.96783   \n",
       "submission_xgb_tfidf_tuned      TF-IDF Vectors       Tuned  0.96396  0.96693   \n",
       "submission_xgb_count_tuned       Count Vectors       Tuned  0.96282  0.96673   \n",
       "submission_sgd_tfidf_tuned      TF-IDF Vectors       Tuned  0.96312  0.96252   \n",
       "submission_lr_wrd2v_tuned     Word2Vec Vectors       Tuned   0.9524  0.94993   \n",
       "submission_lr_wrd2v           Word2Vec Vectors   Not Tuned  0.95233  0.94982   \n",
       "submission_mo_lr_count_tuned     Count Vectors       Tuned  0.95113  0.95159   \n",
       "submission_mo_lr_count           Count Vectors   Not Tuned  0.94866  0.95065   \n",
       "submission_lr_count_tuned        Count Vectors       Tuned  0.94845  0.94248   \n",
       "submission_lr_count              Count Vectors   Not Tuned  0.94845  0.94248   \n",
       "submission_rf_count              Count Vectors   Not Tuned  0.94066  0.94585   \n",
       "submission_rf_wrd2v_tuned     Word2Vec Vectors       Tuned  0.94065  0.94117   \n",
       "submission_oc_lr_count           Count Vectors   Not Tuned  0.93996   0.9441   \n",
       "submission_rf_wrd2v           Word2Vec Vectors   Not Tuned   0.9387  0.94112   \n",
       "submission_adb_tfidf            TF-IDF Vectors   Not Tuned   0.9383  0.94145   \n",
       "submission_rf_tfidf             TF-IDF Vectors   Not Tuned   0.9374  0.93986   \n",
       "submission_adb_count             Count Vectors   Not Tuned  0.93539  0.94218   \n",
       "submission_sgd_wrd2v_tuned    Word2Vec Vectors       Tuned  0.93585  0.93342   \n",
       "submission_gbc_count             Count Vectors   Not Tuned  0.92562  0.93158   \n",
       "submission_oc_lr_count_tuned     Count Vectors       Tuned  0.91488  0.91866   \n",
       "submission_sgd_wrd2v          Word2Vec Vectors   Not Tuned  0.91181   0.9156   \n",
       "submission_mo_mn_tfidf          TF-IDF Vectors   Not Tuned  0.90867  0.91595   \n",
       "submission_oc_mn_tfidf_tuned    TF-IDF Vectors       Tuned  0.90597  0.91409   \n",
       "submission_gbc_tfidf            TF-IDF Vectors   Not Tuned   0.9049  0.91988   \n",
       "submission_rf_count_tuned        Count Vectors       Tuned  0.90325  0.90111   \n",
       "submission_rf_tfidf_tuned       TF-IDF Vectors       Tuned  0.89725  0.89771   \n",
       "submission_sgd_count_tuned       Count Vectors       Tuned  0.89659   0.8979   \n",
       "submission_mo_mn_tfidf_tuned    TF-IDF Vectors       Tuned   0.8962  0.90115   \n",
       "submission_sgd_count             Count Vectors   Not Tuned  0.88323  0.88505   \n",
       "submission_mo_mn_count_tuned     Count Vectors       Tuned  0.87456  0.88221   \n",
       "submission_mn_count              Count Vectors   Not Tuned  0.84551  0.85581   \n",
       "submission_oc_mn_count_tuned     Count Vectors       Tuned  0.84551  0.85581   \n",
       "submission_oc_mn_count           Count Vectors   Not Tuned  0.84551  0.85581   \n",
       "submission_mo_mn_count           Count Vectors   Not Tuned  0.84551  0.85581   \n",
       "submission_mn_tfidf             TF-IDF Vectors       Tuned   0.8293  0.83952   \n",
       "submission_oc_mn_tfidf          TF-IDF Vectors   Not Tuned   0.8251  0.83586   \n",
       "submission_mn_tfidf             TF-IDF Vectors   Not Tuned   0.8251  0.83586   \n",
       "submission_mn_count              Count Vectors       Tuned  0.75966  0.76208   \n",
       "\n",
       "                             test accuracy  \n",
       "submission_lr_tfidf                0.97564  \n",
       "submission_mo_lr_tfidf             0.97564  \n",
       "submission_lr_tfidf_tuned          0.97564  \n",
       "submission_mo_lr_tfidf_tuned       0.97564  \n",
       "submission_oc_lr_tfidf             0.97564  \n",
       "submission_oc_lr_tfidf_tuned       0.97564  \n",
       "submission_sgd_tfidf               0.96901  \n",
       "submission_xgb_tfidf               0.96532  \n",
       "submission_xgb_count               0.96499  \n",
       "submission_xgb_tfidf_tuned         0.96426  \n",
       "submission_xgb_count_tuned         0.96321  \n",
       "submission_sgd_tfidf_tuned         0.96306  \n",
       "submission_lr_wrd2v_tuned          0.95215  \n",
       "submission_lr_wrd2v                0.95208  \n",
       "submission_mo_lr_count_tuned       0.95118  \n",
       "submission_mo_lr_count             0.94886  \n",
       "submission_lr_count_tuned          0.94785  \n",
       "submission_lr_count                0.94785  \n",
       "submission_rf_count                0.94118  \n",
       "submission_rf_wrd2v_tuned           0.9407  \n",
       "submission_oc_lr_count             0.94037  \n",
       "submission_rf_wrd2v                0.93894  \n",
       "submission_adb_tfidf               0.93862  \n",
       "submission_rf_tfidf                0.93765  \n",
       "submission_adb_count               0.93607  \n",
       "submission_sgd_wrd2v_tuned         0.93561  \n",
       "submission_gbc_count               0.92622  \n",
       "submission_oc_lr_count_tuned       0.91526  \n",
       "submission_sgd_wrd2v               0.91219  \n",
       "submission_mo_mn_tfidf              0.9094  \n",
       "submission_oc_mn_tfidf_tuned       0.90678  \n",
       "submission_gbc_tfidf                0.9064  \n",
       "submission_rf_count_tuned          0.90304  \n",
       "submission_rf_tfidf_tuned           0.8973  \n",
       "submission_sgd_count_tuned         0.89672  \n",
       "submission_mo_mn_tfidf_tuned       0.89669  \n",
       "submission_sgd_count               0.88341  \n",
       "submission_mo_mn_count_tuned       0.87532  \n",
       "submission_mn_count                0.84654  \n",
       "submission_oc_mn_count_tuned       0.84654  \n",
       "submission_oc_mn_count             0.84654  \n",
       "submission_mo_mn_count             0.84654  \n",
       "submission_mn_tfidf                0.83032  \n",
       "submission_oc_mn_tfidf             0.82618  \n",
       "submission_mn_tfidf                0.82618  \n",
       "submission_mn_count                 0.7599  "
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_results.T.sort_values('test accuracy', ascending=False).drop_duplicates(subset=['model', 'vector', 'tuned'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}