17-Dealing_with_Categorical_Variables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dealing with Categorical Varaibles\n",
    "\n",
    "1. Binary, Ordinal, and Nominal Variables \n",
    "2. Binary Encoding\n",
    "3. Ordinary Encoding\n",
    "4. Nominal Encoding\n",
    "5. Other Encoding Techniques"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Binary, Ordinal, and Nominal Variables\n",
    "\n",
    "Categorical data is a common type of non-numerical data that contains label values and not numbers. Some examples include:\n",
    "<br>Colors: Red, Green, Blue\n",
    "<br>Cities: New York, Austin, Denver\n",
    "<br>Gender: Male, Female\n",
    "<br>Place: First, Second, Third\n",
    "\n",
    "<br>According to Wikipedia, “a categorical variable is a variable that can take on one of a limited, and usually fixed number of possible values.”\n",
    "\n",
    "<br>It is common to refer to a possible value of a categorical variable as a level.\n",
    "\n",
    "<br>There are several different types of categorical data including:\n",
    "<br>Binary: A variable that has only 2 values. For example, True/False or Yes/No.\n",
    "<br>Ordinal: A variable that has some order associated with it like our place example above.\n",
    "<br>Nominal: A variable that has no numerical importance, for example color or city.\n",
    "\n",
    "<br>Many machine learning algorithms cannot work with categorical data directly. They require data to be numeric. Therefore, it is essential to know how to encode categorical variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Binary Encoding\n",
    "\n",
    "Binary features are those with only two possible values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load labraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import os \n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# Display all the columns of the dataframe\n",
    "pd.pandas.set_option('display.max_columns', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create dataframe\n",
    "df = pd.DataFrame([\n",
    "    ['green', 'M',   10.1, 'class1'],\n",
    "    ['red',   'L',   13.5, 'class2'],\n",
    "    ['blue',  'XL',  15.3, 'class1'],\n",
    "    ['black', 'XXL', 17.1, 'class2'],\n",
    "    ['grey',  np.NAN, 19.1, 'class1'],\n",
    "    ])\n",
    "\n",
    "#Create column names\n",
    "df.columns = ['Color','Size','Price','ClassLabel']\n",
    "\n",
    "# See dataframe\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Because these are binary features, we can use Panda’s replace() to encode them\n",
    "# Here we pass a dictionary to replace() with the current value as the key and the desired value as the value\n",
    "\n",
    "df['ClassLabel_Encoded'] = df['ClassLabel'].replace({'class1':0, 'class2':1})\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using ```replace()``` is very helpful, for binary features, but what if we have categorical features with more categories?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Ordinal Feature Encoding\n",
    "\n",
    "Ordinal features are those with some order associated with them. We can tell from our sample of ordinal features above these features have an order that may be important.\n",
    "\n",
    "The machine learning model may be able to use the order information to make better predictions and we want to preserve it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See our dataframe\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ordinal features, we use integer encoding. To integer encode our data we simply convert labels to integer values.\n",
    "\n",
    "While there are many methods for integer encoding, we will discuss two here:\n",
    "- Sklearn’s LabelEncoder()\n",
    "- Panda’s map()\n",
    "        \n",
    "We can label encode data with Sklearn’s LabelEncoder():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import library\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "# Copy data set\n",
    "ordinal_features = df.copy()\n",
    "\n",
    "#label encoder can't handle missing values\n",
    "ordinal_features['Size'] = ordinal_features['Size'].fillna('None')\n",
    "\n",
    "# Label encode Size feature\n",
    "label_encoder = LabelEncoder()\n",
    "\n",
    "ordinal_features['Size_Encoded'] = label_encoder.fit_transform(ordinal_features['Size'])\n",
    "\n",
    "# Print sample of dataset\n",
    "ordinal_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we see the encoded feature ```Size```. We can see the value has been encoded acording to the alphabetic order (L-0, M-1, NAN-2, XL-3, XXL-4).\n",
    "\n",
    "While using ```LabelEncoder()``` is very quick and easy, it may not be the best choice here: the order of our encoding is not exactly right. Also, we had to handle our null values before being able to use it.\n",
    "\n",
    "Another downside to ```LabelEncoder()``` is the fact that the documentation states it should be used for encoding target values (y) and not for the inputs (x). Let’s explore a different method for encoding our ordinal features.\n",
    "\n",
    "Another option here is to use ```map()```.\n",
    "\n",
    "Panda’s ```map()``` substitutes each value with another specified value, similar to ```replace()``` that we used above. Here we create a dictionary with our desired mapping and apply the mapping to our series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create dictionary of ordinal to integer mapping\n",
    "size_map = {\n",
    "            np.NAN : 0,\n",
    "            'L'    : 1, \n",
    "            'M'    : 2, \n",
    "            'XL'   : 3, \n",
    "            'XXL'  : 4\n",
    "        }\n",
    "\n",
    "# Apply using map\n",
    "df['Size_Encoded'] = df['Size'].map(size_map)\n",
    "\n",
    "# See our data\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using ```map()``` allowed us to specify the order of the values in our categorical feature to ensure they are in a meaningful arrangement.\n",
    "\n",
    "These methods should only be used for ordinal features, where the order matters. For features where order is not important we must explore other techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Nominal Features\n",
    "\n",
    "Nominal features are categorical features that have no numerical importance. Order does not matter like in our example of color feature.\n",
    "\n",
    "**One-hot encoding is a better technique when order doesn’t matter.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Working with Color - Nominal Feature\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In one hot encoding, a new binary (dummy) variable is created for each unique value in the categorical variable. In our dataset, we have 5 unique colors and so we create 5 new features, one for each color. If the value is true, the integer 1 is placed in the field, if false then a 0.\n",
    "\n",
    "Here we can use Sklearn’s `OneHotEncoder()` to one hot encode our nominal features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Library\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "# Creating one hot encoder object \n",
    "onehotencoder = OneHotEncoder()\n",
    "\n",
    "# Reshape the 1-D country array to 2-D as fit_transform expects 2-D and finally fit the object \n",
    "X = onehotencoder.fit_transform(df['Color'].values.reshape(-1,1)).toarray()\n",
    "\n",
    "# To add this back into the original dataframe \n",
    "dfOneHot = pd.DataFrame(X, columns = [\"Color_\"+str(onehotencoder.categories_[0][i]) for i in range(len(onehotencoder.categories_[0]))]) \n",
    "df1 = pd.concat([df, dfOneHot], axis=1)\n",
    "\n",
    "# Droping the color column \n",
    "df1= df1.drop(['Color'], axis=1) \n",
    "\n",
    "# Printing to verify \n",
    "df1.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can use Panda’s `get_dummies()` to one hot encode our nominal features.\n",
    "\n",
    "This method converts a categorical variable to dummy variables and returns a dataframe. The `drop_first` parameter is helpful to get k-1 dummies by removing the first level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One-Hot Encoding without froping first column, drop_first = False\n",
    "nominal_features_F = pd.get_dummies(df[['Color', 'Size_Encoded', 'Price', 'ClassLabel_Encoded']], drop_first=False)\n",
    "nominal_features_F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One-Hot Encoding with froping first column, drop_first = True\n",
    "nominal_features_T = pd.get_dummies(df[['Color', 'Size_Encoded', 'Price', 'ClassLabel_Encoded']], drop_first=True)\n",
    "nominal_features_T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the sample of our one hot encoded nominal features above that this type of encoding can greatly increase our number of columns. We input 1 column and after encoding now have 4! It introduces sparsity in the dataset i.e several columns having 0s and a few of them having 1s. In other words, it creates multiple dummy features in the dataset without adding much information.\n",
    "\n",
    "In the situation of high cardinality features, those with many possible values, we may need to do some manipulation prior to encoding. For example, for values occurring only a small percent of the time, we could group them into an “other” category.\n",
    "\n",
    "Also, they might lead to a Dummy variable trap. It is a phenomenon where features are highly correlated. That means using the other variables, we can easily predict the value of a variable.\n",
    "\n",
    "Due to the massive increase in the dataset, coding slows down the learning of the model along with deteriorating the overall performance that ultimately makes the model computationally expensive. Further, while using tree-based models these encodings are not an optimum choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Other Encoding Techniques\n",
    "\n",
    "THere are many other techniques to deal with categorical variables that i will reffer you to explore on your own.\n",
    "Here is the list some of them:\n",
    "1. Helmert Encoding\n",
    "2. Frequency Encoding\n",
    "3. Mean Encoding\n",
    "4. Weight of Evidence Encoding\n",
    "5. Probability Ratio Encoding\n",
    "6. Hashing Encoding\n",
    "7. Backward Difference Encoding\n",
    "8. James-Stein Encoding\n",
    "9. M-estimator Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}