3-DataWrangling-.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Data Wrangling**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will perform the following:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Identify duplicate values in the dataset.\n",
    "\n",
    "*   Remove duplicate values from the dataset.\n",
    "\n",
    "*   Identify missing values in the dataset.\n",
    "\n",
    "*   Impute the missing values in the dataset.\n",
    "\n",
    "*   Normalize data in the dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, we import pandas module.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the dataset into a dataframe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>22.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>New Zealand</td>\n",
       "      <td>No</td>\n",
       "      <td>Some college/university study without earning ...</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Bisexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Somewhat more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Cour...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Australia</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>29.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>Hispanic or Latino/Latina;Multiracial</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11547</th>\n",
       "      <td>25136</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Cour...</td>\n",
       "      <td>36.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11548</th>\n",
       "      <td>25137</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Poland</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>A lot more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11549</th>\n",
       "      <td>25138</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>A lot more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>34.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11550</th>\n",
       "      <td>25141</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>OSS is, on average, of LOWER quality than prop...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Switzerland</td>\n",
       "      <td>No</td>\n",
       "      <td>Secondary school (e.g. American high school, G...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11551</th>\n",
       "      <td>25142</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Other doctoral degree (Ph.D, Ed.D., etc.)</td>\n",
       "      <td>A natural science (ex. biology, chemistry, phy...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Bisexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11552 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Respondent                      MainBranch Hobbyist  \\\n",
       "0               4  I am a developer by profession       No   \n",
       "1               9  I am a developer by profession      Yes   \n",
       "2              13  I am a developer by profession      Yes   \n",
       "3              16  I am a developer by profession      Yes   \n",
       "4              17  I am a developer by profession      Yes   \n",
       "...           ...                             ...      ...   \n",
       "11547       25136  I am a developer by profession      Yes   \n",
       "11548       25137  I am a developer by profession      Yes   \n",
       "11549       25138  I am a developer by profession      Yes   \n",
       "11550       25141  I am a developer by profession      Yes   \n",
       "11551       25142  I am a developer by profession      Yes   \n",
       "\n",
       "                                             OpenSourcer  \\\n",
       "0                                                  Never   \n",
       "1                             Once a month or more often   \n",
       "2      Less than once a month but more than once per ...   \n",
       "3                                                  Never   \n",
       "4      Less than once a month but more than once per ...   \n",
       "...                                                  ...   \n",
       "11547                                              Never   \n",
       "11548                                              Never   \n",
       "11549                            Less than once per year   \n",
       "11550  Less than once a month but more than once per ...   \n",
       "11551  Less than once a month but more than once per ...   \n",
       "\n",
       "                                              OpenSource          Employment  \\\n",
       "0      The quality of OSS and closed source software ...  Employed full-time   \n",
       "1      The quality of OSS and closed source software ...  Employed full-time   \n",
       "2      OSS is, on average, of HIGHER quality than pro...  Employed full-time   \n",
       "3      The quality of OSS and closed source software ...  Employed full-time   \n",
       "4      The quality of OSS and closed source software ...  Employed full-time   \n",
       "...                                                  ...                 ...   \n",
       "11547  OSS is, on average, of HIGHER quality than pro...  Employed full-time   \n",
       "11548  The quality of OSS and closed source software ...  Employed full-time   \n",
       "11549  The quality of OSS and closed source software ...  Employed full-time   \n",
       "11550  OSS is, on average, of LOWER quality than prop...  Employed full-time   \n",
       "11551  OSS is, on average, of HIGHER quality than pro...  Employed full-time   \n",
       "\n",
       "              Country Student  \\\n",
       "0       United States      No   \n",
       "1         New Zealand      No   \n",
       "2       United States      No   \n",
       "3      United Kingdom      No   \n",
       "4           Australia      No   \n",
       "...               ...     ...   \n",
       "11547   United States      No   \n",
       "11548          Poland      No   \n",
       "11549   United States      No   \n",
       "11550     Switzerland      No   \n",
       "11551  United Kingdom      No   \n",
       "\n",
       "                                                 EdLevel  \\\n",
       "0               Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "1      Some college/university study without earning ...   \n",
       "2            Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "3            Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "4               Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "...                                                  ...   \n",
       "11547        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "11548        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "11549        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "11550  Secondary school (e.g. American high school, G...   \n",
       "11551          Other doctoral degree (Ph.D, Ed.D., etc.)   \n",
       "\n",
       "                                          UndergradMajor  ...  \\\n",
       "0      Computer science, computer engineering, or sof...  ...   \n",
       "1      Computer science, computer engineering, or sof...  ...   \n",
       "2      Computer science, computer engineering, or sof...  ...   \n",
       "3                                                    NaN  ...   \n",
       "4      Computer science, computer engineering, or sof...  ...   \n",
       "...                                                  ...  ...   \n",
       "11547  Computer science, computer engineering, or sof...  ...   \n",
       "11548  Computer science, computer engineering, or sof...  ...   \n",
       "11549  Computer science, computer engineering, or sof...  ...   \n",
       "11550                                                NaN  ...   \n",
       "11551  A natural science (ex. biology, chemistry, phy...  ...   \n",
       "\n",
       "                                  WelcomeChange  \\\n",
       "0       Just as welcome now as I felt last year   \n",
       "1       Just as welcome now as I felt last year   \n",
       "2      Somewhat more welcome now than last year   \n",
       "3       Just as welcome now as I felt last year   \n",
       "4       Just as welcome now as I felt last year   \n",
       "...                                         ...   \n",
       "11547   Just as welcome now as I felt last year   \n",
       "11548     A lot more welcome now than last year   \n",
       "11549     A lot more welcome now than last year   \n",
       "11550  Somewhat less welcome now than last year   \n",
       "11551   Just as welcome now as I felt last year   \n",
       "\n",
       "                                            SONewContent   Age Gender Trans  \\\n",
       "0      Tech articles written by other developers;Indu...  22.0    Man    No   \n",
       "1                                                    NaN  23.0    Man    No   \n",
       "2      Tech articles written by other developers;Cour...  28.0    Man    No   \n",
       "3      Tech articles written by other developers;Indu...  26.0    Man    No   \n",
       "4      Tech articles written by other developers;Indu...  29.0    Man    No   \n",
       "...                                                  ...   ...    ...   ...   \n",
       "11547  Tech articles written by other developers;Cour...  36.0    Man    No   \n",
       "11548  Tech articles written by other developers;Tech...  25.0    Man    No   \n",
       "11549  Tech articles written by other developers;Indu...  34.0    Man    No   \n",
       "11550                                                NaN  25.0    Man    No   \n",
       "11551  Tech articles written by other developers;Tech...  30.0    Man    No   \n",
       "\n",
       "                     Sexuality                              Ethnicity  \\\n",
       "0      Straight / Heterosexual           White or of European descent   \n",
       "1                     Bisexual           White or of European descent   \n",
       "2      Straight / Heterosexual           White or of European descent   \n",
       "3      Straight / Heterosexual           White or of European descent   \n",
       "4      Straight / Heterosexual  Hispanic or Latino/Latina;Multiracial   \n",
       "...                        ...                                    ...   \n",
       "11547  Straight / Heterosexual           White or of European descent   \n",
       "11548  Straight / Heterosexual           White or of European descent   \n",
       "11549  Straight / Heterosexual           White or of European descent   \n",
       "11550  Straight / Heterosexual           White or of European descent   \n",
       "11551                 Bisexual           White or of European descent   \n",
       "\n",
       "      Dependents           SurveyLength                  SurveyEase  \n",
       "0             No  Appropriate in length                        Easy  \n",
       "1             No  Appropriate in length  Neither easy nor difficult  \n",
       "2            Yes  Appropriate in length                        Easy  \n",
       "3             No  Appropriate in length  Neither easy nor difficult  \n",
       "4             No  Appropriate in length                        Easy  \n",
       "...          ...                    ...                         ...  \n",
       "11547         No  Appropriate in length                   Difficult  \n",
       "11548         No  Appropriate in length  Neither easy nor difficult  \n",
       "11549        Yes               Too long                        Easy  \n",
       "11550         No  Appropriate in length                        Easy  \n",
       "11551         No  Appropriate in length                        Easy  \n",
       "\n",
       "[11552 rows x 85 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/LargeData/m1_survey_data.csv\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding duplicates\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find how many duplicate rows exist in the dataframe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "154"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "duplicated_rows = df[df.duplicated()]\n",
    "len(duplicated_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing duplicates\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove the duplicate rows from the dataframe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop_duplicates(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We verify if duplicates were actually dropped.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11398, 85)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding Missing values\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find the missing values for all columns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11547</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11548</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11549</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11550</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11551</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11398 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Respondent  MainBranch  Hobbyist  OpenSourcer  OpenSource  Employment  \\\n",
       "0           False       False     False        False       False       False   \n",
       "1           False       False     False        False       False       False   \n",
       "2           False       False     False        False       False       False   \n",
       "3           False       False     False        False       False       False   \n",
       "4           False       False     False        False       False       False   \n",
       "...           ...         ...       ...          ...         ...         ...   \n",
       "11547       False       False     False        False       False       False   \n",
       "11548       False       False     False        False       False       False   \n",
       "11549       False       False     False        False       False       False   \n",
       "11550       False       False     False        False       False       False   \n",
       "11551       False       False     False        False       False       False   \n",
       "\n",
       "       Country  Student  EdLevel  UndergradMajor  ...  WelcomeChange  \\\n",
       "0        False    False    False           False  ...          False   \n",
       "1        False    False    False           False  ...          False   \n",
       "2        False    False    False           False  ...          False   \n",
       "3        False    False    False            True  ...          False   \n",
       "4        False    False    False           False  ...          False   \n",
       "...        ...      ...      ...             ...  ...            ...   \n",
       "11547    False    False    False           False  ...          False   \n",
       "11548    False    False    False           False  ...          False   \n",
       "11549    False    False    False           False  ...          False   \n",
       "11550    False    False    False            True  ...          False   \n",
       "11551    False    False    False           False  ...          False   \n",
       "\n",
       "       SONewContent    Age  Gender  Trans  Sexuality  Ethnicity  Dependents  \\\n",
       "0             False  False   False  False      False      False       False   \n",
       "1              True  False   False  False      False      False       False   \n",
       "2             False  False   False  False      False      False       False   \n",
       "3             False  False   False  False      False      False       False   \n",
       "4             False  False   False  False      False      False       False   \n",
       "...             ...    ...     ...    ...        ...        ...         ...   \n",
       "11547         False  False   False  False      False      False       False   \n",
       "11548         False  False   False  False      False      False       False   \n",
       "11549         False  False   False  False      False      False       False   \n",
       "11550          True  False   False  False      False      False       False   \n",
       "11551         False  False   False  False      False      False       False   \n",
       "\n",
       "       SurveyLength  SurveyEase  \n",
       "0             False       False  \n",
       "1             False       False  \n",
       "2             False       False  \n",
       "3             False       False  \n",
       "4             False       False  \n",
       "...             ...         ...  \n",
       "11547         False       False  \n",
       "11548         False       False  \n",
       "11549         False       False  \n",
       "11550         False       False  \n",
       "11551         False       False  \n",
       "\n",
       "[11398 rows x 85 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "missing_data = df.isnull()\n",
    "missing_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find out how many rows are missing in the column 'WorkLoc'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WorkLoc\n",
      "False    11366\n",
      "True        32\n",
      "Name: WorkLoc, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#Counting missing values for a single column\n",
    "\n",
    "print('WorkLoc')\n",
    "print (missing_data['WorkLoc'].value_counts())  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Respondent\n",
      "False    11398\n",
      "Name: Respondent, dtype: int64\n",
      "\n",
      "MainBranch\n",
      "False    11398\n",
      "Name: MainBranch, dtype: int64\n",
      "\n",
      "Hobbyist\n",
      "False    11398\n",
      "Name: Hobbyist, dtype: int64\n",
      "\n",
      "OpenSourcer\n",
      "False    11398\n",
      "Name: OpenSourcer, dtype: int64\n",
      "\n",
      "OpenSource\n",
      "False    11317\n",
      "True        81\n",
      "Name: OpenSource, dtype: int64\n",
      "\n",
      "Employment\n",
      "False    11398\n",
      "Name: Employment, dtype: int64\n",
      "\n",
      "Country\n",
      "False    11398\n",
      "Name: Country, dtype: int64\n",
      "\n",
      "Student\n",
      "False    11347\n",
      "True        51\n",
      "Name: Student, dtype: int64\n",
      "\n",
      "EdLevel\n",
      "False    11286\n",
      "True       112\n",
      "Name: EdLevel, dtype: int64\n",
      "\n",
      "UndergradMajor\n",
      "False    10661\n",
      "True       737\n",
      "Name: UndergradMajor, dtype: int64\n",
      "\n",
      "EduOther\n",
      "False    11234\n",
      "True       164\n",
      "Name: EduOther, dtype: int64\n",
      "\n",
      "OrgSize\n",
      "False    11302\n",
      "True        96\n",
      "Name: OrgSize, dtype: int64\n",
      "\n",
      "DevType\n",
      "False    11333\n",
      "True        65\n",
      "Name: DevType, dtype: int64\n",
      "\n",
      "YearsCode\n",
      "False    11389\n",
      "True         9\n",
      "Name: YearsCode, dtype: int64\n",
      "\n",
      "Age1stCode\n",
      "False    11385\n",
      "True        13\n",
      "Name: Age1stCode, dtype: int64\n",
      "\n",
      "YearsCodePro\n",
      "False    11382\n",
      "True        16\n",
      "Name: YearsCodePro, dtype: int64\n",
      "\n",
      "CareerSat\n",
      "False    11398\n",
      "Name: CareerSat, dtype: int64\n",
      "\n",
      "JobSat\n",
      "False    11397\n",
      "True         1\n",
      "Name: JobSat, dtype: int64\n",
      "\n",
      "MgrIdiot\n",
      "False    10905\n",
      "True       493\n",
      "Name: MgrIdiot, dtype: int64\n",
      "\n",
      "MgrMoney\n",
      "False    10901\n",
      "True       497\n",
      "Name: MgrMoney, dtype: int64\n",
      "\n",
      "MgrWant\n",
      "False    10905\n",
      "True       493\n",
      "Name: MgrWant, dtype: int64\n",
      "\n",
      "JobSeek\n",
      "False    11398\n",
      "Name: JobSeek, dtype: int64\n",
      "\n",
      "LastHireDate\n",
      "False    11398\n",
      "Name: LastHireDate, dtype: int64\n",
      "\n",
      "LastInt\n",
      "False    10985\n",
      "True       413\n",
      "Name: LastInt, dtype: int64\n",
      "\n",
      "FizzBuzz\n",
      "False    11361\n",
      "True        37\n",
      "Name: FizzBuzz, dtype: int64\n",
      "\n",
      "JobFactors\n",
      "False    11395\n",
      "True         3\n",
      "Name: JobFactors, dtype: int64\n",
      "\n",
      "ResumeUpdate\n",
      "False    11359\n",
      "True        39\n",
      "Name: ResumeUpdate, dtype: int64\n",
      "\n",
      "CurrencySymbol\n",
      "False    11398\n",
      "Name: CurrencySymbol, dtype: int64\n",
      "\n",
      "CurrencyDesc\n",
      "False    11398\n",
      "Name: CurrencyDesc, dtype: int64\n",
      "\n",
      "CompTotal\n",
      "False    10589\n",
      "True       809\n",
      "Name: CompTotal, dtype: int64\n",
      "\n",
      "CompFreq\n",
      "False    11192\n",
      "True       206\n",
      "Name: CompFreq, dtype: int64\n",
      "\n",
      "ConvertedComp\n",
      "False    10582\n",
      "True       816\n",
      "Name: ConvertedComp, dtype: int64\n",
      "\n",
      "WorkWeekHrs\n",
      "False    11276\n",
      "True       122\n",
      "Name: WorkWeekHrs, dtype: int64\n",
      "\n",
      "WorkPlan\n",
      "False    11277\n",
      "True       121\n",
      "Name: WorkPlan, dtype: int64\n",
      "\n",
      "WorkChallenge\n",
      "False    11234\n",
      "True       164\n",
      "Name: WorkChallenge, dtype: int64\n",
      "\n",
      "WorkRemote\n",
      "False    11390\n",
      "True         8\n",
      "Name: WorkRemote, dtype: int64\n",
      "\n",
      "WorkLoc\n",
      "False    11366\n",
      "True        32\n",
      "Name: WorkLoc, dtype: int64\n",
      "\n",
      "ImpSyn\n",
      "False    11393\n",
      "True         5\n",
      "Name: ImpSyn, dtype: int64\n",
      "\n",
      "CodeRev\n",
      "False    11397\n",
      "True         1\n",
      "Name: CodeRev, dtype: int64\n",
      "\n",
      "CodeRevHrs\n",
      "False    8972\n",
      "True     2426\n",
      "Name: CodeRevHrs, dtype: int64\n",
      "\n",
      "UnitTests\n",
      "False    11369\n",
      "True        29\n",
      "Name: UnitTests, dtype: int64\n",
      "\n",
      "PurchaseHow\n",
      "False    11202\n",
      "True       196\n",
      "Name: PurchaseHow, dtype: int64\n",
      "\n",
      "PurchaseWhat\n",
      "False    11360\n",
      "True        38\n",
      "Name: PurchaseWhat, dtype: int64\n",
      "\n",
      "LanguageWorkedWith\n",
      "False    11387\n",
      "True        11\n",
      "Name: LanguageWorkedWith, dtype: int64\n",
      "\n",
      "LanguageDesireNextYear\n",
      "False    11264\n",
      "True       134\n",
      "Name: LanguageDesireNextYear, dtype: int64\n",
      "\n",
      "DatabaseWorkedWith\n",
      "False    10945\n",
      "True       453\n",
      "Name: DatabaseWorkedWith, dtype: int64\n",
      "\n",
      "DatabaseDesireNextYear\n",
      "False    10356\n",
      "True      1042\n",
      "Name: DatabaseDesireNextYear, dtype: int64\n",
      "\n",
      "PlatformWorkedWith\n",
      "False    10987\n",
      "True       411\n",
      "Name: PlatformWorkedWith, dtype: int64\n",
      "\n",
      "PlatformDesireNextYear\n",
      "False    10854\n",
      "True       544\n",
      "Name: PlatformDesireNextYear, dtype: int64\n",
      "\n",
      "WebFrameWorkedWith\n",
      "False    10005\n",
      "True      1393\n",
      "Name: WebFrameWorkedWith, dtype: int64\n",
      "\n",
      "WebFrameDesireNextYear\n",
      "False    9781\n",
      "True     1617\n",
      "Name: WebFrameDesireNextYear, dtype: int64\n",
      "\n",
      "MiscTechWorkedWith\n",
      "False    9216\n",
      "True     2182\n",
      "Name: MiscTechWorkedWith, dtype: int64\n",
      "\n",
      "MiscTechDesireNextYear\n",
      "False    9943\n",
      "True     1455\n",
      "Name: MiscTechDesireNextYear, dtype: int64\n",
      "\n",
      "DevEnviron\n",
      "False    11369\n",
      "True        29\n",
      "Name: DevEnviron, dtype: int64\n",
      "\n",
      "OpSys\n",
      "False    11364\n",
      "True        34\n",
      "Name: OpSys, dtype: int64\n",
      "\n",
      "Containers\n",
      "False    11316\n",
      "True        82\n",
      "Name: Containers, dtype: int64\n",
      "\n",
      "BlockchainOrg\n",
      "False    9076\n",
      "True     2322\n",
      "Name: BlockchainOrg, dtype: int64\n",
      "\n",
      "BlockchainIs\n",
      "False    8788\n",
      "True     2610\n",
      "Name: BlockchainIs, dtype: int64\n",
      "\n",
      "BetterLife\n",
      "False    11300\n",
      "True        98\n",
      "Name: BetterLife, dtype: int64\n",
      "\n",
      "ITperson\n",
      "False    11363\n",
      "True        35\n",
      "Name: ITperson, dtype: int64\n",
      "\n",
      "OffOn\n",
      "False    11360\n",
      "True        38\n",
      "Name: OffOn, dtype: int64\n",
      "\n",
      "SocialMedia\n",
      "False    11105\n",
      "True       293\n",
      "Name: SocialMedia, dtype: int64\n",
      "\n",
      "Extraversion\n",
      "False    11378\n",
      "True        20\n",
      "Name: Extraversion, dtype: int64\n",
      "\n",
      "ScreenName\n",
      "False    10891\n",
      "True       507\n",
      "Name: ScreenName, dtype: int64\n",
      "\n",
      "SOVisit1st\n",
      "False    11073\n",
      "True       325\n",
      "Name: SOVisit1st, dtype: int64\n",
      "\n",
      "SOVisitFreq\n",
      "False    11393\n",
      "True         5\n",
      "Name: SOVisitFreq, dtype: int64\n",
      "\n",
      "SOVisitTo\n",
      "False    11397\n",
      "True         1\n",
      "Name: SOVisitTo, dtype: int64\n",
      "\n",
      "SOFindAnswer\n",
      "False    11395\n",
      "True         3\n",
      "Name: SOFindAnswer, dtype: int64\n",
      "\n",
      "SOTimeSaved\n",
      "False    11348\n",
      "True        50\n",
      "Name: SOTimeSaved, dtype: int64\n",
      "\n",
      "SOHowMuchTime\n",
      "False    9481\n",
      "True     1917\n",
      "Name: SOHowMuchTime, dtype: int64\n",
      "\n",
      "SOAccount\n",
      "False    11397\n",
      "True         1\n",
      "Name: SOAccount, dtype: int64\n",
      "\n",
      "SOPartFreq\n",
      "False    10270\n",
      "True      1128\n",
      "Name: SOPartFreq, dtype: int64\n",
      "\n",
      "SOJobs\n",
      "False    11392\n",
      "True         6\n",
      "Name: SOJobs, dtype: int64\n",
      "\n",
      "EntTeams\n",
      "False    11393\n",
      "True         5\n",
      "Name: EntTeams, dtype: int64\n",
      "\n",
      "SOComm\n",
      "False    11398\n",
      "Name: SOComm, dtype: int64\n",
      "\n",
      "WelcomeChange\n",
      "False    11313\n",
      "True        85\n",
      "Name: WelcomeChange, dtype: int64\n",
      "\n",
      "SONewContent\n",
      "False    9433\n",
      "True     1965\n",
      "Name: SONewContent, dtype: int64\n",
      "\n",
      "Age\n",
      "False    11111\n",
      "True       287\n",
      "Name: Age, dtype: int64\n",
      "\n",
      "Gender\n",
      "False    11325\n",
      "True        73\n",
      "Name: Gender, dtype: int64\n",
      "\n",
      "Trans\n",
      "False    11275\n",
      "True       123\n",
      "Name: Trans, dtype: int64\n",
      "\n",
      "Sexuality\n",
      "False    10856\n",
      "True       542\n",
      "Name: Sexuality, dtype: int64\n",
      "\n",
      "Ethnicity\n",
      "False    10723\n",
      "True       675\n",
      "Name: Ethnicity, dtype: int64\n",
      "\n",
      "Dependents\n",
      "False    11258\n",
      "True       140\n",
      "Name: Dependents, dtype: int64\n",
      "\n",
      "SurveyLength\n",
      "False    11379\n",
      "True        19\n",
      "Name: SurveyLength, dtype: int64\n",
      "\n",
      "SurveyEase\n",
      "False    11384\n",
      "True        14\n",
      "Name: SurveyEase, dtype: int64\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Counting missing values for all columns\n",
    "\n",
    "for column in missing_data.columns.values.tolist():\n",
    "    print(column)\n",
    "    print (missing_data[column].value_counts())\n",
    "    print(\"\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imputing missing values\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find the  value counts for the column WorkLoc.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WorkLoc\n",
      "Office                                            6806\n",
      "Home                                              3589\n",
      "Other place, such as a coworking space or cafe     971\n",
      "Name: WorkLoc, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('WorkLoc')\n",
    "print (df['WorkLoc'].value_counts())  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We identify the value that is most frequent (majority) in the WorkLoc column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The most frequent value in WorkLoc column is Office"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we impute (replace) all the empty rows in the column WorkLoc with the value that you have identified as majority.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['WorkLoc'].replace(np.nan, 'Office', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After imputation there should ideally not be any empty rows in the WorkLoc column.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We verify if imputing was successful.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WorkLoc\n",
      "False    11398\n",
      "Name: WorkLoc, dtype: int64\n",
      "\n",
      "WorkLoc\n",
      "Office                                            6838\n",
      "Home                                              3589\n",
      "Other place, such as a coworking space or cafe     971\n",
      "Name: WorkLoc, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "missing_data = df.isnull()\n",
    "print('WorkLoc')\n",
    "print (missing_data['WorkLoc'].value_counts())  \n",
    "print('')\n",
    "print('WorkLoc')\n",
    "print (df['WorkLoc'].value_counts())  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the majority category under the column **Employment**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Employment\n",
      "Employed full-time    10968\n",
      "Employed part-time      430\n",
      "Name: Employment, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('Employment')\n",
    "print (df['Employment'].value_counts())  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under the column \" UndergradMajor\", which category has the minimum number of rows?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UndergradMajor\n",
      "Computer science, computer engineering, or software engineering          6953\n",
      "Information systems, information technology, or system administration     794\n",
      "Another engineering discipline (ex. civil, electrical, mechanical)        759\n",
      "Web development or web design                                             410\n",
      "A natural science (ex. biology, chemistry, physics)                       403\n",
      "Mathematics or statistics                                                 372\n",
      "A business discipline (ex. accounting, finance, marketing)                244\n",
      "A social science (ex. anthropology, psychology, political science)        210\n",
      "A humanities discipline (ex. literature, history, philosophy)             207\n",
      "Fine arts or performing arts (ex. graphic design, music, studio art)      161\n",
      "I never declared a major                                                  124\n",
      "A health science (ex. nursing, pharmacy, radiology)                        24\n",
      "Name: UndergradMajor, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('UndergradMajor')\n",
    "print (df['UndergradMajor'].value_counts())  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column ‘ConvertedComp’ contains the annual compensation of the survey respondents. What is the best approach to impute the missing values in this column?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The best approach to impute missing values is the median"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two columns in the dataset that talk about compensation.\n",
    "\n",
    "One is \"CompFreq\". This column shows how often a developer is paid (Yearly, Monthly, Weekly).\n",
    "\n",
    "The other is \"CompTotal\". This column talks about how much the developer is paid per Year, Month, or Week depending upon his/her \"CompFreq\".\n",
    "\n",
    "This makes it difficult to compare the total compensation of the developers.\n",
    "\n",
    "In this section we will create a new column called 'NormalizedAnnualCompensation' which contains the 'Annual Compensation' irrespective of the 'CompFreq'.\n",
    "\n",
    "Once this column is ready, it makes comparison of salaries easy.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with listing out the various categories in the column 'CompFreq'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CompFreq\n",
      "Yearly     6073\n",
      "Monthly    4788\n",
      "Weekly      331\n",
      "Name: CompFreq, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('CompFreq')\n",
    "print (df['CompFreq'].value_counts())  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a new column named 'NormalizedAnnualCompensation', using the below logic to arrive at the values for the column NormalizedAnnualCompensation.\n",
    "\n",
    "If the CompFreq is Yearly then use the exising value in CompTotal.\n",
    "\n",
    "If the CompFreq is Monthly then multiply the value in CompTotal with 12 (months in an year).\n",
    "\n",
    "If the CompFreq is Weekly then multiply the value in CompTotal with 52 (weeks in an year).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "      <th>NormalizedAnnualCompensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>22.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "      <td>61000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>New Zealand</td>\n",
       "      <td>No</td>\n",
       "      <td>Some college/university study without earning ...</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Bisexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "      <td>138000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Tech articles written by other developers;Cour...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "      <td>90000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "      <td>348000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Australia</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>29.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>Hispanic or Latino/Latina;Multiracial</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "      <td>90000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 86 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Respondent                      MainBranch Hobbyist  \\\n",
       "0           4  I am a developer by profession       No   \n",
       "1           9  I am a developer by profession      Yes   \n",
       "2          13  I am a developer by profession      Yes   \n",
       "3          16  I am a developer by profession      Yes   \n",
       "4          17  I am a developer by profession      Yes   \n",
       "\n",
       "                                         OpenSourcer  \\\n",
       "0                                              Never   \n",
       "1                         Once a month or more often   \n",
       "2  Less than once a month but more than once per ...   \n",
       "3                                              Never   \n",
       "4  Less than once a month but more than once per ...   \n",
       "\n",
       "                                          OpenSource          Employment  \\\n",
       "0  The quality of OSS and closed source software ...  Employed full-time   \n",
       "1  The quality of OSS and closed source software ...  Employed full-time   \n",
       "2  OSS is, on average, of HIGHER quality than pro...  Employed full-time   \n",
       "3  The quality of OSS and closed source software ...  Employed full-time   \n",
       "4  The quality of OSS and closed source software ...  Employed full-time   \n",
       "\n",
       "          Country Student                                            EdLevel  \\\n",
       "0   United States      No           Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "1     New Zealand      No  Some college/university study without earning ...   \n",
       "2   United States      No        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "3  United Kingdom      No        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "4       Australia      No           Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "\n",
       "                                      UndergradMajor  ...  \\\n",
       "0  Computer science, computer engineering, or sof...  ...   \n",
       "1  Computer science, computer engineering, or sof...  ...   \n",
       "2  Computer science, computer engineering, or sof...  ...   \n",
       "3                                                NaN  ...   \n",
       "4  Computer science, computer engineering, or sof...  ...   \n",
       "\n",
       "                                        SONewContent   Age Gender Trans  \\\n",
       "0  Tech articles written by other developers;Indu...  22.0    Man    No   \n",
       "1                                                NaN  23.0    Man    No   \n",
       "2  Tech articles written by other developers;Cour...  28.0    Man    No   \n",
       "3  Tech articles written by other developers;Indu...  26.0    Man    No   \n",
       "4  Tech articles written by other developers;Indu...  29.0    Man    No   \n",
       "\n",
       "                 Sexuality                              Ethnicity Dependents  \\\n",
       "0  Straight / Heterosexual           White or of European descent         No   \n",
       "1                 Bisexual           White or of European descent         No   \n",
       "2  Straight / Heterosexual           White or of European descent        Yes   \n",
       "3  Straight / Heterosexual           White or of European descent         No   \n",
       "4  Straight / Heterosexual  Hispanic or Latino/Latina;Multiracial         No   \n",
       "\n",
       "            SurveyLength                  SurveyEase  \\\n",
       "0  Appropriate in length                        Easy   \n",
       "1  Appropriate in length  Neither easy nor difficult   \n",
       "2  Appropriate in length                        Easy   \n",
       "3  Appropriate in length  Neither easy nor difficult   \n",
       "4  Appropriate in length                        Easy   \n",
       "\n",
       "  NormalizedAnnualCompensation  \n",
       "0                      61000.0  \n",
       "1                     138000.0  \n",
       "2                      90000.0  \n",
       "3                     348000.0  \n",
       "4                      90000.0  \n",
       "\n",
       "[5 rows x 86 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_annual_comp = []\n",
    "\n",
    "for i in range(len(df)):\n",
    "    \n",
    "    if df['CompFreq'].iloc[i] == 'Weekly':\n",
    "        norm_annual_comp.append(df['CompTotal'].iloc[i]*52)\n",
    "    elif df['CompFreq'].iloc[i] == 'Monthly':\n",
    "        norm_annual_comp.append(df['CompTotal'].iloc[i]*12)\n",
    "    else:\n",
    "        norm_annual_comp.append(df['CompTotal'].iloc[i]*1)\n",
    "\n",
    "df['NormalizedAnnualCompensation']= norm_annual_comp\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the median **NormalizedAnnualCompensation**?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1.058900e+04\n",
       "mean     6.133295e+06\n",
       "std      9.838157e+07\n",
       "min      0.000000e+00\n",
       "25%      5.200000e+04\n",
       "50%      1.000000e+05\n",
       "75%      3.600000e+05\n",
       "max      8.400000e+09\n",
       "Name: NormalizedAnnualCompensation, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['NormalizedAnnualCompensation'].describe()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}