02_DataManipulation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data manipulation with Numpy and Pandas\n",
    "\n",
    "Handling large data in Python is very elegant. In the simplest way you can use plain arrays. However, they are pretty slow. Numpy and Panda are two great libraries for dealing with tabular datasets. Numpy is used for homogenous n-dimensional data (matrices). Pandas is used for heterogenous tables (CSV, MS Excel tables). Pandas is internally based on Numpy. See http://scipy-lectures.github.io/ for a more detailed tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.74142332  0.11256776  0.72308845  0.72888687  0.32107623]\n",
      " [ 0.60544955  0.51283002  0.59559812  0.3668884   0.98974459]\n",
      " [ 0.37836809  0.26036109  0.20928187  0.36516898  0.4856415 ]]\n"
     ]
    }
   ],
   "source": [
    "#generating a random array\n",
    "X = np.random.random((3, 5))  # a 3 x 5 array\n",
    "\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.74142332240904052"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get a single element\n",
    "X[0, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.60544955,  0.51283002,  0.59559812,  0.3668884 ,  0.98974459])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get a row\n",
    "X[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.11256776,  0.51283002,  0.26036109])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get a column\n",
    "X[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.74142332,  0.60544955,  0.37836809],\n",
       "       [ 0.11256776,  0.51283002,  0.26036109],\n",
       "       [ 0.72308845,  0.59559812,  0.20928187],\n",
       "       [ 0.72888687,  0.3668884 ,  0.36516898],\n",
       "       [ 0.32107623,  0.98974459,  0.4856415 ]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#ransposing an array\n",
    "X.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, 5)\n",
      "[[ 0.74142332  0.11256776  0.72308845]\n",
      " [ 0.72888687  0.32107623  0.60544955]\n",
      " [ 0.51283002  0.59559812  0.3668884 ]\n",
      " [ 0.98974459  0.37836809  0.26036109]\n",
      " [ 0.20928187  0.36516898  0.4856415 ]]\n"
     ]
    }
   ],
   "source": [
    "print(X.shape)\n",
    "print(X.reshape(5, 3)) #change the layout of the matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 1 0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 0.72888687,  0.11256776,  0.74142332],\n",
       "       [ 0.3668884 ,  0.51283002,  0.60544955],\n",
       "       [ 0.36516898,  0.26036109,  0.37836809]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#indexing by an array of integers (fancy indexing)\n",
    "indices = np.array([3, 1, 0])\n",
    "print(indices)\n",
    "X[:, indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operations along an axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.74142332,  0.11256776,  0.72308845,  0.72888687,  0.32107623],\n",
       "       [ 0.60544955,  0.51283002,  0.59559812,  0.3668884 ,  0.98974459],\n",
       "       [ 0.37836809,  0.26036109,  0.20928187,  0.36516898,  0.4856415 ]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 5)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2.62704263,  3.07051067,  1.69882152])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(X, axis=1) # 1...columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.74142332,  0.51283002,  0.72308845,  0.72888687,  0.98974459])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(X, axis=0) # 0...rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A quick-ish introduction to Pandas\n",
    "\n",
    "based on http://pandas.pydata.org/pandas-docs/stable/10min.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>car</th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mazda RX4</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>160.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.90</td>\n",
       "      <td>2.620</td>\n",
       "      <td>16.46</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mazda RX4 Wag</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>160.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.90</td>\n",
       "      <td>2.875</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Datsun 710</td>\n",
       "      <td>22.8</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>93</td>\n",
       "      <td>3.85</td>\n",
       "      <td>2.320</td>\n",
       "      <td>18.61</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hornet 4 Drive</td>\n",
       "      <td>21.4</td>\n",
       "      <td>6</td>\n",
       "      <td>258.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.08</td>\n",
       "      <td>3.215</td>\n",
       "      <td>19.44</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Hornet Sportabout</td>\n",
       "      <td>18.7</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>175</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.440</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 car   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "0          Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4   \n",
       "1      Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4   \n",
       "2         Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4   \n",
       "3     Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3   \n",
       "4  Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3   \n",
       "\n",
       "   carb  \n",
       "0     4  \n",
       "1     4  \n",
       "2     1  \n",
       "3     1  \n",
       "4     2  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#use a standard dataset of heterogenous data\n",
    "cars = pd.read_csv('data/mtcars.csv')\n",
    "cars.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['car', 'mpg', 'cyl', 'disp', 'hp', 'drat', 'wt', 'qsec', 'vs', 'am',\n",
       "       'gear', 'carb'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#list all columns\n",
    "cars.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mazda RX4</th>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>160.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.90</td>\n",
       "      <td>2.620</td>\n",
       "      <td>16.46</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mazda RX4 Wag</th>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>160.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.90</td>\n",
       "      <td>2.875</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Datsun 710</th>\n",
       "      <td>22.8</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>93</td>\n",
       "      <td>3.85</td>\n",
       "      <td>2.320</td>\n",
       "      <td>18.61</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hornet 4 Drive</th>\n",
       "      <td>21.4</td>\n",
       "      <td>6</td>\n",
       "      <td>258.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.08</td>\n",
       "      <td>3.215</td>\n",
       "      <td>19.44</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hornet Sportabout</th>\n",
       "      <td>18.7</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>175</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.440</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "car                                                                          \n",
       "Mazda RX4          21.0    6  160.0  110  3.90  2.620  16.46   0   1     4   \n",
       "Mazda RX4 Wag      21.0    6  160.0  110  3.90  2.875  17.02   0   1     4   \n",
       "Datsun 710         22.8    4  108.0   93  3.85  2.320  18.61   1   1     4   \n",
       "Hornet 4 Drive     21.4    6  258.0  110  3.08  3.215  19.44   1   0     3   \n",
       "Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3   \n",
       "\n",
       "                   carb  \n",
       "car                      \n",
       "Mazda RX4             4  \n",
       "Mazda RX4 Wag         4  \n",
       "Datsun 710            1  \n",
       "Hornet 4 Drive        1  \n",
       "Hornet Sportabout     2  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we want to use the car as the \"primary key\" of a row\n",
    "cars.index = cars.pop('car')\n",
    "cars.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>32.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>20.090625</td>\n",
       "      <td>6.187500</td>\n",
       "      <td>230.721875</td>\n",
       "      <td>146.687500</td>\n",
       "      <td>3.596563</td>\n",
       "      <td>3.217250</td>\n",
       "      <td>17.848750</td>\n",
       "      <td>0.437500</td>\n",
       "      <td>0.406250</td>\n",
       "      <td>3.687500</td>\n",
       "      <td>2.8125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>6.026948</td>\n",
       "      <td>1.785922</td>\n",
       "      <td>123.938694</td>\n",
       "      <td>68.562868</td>\n",
       "      <td>0.534679</td>\n",
       "      <td>0.978457</td>\n",
       "      <td>1.786943</td>\n",
       "      <td>0.504016</td>\n",
       "      <td>0.498991</td>\n",
       "      <td>0.737804</td>\n",
       "      <td>1.6152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>10.400000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>71.100000</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>2.760000</td>\n",
       "      <td>1.513000</td>\n",
       "      <td>14.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>15.425000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>120.825000</td>\n",
       "      <td>96.500000</td>\n",
       "      <td>3.080000</td>\n",
       "      <td>2.581250</td>\n",
       "      <td>16.892500</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>19.200000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>196.300000</td>\n",
       "      <td>123.000000</td>\n",
       "      <td>3.695000</td>\n",
       "      <td>3.325000</td>\n",
       "      <td>17.710000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>22.800000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>326.000000</td>\n",
       "      <td>180.000000</td>\n",
       "      <td>3.920000</td>\n",
       "      <td>3.610000</td>\n",
       "      <td>18.900000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>33.900000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>472.000000</td>\n",
       "      <td>335.000000</td>\n",
       "      <td>4.930000</td>\n",
       "      <td>5.424000</td>\n",
       "      <td>22.900000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>8.0000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             mpg        cyl        disp          hp       drat         wt  \\\n",
       "count  32.000000  32.000000   32.000000   32.000000  32.000000  32.000000   \n",
       "mean   20.090625   6.187500  230.721875  146.687500   3.596563   3.217250   \n",
       "std     6.026948   1.785922  123.938694   68.562868   0.534679   0.978457   \n",
       "min    10.400000   4.000000   71.100000   52.000000   2.760000   1.513000   \n",
       "25%    15.425000   4.000000  120.825000   96.500000   3.080000   2.581250   \n",
       "50%    19.200000   6.000000  196.300000  123.000000   3.695000   3.325000   \n",
       "75%    22.800000   8.000000  326.000000  180.000000   3.920000   3.610000   \n",
       "max    33.900000   8.000000  472.000000  335.000000   4.930000   5.424000   \n",
       "\n",
       "            qsec         vs         am       gear     carb  \n",
       "count  32.000000  32.000000  32.000000  32.000000  32.0000  \n",
       "mean   17.848750   0.437500   0.406250   3.687500   2.8125  \n",
       "std     1.786943   0.504016   0.498991   0.737804   1.6152  \n",
       "min    14.500000   0.000000   0.000000   3.000000   1.0000  \n",
       "25%    16.892500   0.000000   0.000000   3.000000   2.0000  \n",
       "50%    17.710000   0.000000   0.000000   4.000000   2.0000  \n",
       "75%    18.900000   1.000000   1.000000   4.000000   4.0000  \n",
       "max    22.900000   1.000000   1.000000   5.000000   8.0000  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#describe our dataset\n",
    "cars.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AMC Javelin</th>\n",
       "      <td>15.2</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.435</td>\n",
       "      <td>17.30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cadillac Fleetwood</th>\n",
       "      <td>10.4</td>\n",
       "      <td>8</td>\n",
       "      <td>472.0</td>\n",
       "      <td>205</td>\n",
       "      <td>2.93</td>\n",
       "      <td>5.250</td>\n",
       "      <td>17.98</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Camaro Z28</th>\n",
       "      <td>13.3</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>245</td>\n",
       "      <td>3.73</td>\n",
       "      <td>3.840</td>\n",
       "      <td>15.41</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chrysler Imperial</th>\n",
       "      <td>14.7</td>\n",
       "      <td>8</td>\n",
       "      <td>440.0</td>\n",
       "      <td>230</td>\n",
       "      <td>3.23</td>\n",
       "      <td>5.345</td>\n",
       "      <td>17.42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Datsun 710</th>\n",
       "      <td>22.8</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>93</td>\n",
       "      <td>3.85</td>\n",
       "      <td>2.320</td>\n",
       "      <td>18.61</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "car                                                                           \n",
       "AMC Javelin         15.2    8  304.0  150  3.15  3.435  17.30   0   0     3   \n",
       "Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3   \n",
       "Camaro Z28          13.3    8  350.0  245  3.73  3.840  15.41   0   0     3   \n",
       "Chrysler Imperial   14.7    8  440.0  230  3.23  5.345  17.42   0   0     3   \n",
       "Datsun 710          22.8    4  108.0   93  3.85  2.320  18.61   1   1     4   \n",
       "\n",
       "                    carb  \n",
       "car                       \n",
       "AMC Javelin            2  \n",
       "Cadillac Fleetwood     4  \n",
       "Camaro Z28             4  \n",
       "Chrysler Imperial      4  \n",
       "Datsun 710             1  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.sort_index(inplace=True)\n",
    "cars.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Cadillac Fleetwood</th>\n",
       "      <td>10.4</td>\n",
       "      <td>8</td>\n",
       "      <td>472.0</td>\n",
       "      <td>205</td>\n",
       "      <td>2.93</td>\n",
       "      <td>5.250</td>\n",
       "      <td>17.98</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lincoln Continental</th>\n",
       "      <td>10.4</td>\n",
       "      <td>8</td>\n",
       "      <td>460.0</td>\n",
       "      <td>215</td>\n",
       "      <td>3.00</td>\n",
       "      <td>5.424</td>\n",
       "      <td>17.82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Camaro Z28</th>\n",
       "      <td>13.3</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>245</td>\n",
       "      <td>3.73</td>\n",
       "      <td>3.840</td>\n",
       "      <td>15.41</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Duster 360</th>\n",
       "      <td>14.3</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>245</td>\n",
       "      <td>3.21</td>\n",
       "      <td>3.570</td>\n",
       "      <td>15.84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chrysler Imperial</th>\n",
       "      <td>14.7</td>\n",
       "      <td>8</td>\n",
       "      <td>440.0</td>\n",
       "      <td>230</td>\n",
       "      <td>3.23</td>\n",
       "      <td>5.345</td>\n",
       "      <td>17.42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maserati Bora</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>301.0</td>\n",
       "      <td>335</td>\n",
       "      <td>3.54</td>\n",
       "      <td>3.570</td>\n",
       "      <td>14.60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AMC Javelin</th>\n",
       "      <td>15.2</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.435</td>\n",
       "      <td>17.30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Merc 450SLC</th>\n",
       "      <td>15.2</td>\n",
       "      <td>8</td>\n",
       "      <td>275.8</td>\n",
       "      <td>180</td>\n",
       "      <td>3.07</td>\n",
       "      <td>3.780</td>\n",
       "      <td>18.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dodge Challenger</th>\n",
       "      <td>15.5</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>2.76</td>\n",
       "      <td>3.520</td>\n",
       "      <td>16.87</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ford Pantera L</th>\n",
       "      <td>15.8</td>\n",
       "      <td>8</td>\n",
       "      <td>351.0</td>\n",
       "      <td>264</td>\n",
       "      <td>4.22</td>\n",
       "      <td>3.170</td>\n",
       "      <td>14.50</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Merc 450SE</th>\n",
       "      <td>16.4</td>\n",
       "      <td>8</td>\n",
       "      <td>275.8</td>\n",
       "      <td>180</td>\n",
       "      <td>3.07</td>\n",
       "      <td>4.070</td>\n",
       "      <td>17.40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Merc 450SL</th>\n",
       "      <td>17.3</td>\n",
       "      <td>8</td>\n",
       "      <td>275.8</td>\n",
       "      <td>180</td>\n",
       "      <td>3.07</td>\n",
       "      <td>3.730</td>\n",
       "      <td>17.60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Merc 280C</th>\n",
       "      <td>17.8</td>\n",
       "      <td>6</td>\n",
       "      <td>167.6</td>\n",
       "      <td>123</td>\n",
       "      <td>3.92</td>\n",
       "      <td>3.440</td>\n",
       "      <td>18.90</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Valiant</th>\n",
       "      <td>18.1</td>\n",
       "      <td>6</td>\n",
       "      <td>225.0</td>\n",
       "      <td>105</td>\n",
       "      <td>2.76</td>\n",
       "      <td>3.460</td>\n",
       "      <td>20.22</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hornet Sportabout</th>\n",
       "      <td>18.7</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>175</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.440</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "car                                                                            \n",
       "Cadillac Fleetwood   10.4    8  472.0  205  2.93  5.250  17.98   0   0     3   \n",
       "Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3   \n",
       "Camaro Z28           13.3    8  350.0  245  3.73  3.840  15.41   0   0     3   \n",
       "Duster 360           14.3    8  360.0  245  3.21  3.570  15.84   0   0     3   \n",
       "Chrysler Imperial    14.7    8  440.0  230  3.23  5.345  17.42   0   0     3   \n",
       "Maserati Bora        15.0    8  301.0  335  3.54  3.570  14.60   0   1     5   \n",
       "AMC Javelin          15.2    8  304.0  150  3.15  3.435  17.30   0   0     3   \n",
       "Merc 450SLC          15.2    8  275.8  180  3.07  3.780  18.00   0   0     3   \n",
       "Dodge Challenger     15.5    8  318.0  150  2.76  3.520  16.87   0   0     3   \n",
       "Ford Pantera L       15.8    8  351.0  264  4.22  3.170  14.50   0   1     5   \n",
       "Merc 450SE           16.4    8  275.8  180  3.07  4.070  17.40   0   0     3   \n",
       "Merc 450SL           17.3    8  275.8  180  3.07  3.730  17.60   0   0     3   \n",
       "Merc 280C            17.8    6  167.6  123  3.92  3.440  18.90   1   0     4   \n",
       "Valiant              18.1    6  225.0  105  2.76  3.460  20.22   1   0     3   \n",
       "Hornet Sportabout    18.7    8  360.0  175  3.15  3.440  17.02   0   0     3   \n",
       "\n",
       "                     carb  \n",
       "car                        \n",
       "Cadillac Fleetwood      4  \n",
       "Lincoln Continental     4  \n",
       "Camaro Z28              4  \n",
       "Duster 360              4  \n",
       "Chrysler Imperial       4  \n",
       "Maserati Bora           8  \n",
       "AMC Javelin             2  \n",
       "Merc 450SLC             3  \n",
       "Dodge Challenger        2  \n",
       "Ford Pantera L          4  \n",
       "Merc 450SE              3  \n",
       "Merc 450SL              3  \n",
       "Merc 280C               4  \n",
       "Valiant                 1  \n",
       "Hornet Sportabout       2  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.sort_values('mpg').head(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Maserati Bora</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>301.0</td>\n",
       "      <td>335</td>\n",
       "      <td>3.54</td>\n",
       "      <td>3.570</td>\n",
       "      <td>14.60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ford Pantera L</th>\n",
       "      <td>15.8</td>\n",
       "      <td>8</td>\n",
       "      <td>351.0</td>\n",
       "      <td>264</td>\n",
       "      <td>4.22</td>\n",
       "      <td>3.170</td>\n",
       "      <td>14.50</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Camaro Z28</th>\n",
       "      <td>13.3</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>245</td>\n",
       "      <td>3.73</td>\n",
       "      <td>3.840</td>\n",
       "      <td>15.41</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Duster 360</th>\n",
       "      <td>14.3</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>245</td>\n",
       "      <td>3.21</td>\n",
       "      <td>3.570</td>\n",
       "      <td>15.84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chrysler Imperial</th>\n",
       "      <td>14.7</td>\n",
       "      <td>8</td>\n",
       "      <td>440.0</td>\n",
       "      <td>230</td>\n",
       "      <td>3.23</td>\n",
       "      <td>5.345</td>\n",
       "      <td>17.42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "car                                                                          \n",
       "Maserati Bora      15.0    8  301.0  335  3.54  3.570  14.60   0   1     5   \n",
       "Ford Pantera L     15.8    8  351.0  264  4.22  3.170  14.50   0   1     5   \n",
       "Camaro Z28         13.3    8  350.0  245  3.73  3.840  15.41   0   0     3   \n",
       "Duster 360         14.3    8  360.0  245  3.21  3.570  15.84   0   0     3   \n",
       "Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3   \n",
       "\n",
       "                   carb  \n",
       "car                      \n",
       "Maserati Bora         8  \n",
       "Ford Pantera L        4  \n",
       "Camaro Z28            4  \n",
       "Duster 360            4  \n",
       "Chrysler Imperial     4  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.sort_values('hp', ascending=False).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selection\n",
    "\n",
    "**Note**: While many of the NumPy accessor methods work on DataFrames, you can also use the optimized pandas-specific data accessor methods, `.at`, `.iat`, `.loc`, `.iloc` and `.ix`.\n",
    "\n",
    "See the [Indexing section](http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing) and below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "car\n",
       "AMC Javelin            15.2\n",
       "Cadillac Fleetwood     10.4\n",
       "Camaro Z28             13.3\n",
       "Chrysler Imperial      14.7\n",
       "Datsun 710             22.8\n",
       "Dodge Challenger       15.5\n",
       "Duster 360             14.3\n",
       "Ferrari Dino           19.7\n",
       "Fiat 128               32.4\n",
       "Fiat X1-9              27.3\n",
       "Ford Pantera L         15.8\n",
       "Honda Civic            30.4\n",
       "Hornet 4 Drive         21.4\n",
       "Hornet Sportabout      18.7\n",
       "Lincoln Continental    10.4\n",
       "Lotus Europa           30.4\n",
       "Maserati Bora          15.0\n",
       "Mazda RX4              21.0\n",
       "Mazda RX4 Wag          21.0\n",
       "Merc 230               22.8\n",
       "Merc 240D              24.4\n",
       "Merc 280               19.2\n",
       "Merc 280C              17.8\n",
       "Merc 450SE             16.4\n",
       "Merc 450SL             17.3\n",
       "Merc 450SLC            15.2\n",
       "Pontiac Firebird       19.2\n",
       "Porsche 914-2          26.0\n",
       "Toyota Corolla         33.9\n",
       "Toyota Corona          21.5\n",
       "Valiant                18.1\n",
       "Volvo 142E             21.4\n",
       "Name: mpg, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#single column\n",
    "cars['mpg']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Camaro Z28</th>\n",
       "      <td>13.3</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>245</td>\n",
       "      <td>3.73</td>\n",
       "      <td>3.840</td>\n",
       "      <td>15.41</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chrysler Imperial</th>\n",
       "      <td>14.7</td>\n",
       "      <td>8</td>\n",
       "      <td>440.0</td>\n",
       "      <td>230</td>\n",
       "      <td>3.23</td>\n",
       "      <td>5.345</td>\n",
       "      <td>17.42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Datsun 710</th>\n",
       "      <td>22.8</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>93</td>\n",
       "      <td>3.85</td>\n",
       "      <td>2.320</td>\n",
       "      <td>18.61</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "car                                                                          \n",
       "Camaro Z28         13.3    8  350.0  245  3.73  3.840  15.41   0   0     3   \n",
       "Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3   \n",
       "Datsun 710         22.8    4  108.0   93  3.85  2.320  18.61   1   1     4   \n",
       "\n",
       "                   carb  \n",
       "car                      \n",
       "Camaro Z28            4  \n",
       "Chrysler Imperial     4  \n",
       "Datsun 710            1  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#or a slice of rows\n",
    "cars[2:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Fiat 128</th>\n",
       "      <td>32.4</td>\n",
       "      <td>4</td>\n",
       "      <td>78.7</td>\n",
       "      <td>66</td>\n",
       "      <td>4.08</td>\n",
       "      <td>2.200</td>\n",
       "      <td>19.47</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fiat X1-9</th>\n",
       "      <td>27.3</td>\n",
       "      <td>4</td>\n",
       "      <td>79.0</td>\n",
       "      <td>66</td>\n",
       "      <td>4.08</td>\n",
       "      <td>1.935</td>\n",
       "      <td>18.90</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ford Pantera L</th>\n",
       "      <td>15.8</td>\n",
       "      <td>8</td>\n",
       "      <td>351.0</td>\n",
       "      <td>264</td>\n",
       "      <td>4.22</td>\n",
       "      <td>3.170</td>\n",
       "      <td>14.50</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Honda Civic</th>\n",
       "      <td>30.4</td>\n",
       "      <td>4</td>\n",
       "      <td>75.7</td>\n",
       "      <td>52</td>\n",
       "      <td>4.93</td>\n",
       "      <td>1.615</td>\n",
       "      <td>18.52</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hornet 4 Drive</th>\n",
       "      <td>21.4</td>\n",
       "      <td>6</td>\n",
       "      <td>258.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.08</td>\n",
       "      <td>3.215</td>\n",
       "      <td>19.44</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hornet Sportabout</th>\n",
       "      <td>18.7</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>175</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.440</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lincoln Continental</th>\n",
       "      <td>10.4</td>\n",
       "      <td>8</td>\n",
       "      <td>460.0</td>\n",
       "      <td>215</td>\n",
       "      <td>3.00</td>\n",
       "      <td>5.424</td>\n",
       "      <td>17.82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lotus Europa</th>\n",
       "      <td>30.4</td>\n",
       "      <td>4</td>\n",
       "      <td>95.1</td>\n",
       "      <td>113</td>\n",
       "      <td>3.77</td>\n",
       "      <td>1.513</td>\n",
       "      <td>16.90</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "car                                                                            \n",
       "Fiat 128             32.4    4   78.7   66  4.08  2.200  19.47   1   1     4   \n",
       "Fiat X1-9            27.3    4   79.0   66  4.08  1.935  18.90   1   1     4   \n",
       "Ford Pantera L       15.8    8  351.0  264  4.22  3.170  14.50   0   1     5   \n",
       "Honda Civic          30.4    4   75.7   52  4.93  1.615  18.52   1   1     4   \n",
       "Hornet 4 Drive       21.4    6  258.0  110  3.08  3.215  19.44   1   0     3   \n",
       "Hornet Sportabout    18.7    8  360.0  175  3.15  3.440  17.02   0   0     3   \n",
       "Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3   \n",
       "Lotus Europa         30.4    4   95.1  113  3.77  1.513  16.90   1   1     5   \n",
       "\n",
       "                     carb  \n",
       "car                        \n",
       "Fiat 128                1  \n",
       "Fiat X1-9               1  \n",
       "Ford Pantera L          4  \n",
       "Honda Civic             2  \n",
       "Hornet 4 Drive          1  \n",
       "Hornet Sportabout       2  \n",
       "Lincoln Continental     4  \n",
       "Lotus Europa            2  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#by label = primary key\n",
    "cars.loc['Fiat 128':'Lotus Europa']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mpg      14.700\n",
       "cyl       8.000\n",
       "disp    440.000\n",
       "hp      230.000\n",
       "drat      3.230\n",
       "wt        5.345\n",
       "qsec     17.420\n",
       "vs        0.000\n",
       "am        0.000\n",
       "gear      3.000\n",
       "carb      4.000\n",
       "Name: Chrysler Imperial, dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#selection by position\n",
    "cars.iloc[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Chrysler Imperial</th>\n",
       "      <td>14.7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Datsun 710</th>\n",
       "      <td>22.8</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    mpg  cyl\n",
       "car                         \n",
       "Chrysler Imperial  14.7    8\n",
       "Datsun 710         22.8    4"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.iloc[3:5, 0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AMC Javelin</th>\n",
       "      <td>15.2</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.435</td>\n",
       "      <td>17.30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cadillac Fleetwood</th>\n",
       "      <td>10.4</td>\n",
       "      <td>8</td>\n",
       "      <td>472.0</td>\n",
       "      <td>205</td>\n",
       "      <td>2.93</td>\n",
       "      <td>5.250</td>\n",
       "      <td>17.98</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Camaro Z28</th>\n",
       "      <td>13.3</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>245</td>\n",
       "      <td>3.73</td>\n",
       "      <td>3.840</td>\n",
       "      <td>15.41</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chrysler Imperial</th>\n",
       "      <td>14.7</td>\n",
       "      <td>8</td>\n",
       "      <td>440.0</td>\n",
       "      <td>230</td>\n",
       "      <td>3.23</td>\n",
       "      <td>5.345</td>\n",
       "      <td>17.42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dodge Challenger</th>\n",
       "      <td>15.5</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>2.76</td>\n",
       "      <td>3.520</td>\n",
       "      <td>16.87</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Duster 360</th>\n",
       "      <td>14.3</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>245</td>\n",
       "      <td>3.21</td>\n",
       "      <td>3.570</td>\n",
       "      <td>15.84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ford Pantera L</th>\n",
       "      <td>15.8</td>\n",
       "      <td>8</td>\n",
       "      <td>351.0</td>\n",
       "      <td>264</td>\n",
       "      <td>4.22</td>\n",
       "      <td>3.170</td>\n",
       "      <td>14.50</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hornet Sportabout</th>\n",
       "      <td>18.7</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>175</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.440</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lincoln Continental</th>\n",
       "      <td>10.4</td>\n",
       "      <td>8</td>\n",
       "      <td>460.0</td>\n",
       "      <td>215</td>\n",
       "      <td>3.00</td>\n",
       "      <td>5.424</td>\n",
       "      <td>17.82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maserati Bora</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>301.0</td>\n",
       "      <td>335</td>\n",
       "      <td>3.54</td>\n",
       "      <td>3.570</td>\n",
       "      <td>14.60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Merc 450SE</th>\n",
       "      <td>16.4</td>\n",
       "      <td>8</td>\n",
       "      <td>275.8</td>\n",
       "      <td>180</td>\n",
       "      <td>3.07</td>\n",
       "      <td>4.070</td>\n",
       "      <td>17.40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Merc 450SL</th>\n",
       "      <td>17.3</td>\n",
       "      <td>8</td>\n",
       "      <td>275.8</td>\n",
       "      <td>180</td>\n",
       "      <td>3.07</td>\n",
       "      <td>3.730</td>\n",
       "      <td>17.60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Merc 450SLC</th>\n",
       "      <td>15.2</td>\n",
       "      <td>8</td>\n",
       "      <td>275.8</td>\n",
       "      <td>180</td>\n",
       "      <td>3.07</td>\n",
       "      <td>3.780</td>\n",
       "      <td>18.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pontiac Firebird</th>\n",
       "      <td>19.2</td>\n",
       "      <td>8</td>\n",
       "      <td>400.0</td>\n",
       "      <td>175</td>\n",
       "      <td>3.08</td>\n",
       "      <td>3.845</td>\n",
       "      <td>17.05</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "car                                                                            \n",
       "AMC Javelin          15.2    8  304.0  150  3.15  3.435  17.30   0   0     3   \n",
       "Cadillac Fleetwood   10.4    8  472.0  205  2.93  5.250  17.98   0   0     3   \n",
       "Camaro Z28           13.3    8  350.0  245  3.73  3.840  15.41   0   0     3   \n",
       "Chrysler Imperial    14.7    8  440.0  230  3.23  5.345  17.42   0   0     3   \n",
       "Dodge Challenger     15.5    8  318.0  150  2.76  3.520  16.87   0   0     3   \n",
       "Duster 360           14.3    8  360.0  245  3.21  3.570  15.84   0   0     3   \n",
       "Ford Pantera L       15.8    8  351.0  264  4.22  3.170  14.50   0   1     5   \n",
       "Hornet Sportabout    18.7    8  360.0  175  3.15  3.440  17.02   0   0     3   \n",
       "Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3   \n",
       "Maserati Bora        15.0    8  301.0  335  3.54  3.570  14.60   0   1     5   \n",
       "Merc 450SE           16.4    8  275.8  180  3.07  4.070  17.40   0   0     3   \n",
       "Merc 450SL           17.3    8  275.8  180  3.07  3.730  17.60   0   0     3   \n",
       "Merc 450SLC          15.2    8  275.8  180  3.07  3.780  18.00   0   0     3   \n",
       "Pontiac Firebird     19.2    8  400.0  175  3.08  3.845  17.05   0   0     3   \n",
       "\n",
       "                     carb  \n",
       "car                        \n",
       "AMC Javelin             2  \n",
       "Cadillac Fleetwood      4  \n",
       "Camaro Z28              4  \n",
       "Chrysler Imperial       4  \n",
       "Dodge Challenger        2  \n",
       "Duster 360              4  \n",
       "Ford Pantera L          4  \n",
       "Hornet Sportabout       2  \n",
       "Lincoln Continental     4  \n",
       "Maserati Bora           8  \n",
       "Merc 450SE              3  \n",
       "Merc 450SL              3  \n",
       "Merc 450SLC             3  \n",
       "Pontiac Firebird        2  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars[cars.cyl > 6] # more than 6 cylinders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cars_na = pd.read_csv('data/mtcars_with_nas.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>car</th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     car    mpg    cyl   disp     hp   drat     wt   qsec     vs     am  \\\n",
       "0  False  False  False  False  False  False  False  False  False  False   \n",
       "1  False  False  False   True  False  False  False  False  False  False   \n",
       "2  False  False  False  False  False  False  False  False  False  False   \n",
       "3  False  False  False  False  False  False  False  False  False  False   \n",
       "\n",
       "    gear   carb  \n",
       "0  False  False  \n",
       "1  False  False  \n",
       "2  False  False  \n",
       "3  False  False  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_na.isnull().head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>car</th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mazda RX4</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>160.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.90</td>\n",
       "      <td>2.620</td>\n",
       "      <td>16.46</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mazda RX4 Wag</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.90</td>\n",
       "      <td>2.875</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Datsun 710</td>\n",
       "      <td>22.8</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>93</td>\n",
       "      <td>3.85</td>\n",
       "      <td>2.320</td>\n",
       "      <td>18.61</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hornet 4 Drive</td>\n",
       "      <td>21.4</td>\n",
       "      <td>6</td>\n",
       "      <td>258.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.08</td>\n",
       "      <td>3.215</td>\n",
       "      <td>19.44</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              car   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "0       Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4   \n",
       "1   Mazda RX4 Wag  21.0    6    1.0  110  3.90  2.875  17.02   0   1     4   \n",
       "2      Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4   \n",
       "3  Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3   \n",
       "\n",
       "   carb  \n",
       "0     4  \n",
       "1     4  \n",
       "2     1  \n",
       "3     1  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#fill with a default value\n",
    "cars_na.fillna(1).head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32, 12)\n",
      "(29, 12)\n",
      "(32, 10)\n"
     ]
    }
   ],
   "source": [
    "#or drop the rows\n",
    "\n",
    "print(cars_na.shape)\n",
    "#drop rows with na values\n",
    "print(cars_na.dropna().shape)\n",
    "#drop columns with na values\n",
    "print(cars_na.dropna(axis=1).shape)\n",
    "\n",
    "#see also http://pandas.pydata.org/pandas-docs/stable/missing_data.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistics\n",
    "\n",
    "In addition to `.describe()` there are plenty of other staticial measures and aggregation methods in Pandas/Numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mpg      20.090625\n",
       "cyl       6.187500\n",
       "disp    230.721875\n",
       "hp      146.687500\n",
       "drat      3.596562\n",
       "wt        3.217250\n",
       "qsec     17.848750\n",
       "vs        0.437500\n",
       "am        0.406250\n",
       "gear      3.687500\n",
       "carb      2.812500\n",
       "dtype: float64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#stats\n",
    "cars.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "car\n",
       "AMC Javelin            46.007727\n",
       "Cadillac Fleetwood     66.232727\n",
       "Camaro Z28             58.752727\n",
       "Chrysler Imperial      65.972273\n",
       "Datsun 710             23.598182\n",
       "Dodge Challenger       47.240909\n",
       "Duster 360             59.720000\n",
       "Ferrari Dino           34.508182\n",
       "Fiat 128               19.440909\n",
       "Fiat X1-9              18.928636\n",
       "Ford Pantera L         60.971818\n",
       "Honda Civic            17.742273\n",
       "Hornet 4 Drive         38.739545\n",
       "Hornet Sportabout      53.664545\n",
       "Lincoln Continental    66.058545\n",
       "Lotus Europa           24.880273\n",
       "Maserati Bora          63.155455\n",
       "Mazda RX4              29.907273\n",
       "Mazda RX4 Wag          29.981364\n",
       "Merc 230               27.233636\n",
       "Merc 240D              24.634545\n",
       "Merc 280               31.860000\n",
       "Merc 280C              31.787273\n",
       "Merc 450SE             46.430909\n",
       "Merc 450SL             46.500000\n",
       "Merc 450SLC            46.350000\n",
       "Pontiac Firebird       57.379545\n",
       "Porsche 914-2          24.779091\n",
       "Toyota Corolla         18.814091\n",
       "Toyota Corona          24.888636\n",
       "Valiant                35.049091\n",
       "Volvo 142E             26.262727\n",
       "dtype: float64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.mean(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cyl</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>26.663636</td>\n",
       "      <td>105.136364</td>\n",
       "      <td>82.636364</td>\n",
       "      <td>4.070909</td>\n",
       "      <td>2.285727</td>\n",
       "      <td>19.137273</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>0.727273</td>\n",
       "      <td>4.090909</td>\n",
       "      <td>1.545455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>19.742857</td>\n",
       "      <td>183.314286</td>\n",
       "      <td>122.285714</td>\n",
       "      <td>3.585714</td>\n",
       "      <td>3.117143</td>\n",
       "      <td>17.977143</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>3.857143</td>\n",
       "      <td>3.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>15.100000</td>\n",
       "      <td>353.100000</td>\n",
       "      <td>209.214286</td>\n",
       "      <td>3.229286</td>\n",
       "      <td>3.999214</td>\n",
       "      <td>16.772143</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>3.285714</td>\n",
       "      <td>3.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           mpg        disp          hp      drat        wt       qsec  \\\n",
       "cyl                                                                     \n",
       "4    26.663636  105.136364   82.636364  4.070909  2.285727  19.137273   \n",
       "6    19.742857  183.314286  122.285714  3.585714  3.117143  17.977143   \n",
       "8    15.100000  353.100000  209.214286  3.229286  3.999214  16.772143   \n",
       "\n",
       "           vs        am      gear      carb  \n",
       "cyl                                          \n",
       "4    0.909091  0.727273  4.090909  1.545455  \n",
       "6    0.571429  0.428571  3.857143  3.428571  \n",
       "8    0.000000  0.142857  3.285714  3.500000  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#grouping\n",
    "cars.groupby('cyl').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>qsec</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cyl</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>26.663636</td>\n",
       "      <td>16.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>19.742857</td>\n",
       "      <td>15.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>15.100000</td>\n",
       "      <td>14.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           mpg  qsec\n",
       "cyl                 \n",
       "4    26.663636  16.7\n",
       "6    19.742857  15.5\n",
       "8    15.100000  14.5"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#grouping different aggregation methods\n",
    "cars.groupby('cyl').agg({ 'mpg': 'mean', 'qsec': 'min'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TASKS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#loading gapminder data (taken from https://github.com/jennybc/gapminder)\n",
    "#file located at 'data/gapminder-unfiltered.tsv' it uses tabular character as separator\n",
    "#use the first column as index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#what are the columns of this dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#what is the maximal year contained?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#just select all data of the year 2007"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#locate Austria and print it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#list the top 10 countries by life expectancy (lifeExp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#what is the total population (pop) per continent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next\n",
    "\n",
    "[(Interactive) Plotting using Matplotlib and Seaborn](03_Plotting.ipynb)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}