PythonPandasCookbook1.1.ipynb

{
 "metadata": {
  "name": "PythonPandasCookbook1.1"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Python for Data Analysis Lightning Tutorials\n",
      "## Pandas Cookbook Series\n",
      "<img src=\"files/images/feynman-bongo.jpg\" width =100, height=200>\n",
      "\n",
      "\n",
      "**Python for Data Analysis Lightning Tutorials** is a series of tutorials in Data Analysis, Statistics, and Graphics using Python. The **Pandas Cookbook** series of tutorials provides recipes for common tasks and moves on to more advanced topics in statistics and time series analysis. \n",
      "\n",
      "<i>Created by Alfred Essa</i>, Aug 8, 2013 \n",
      "\n",
      "Note: IPython Notebook and Data files can be found at my Github Site: github/alfredessa\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Chapter 1: Data Structures\n",
      "\n",
      "### 1.1 Problem. How can I create a Series object in Pandas?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.11 What is a Series?\n",
      "\n",
      "The **Series** data structure in Pandas is a <i>one-dimensional labeled array</i>. \n",
      "\n",
      "* Data in the array can be of any type (integers, strings, floating point numbers, Python objects, etc.). \n",
      "* Data within the array is homogeneous\n",
      "    \n",
      "<img src=\"files/images/series1.jpg\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.12 Series is ndarray-like and dict-like\n",
      "\n",
      "Pandas Series objects are amphibian in character, exhibiting both <i>ndarray-like</i> and <i>dict-like properties</i>. See **Discussion** section below. \n",
      "\n",
      "<img src=\"files/images/amphibian.jpg\" width=175, height =75>\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.13 How can I Create a Series?\n",
      "\n",
      "The basic method to create a Series:\n",
      "    \n",
      "    - s = Series(data, index=index)\n",
      "\n",
      "Here, data can be different things, including:\n",
      "    \n",
      "    - a list\n",
      "    - an array\n",
      "    - a dictionary"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.14 Preliminaries - import pandas library"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas as pd\n",
      "import numpy as np"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 118
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.15 Example 1 : Create a Basic Series Object"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# series constructor with data as a list of integers\n",
      "s1 = pd.Series([33, 19, 15, 89, 11, -5, 9])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 119
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# the default index, if not specified in the Series constructor, is a series of integers\n",
      "s1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 120,
       "text": [
        "0    33\n",
        "1    19\n",
        "2    15\n",
        "3    89\n",
        "4    11\n",
        "5    -5\n",
        "6     9\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 120
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# type of series is pandas series\n",
      "type(s1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 121,
       "text": [
        "pandas.core.series.Series"
       ]
      }
     ],
     "prompt_number": 121
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# retrieve the values of the series\n",
      "s1.values\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 122,
       "text": [
        "array([33, 19, 15, 89, 11, -5,  9])"
       ]
      }
     ],
     "prompt_number": 122
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# type of data values is NumPy ndarray\n",
      "type(s1.values)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 123,
       "text": [
        "numpy.ndarray"
       ]
      }
     ],
     "prompt_number": 123
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# retrieve the indices of the array\n",
      "s1.index"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 124,
       "text": [
        "Int64Index([0, 1, 2, 3, 4, 5, 6], dtype=int64)"
       ]
      }
     ],
     "prompt_number": 124
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# think of a series as a mapping from index to values\n",
      "s1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 125,
       "text": [
        "0    33\n",
        "1    19\n",
        "2    15\n",
        "3    89\n",
        "4    11\n",
        "5    -5\n",
        "6     9\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 125
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<img src=\"files/images/series2.jpg\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.16 Example 2: Creating a Series Object with Meaningful Labels"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# define the data and index as lists\n",
      "data1 = [33, 19, 15, 89, 11, -5, 9]\n",
      "index1 = ['Mon','Tue','Wed','Thu','Fri','Sat','Sun']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 126
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# create series \n",
      "s2 = pd.Series(data1, index=index1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 127
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s2"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 128,
       "text": [
        "Mon    33\n",
        "Tue    19\n",
        "Wed    15\n",
        "Thu    89\n",
        "Fri    11\n",
        "Sat    -5\n",
        "Sun     9\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 128
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# verify index \n",
      "s2.index"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 129,
       "text": [
        "Index([u'Mon', u'Tue', u'Wed', u'Thu', u'Fri', u'Sat', u'Sun'], dtype=object)"
       ]
      }
     ],
     "prompt_number": 129
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<img src=\"files/images/series3.jpg\">"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# we can also give meaningful labels to the series data and the index\n",
      "\n",
      "s2.name='Daily Temperatures'\n",
      "s2.index.name='Weekday'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 130
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s2"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 131,
       "text": [
        "Weekday\n",
        "Mon        33\n",
        "Tue        19\n",
        "Wed        15\n",
        "Thu        89\n",
        "Fri        11\n",
        "Sat        -5\n",
        "Sun         9\n",
        "Name: Daily Temperatures, dtype: int64"
       ]
      }
     ],
     "prompt_number": 131
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.17 Example 3: Data in a Series is homogeneous"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# the second data element in the list is a float\n",
      "data2 = [33, 19.3, 15, 89, 11, -5, 9]\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s3 = pd.Series(data2, index=index1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 132
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# all the data elements are of type float\n",
      "s3"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 133,
       "text": [
        "Mon    33.0\n",
        "Tue    19.3\n",
        "Wed    15.0\n",
        "Thu    89.0\n",
        "Fri    11.0\n",
        "Sat    -5.0\n",
        "Sun     9.0\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 133
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.18 Example 3: Creating a Series from a Python Dict"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dict1 = {'Mon': 33, 'Tue': 19, 'Wed': 15, 'Thu': 89, 'Fri': 11, 'Sat': -5, 'Sun': 9}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s4 = pd.Series(dict1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 134
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s4"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 135,
       "text": [
        "Fri    11\n",
        "Mon    33\n",
        "Sat    -5\n",
        "Sun     9\n",
        "Thu    89\n",
        "Tue    19\n",
        "Wed    15\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 135
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Discussion\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The most general representation of a Series is as an ordered key-value store.\n",
      "\n",
      "* The order is represented by the offset. \n",
      "* The key-value is a mapping from index or label to the data array values. \n",
      "* Index as \"offset\" or \"position\" vs index as \"label\" or \"key\".\n",
      "\n",
      "<img src=\"files/images/series4.jpg\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.19 Series is ndarray-like"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# vectorized operations\n",
      "s4 * 2"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 137,
       "text": [
        "Fri     22\n",
        "Mon     66\n",
        "Sat    -10\n",
        "Sun     18\n",
        "Thu    178\n",
        "Tue     38\n",
        "Wed     30\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 137
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "np.log(s4)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 138,
       "text": [
        "Fri    2.397895\n",
        "Mon    3.496508\n",
        "Sat         NaN\n",
        "Sun    2.197225\n",
        "Thu    4.488636\n",
        "Tue    2.944439\n",
        "Wed    2.708050\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 138
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Note:** NaN (not a number) is the standard missing data marker used in Pandas"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# slice using index labels\n",
      "s4['Thu':'Wed']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 139,
       "text": [
        "Thu    89\n",
        "Tue    19\n",
        "Wed    15\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 139
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# slice using position\n",
      "s4[1:3]\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 140,
       "text": [
        "Mon    33\n",
        "Sat    -5\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 140
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# retrieve value using offset\n",
      "s4[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 141,
       "text": [
        "33"
       ]
      }
     ],
     "prompt_number": 141
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# set value using offset\n",
      "s4[1]=199"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 142
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s4"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 143,
       "text": [
        "Fri     11\n",
        "Mon    199\n",
        "Sat     -5\n",
        "Sun      9\n",
        "Thu     89\n",
        "Tue     19\n",
        "Wed     15\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 143
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# as a subclass of ndarray, Series is a valid argument to most NumPy functions - median\n",
      "s4"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 144,
       "text": [
        "Fri     11\n",
        "Mon    199\n",
        "Sat     -5\n",
        "Sun      9\n",
        "Thu     89\n",
        "Tue     19\n",
        "Wed     15\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 144
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s4.median()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 145,
       "text": [
        "15.0"
       ]
      }
     ],
     "prompt_number": 145
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# maximum \n",
      "s4.max()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 146,
       "text": [
        "199"
       ]
      }
     ],
     "prompt_number": 146
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# cumsum\n",
      "s4.cumsum()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 147,
       "text": [
        "Fri     11\n",
        "Mon    210\n",
        "Sat    205\n",
        "Sun    214\n",
        "Thu    303\n",
        "Tue    322\n",
        "Wed    337\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 147
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# looping over a collection and indices\n",
      "for i,v in enumerate(s4):\n",
      "    print i,v"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0 11\n",
        "1 199\n",
        "2 -5\n",
        "3 9\n",
        "4 89\n",
        "5 19\n",
        "6 15\n"
       ]
      }
     ],
     "prompt_number": 148
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# list comprehension can be used to create a new list\n",
      "new_list = [x**2 for x in s4]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 149
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "new_list"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 150,
       "text": [
        "[121, 39601, 25, 81, 7921, 361, 225]"
       ]
      }
     ],
     "prompt_number": 150
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### 1.21 Series is dict-like"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# is the key in the  \n",
      "'Sun' in s4\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 151,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 151
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 152,
       "text": [
        "19"
       ]
      }
     ],
     "prompt_number": 152
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# retrieve value using key or index\n",
      "s4['Tue']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 153,
       "text": [
        "19"
       ]
      }
     ],
     "prompt_number": 153
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# assignment using key\n",
      "s4['Tue']=200"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 154
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s4"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 155,
       "text": [
        "Fri     11\n",
        "Mon    199\n",
        "Sat     -5\n",
        "Sun      9\n",
        "Thu     89\n",
        "Tue    200\n",
        "Wed     15\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 155
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# looping over dictionary keys and values\n",
      "for k,v in s4.iteritems():\n",
      "    print k,v"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Fri 11\n",
        "Mon 199\n",
        "Sat -5\n",
        "Sun 9\n",
        "Thu 89\n",
        "Tue 200\n",
        "Wed 15\n"
       ]
      }
     ],
     "prompt_number": 157
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Resources"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Pandas Series Documentation"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from IPython.core.display import HTML\n",
      "\n",
      "HTML(\"<iframe src=http://pandas.pydata.org/pandas-docs/dev/dsintro.html#series width=800 height=350></iframe>\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!pwd"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "np.rand(50)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}