12.02_Engineering_time.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Engineering Time\n",
    "\n",
    "In this demo, we are going to extract different ways of representing time from a timestamp. We can extract for example:\n",
    "\n",
    "- hour\n",
    "- minute\n",
    "- second\n",
    "- data\n",
    "- elapsed time\n",
    "\n",
    "We will create a toy dataset for the demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-01-05 11:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-01-05 12:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-01-05 13:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-01-05 14:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-01-05 15:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2015-01-05 16:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2015-01-05 17:20:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 date\n",
       "0 2015-01-05 11:20:00\n",
       "1 2015-01-05 12:20:00\n",
       "2 2015-01-05 13:20:00\n",
       "3 2015-01-05 14:20:00\n",
       "4 2015-01-05 15:20:00\n",
       "5 2015-01-05 16:20:00\n",
       "6 2015-01-05 17:20:00"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's create a toy data set: 1 column 7 different timestamps,\n",
    "# 1 hr difference between timestamp\n",
    "\n",
    "date = pd.Series(pd.date_range('2015-1-5 11:20:00', periods=7, freq='H'))\n",
    "\n",
    "df = pd.DataFrame(dict(date=date))\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract the hr, minute and second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>hour</th>\n",
       "      <th>min</th>\n",
       "      <th>sec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-01-05 11:20:00</td>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-01-05 12:20:00</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-01-05 13:20:00</td>\n",
       "      <td>13</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-01-05 14:20:00</td>\n",
       "      <td>14</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-01-05 15:20:00</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2015-01-05 16:20:00</td>\n",
       "      <td>16</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2015-01-05 17:20:00</td>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 date  hour  min  sec\n",
       "0 2015-01-05 11:20:00    11   20    0\n",
       "1 2015-01-05 12:20:00    12   20    0\n",
       "2 2015-01-05 13:20:00    13   20    0\n",
       "3 2015-01-05 14:20:00    14   20    0\n",
       "4 2015-01-05 15:20:00    15   20    0\n",
       "5 2015-01-05 16:20:00    16   20    0\n",
       "6 2015-01-05 17:20:00    17   20    0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['hour'] = df['date'].dt.hour\n",
    "df['min'] = df['date'].dt.minute\n",
    "df['sec'] = df['date'].dt.second\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract time part"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>hour</th>\n",
       "      <th>min</th>\n",
       "      <th>sec</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-01-05 11:20:00</td>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>11:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-01-05 12:20:00</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>12:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-01-05 13:20:00</td>\n",
       "      <td>13</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>13:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-01-05 14:20:00</td>\n",
       "      <td>14</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>14:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-01-05 15:20:00</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>15:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2015-01-05 16:20:00</td>\n",
       "      <td>16</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>16:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2015-01-05 17:20:00</td>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>17:20:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 date  hour  min  sec      time\n",
       "0 2015-01-05 11:20:00    11   20    0  11:20:00\n",
       "1 2015-01-05 12:20:00    12   20    0  12:20:00\n",
       "2 2015-01-05 13:20:00    13   20    0  13:20:00\n",
       "3 2015-01-05 14:20:00    14   20    0  14:20:00\n",
       "4 2015-01-05 15:20:00    15   20    0  15:20:00\n",
       "5 2015-01-05 16:20:00    16   20    0  16:20:00\n",
       "6 2015-01-05 17:20:00    17   20    0  17:20:00"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['time'] = df['date'].dt.time\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract hr, min, sec, at the same time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>hour</th>\n",
       "      <th>min</th>\n",
       "      <th>sec</th>\n",
       "      <th>time</th>\n",
       "      <th>h</th>\n",
       "      <th>m</th>\n",
       "      <th>s</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-01-05 11:20:00</td>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>11:20:00</td>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-01-05 12:20:00</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>12:20:00</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-01-05 13:20:00</td>\n",
       "      <td>13</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>13:20:00</td>\n",
       "      <td>13</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-01-05 14:20:00</td>\n",
       "      <td>14</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>14:20:00</td>\n",
       "      <td>14</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-01-05 15:20:00</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>15:20:00</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2015-01-05 16:20:00</td>\n",
       "      <td>16</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>16:20:00</td>\n",
       "      <td>16</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2015-01-05 17:20:00</td>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>17:20:00</td>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 date  hour  min  sec      time   h   m  s\n",
       "0 2015-01-05 11:20:00    11   20    0  11:20:00  11  20  0\n",
       "1 2015-01-05 12:20:00    12   20    0  12:20:00  12  20  0\n",
       "2 2015-01-05 13:20:00    13   20    0  13:20:00  13  20  0\n",
       "3 2015-01-05 14:20:00    14   20    0  14:20:00  14  20  0\n",
       "4 2015-01-05 15:20:00    15   20    0  15:20:00  15  20  0\n",
       "5 2015-01-05 16:20:00    16   20    0  16:20:00  16  20  0\n",
       "6 2015-01-05 17:20:00    17   20    0  17:20:00  17  20  0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now let's repeat what we did in cell 3 in 1 command\n",
    "\n",
    "df[['h','m','s']] = pd.DataFrame([(x.hour, x.minute, x.second) for x in df['time']])\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate time difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Start_date</th>\n",
       "      <th>End_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-31 12:00:00</td>\n",
       "      <td>2013-03-17 21:45:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-02-29 12:00:00</td>\n",
       "      <td>2013-03-24 21:45:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-03-31 12:00:00</td>\n",
       "      <td>2013-03-31 21:45:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-04-30 12:00:00</td>\n",
       "      <td>2013-04-07 21:45:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-05-31 12:00:00</td>\n",
       "      <td>2013-04-14 21:45:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2012-06-30 12:00:00</td>\n",
       "      <td>2013-04-21 21:45:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2012-07-31 12:00:00</td>\n",
       "      <td>2013-04-28 21:45:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Start_date            End_date\n",
       "0 2012-01-31 12:00:00 2013-03-17 21:45:00\n",
       "1 2012-02-29 12:00:00 2013-03-24 21:45:00\n",
       "2 2012-03-31 12:00:00 2013-03-31 21:45:00\n",
       "3 2012-04-30 12:00:00 2013-04-07 21:45:00\n",
       "4 2012-05-31 12:00:00 2013-04-14 21:45:00\n",
       "5 2012-06-30 12:00:00 2013-04-21 21:45:00\n",
       "6 2012-07-31 12:00:00 2013-04-28 21:45:00"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's create another toy dataframe with 2 timestamp columns\n",
    "# and 7 rows each, in the first column the timestamps change monthly,\n",
    "# in the second column the timestamps change weekly\n",
    "\n",
    "date1 = pd.Series(pd.date_range('2012-1-1 12:00:00', periods=7, freq='M'))\n",
    "date2 = pd.Series(pd.date_range('2013-3-11 21:45:00', periods=7, freq='W'))\n",
    " \n",
    "df = pd.DataFrame(dict(Start_date = date1, End_date = date2))\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Start_date</th>\n",
       "      <th>End_date</th>\n",
       "      <th>diff_seconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-31 12:00:00</td>\n",
       "      <td>2013-03-17 21:45:00</td>\n",
       "      <td>35545500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-02-29 12:00:00</td>\n",
       "      <td>2013-03-24 21:45:00</td>\n",
       "      <td>33644700.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-03-31 12:00:00</td>\n",
       "      <td>2013-03-31 21:45:00</td>\n",
       "      <td>31571100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-04-30 12:00:00</td>\n",
       "      <td>2013-04-07 21:45:00</td>\n",
       "      <td>29583900.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-05-31 12:00:00</td>\n",
       "      <td>2013-04-14 21:45:00</td>\n",
       "      <td>27510300.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2012-06-30 12:00:00</td>\n",
       "      <td>2013-04-21 21:45:00</td>\n",
       "      <td>25523100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2012-07-31 12:00:00</td>\n",
       "      <td>2013-04-28 21:45:00</td>\n",
       "      <td>23449500.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Start_date            End_date  diff_seconds\n",
       "0 2012-01-31 12:00:00 2013-03-17 21:45:00    35545500.0\n",
       "1 2012-02-29 12:00:00 2013-03-24 21:45:00    33644700.0\n",
       "2 2012-03-31 12:00:00 2013-03-31 21:45:00    31571100.0\n",
       "3 2012-04-30 12:00:00 2013-04-07 21:45:00    29583900.0\n",
       "4 2012-05-31 12:00:00 2013-04-14 21:45:00    27510300.0\n",
       "5 2012-06-30 12:00:00 2013-04-21 21:45:00    25523100.0\n",
       "6 2012-07-31 12:00:00 2013-04-28 21:45:00    23449500.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's calculate the time elapsed in seconds\n",
    "\n",
    "df['diff_seconds'] = df['End_date'] - df['Start_date']\n",
    "df['diff_seconds']=df['diff_seconds']/np.timedelta64(1,'s')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Start_date</th>\n",
       "      <th>End_date</th>\n",
       "      <th>diff_seconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-31 12:00:00</td>\n",
       "      <td>2013-03-17 21:45:00</td>\n",
       "      <td>592425.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-02-29 12:00:00</td>\n",
       "      <td>2013-03-24 21:45:00</td>\n",
       "      <td>560745.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-03-31 12:00:00</td>\n",
       "      <td>2013-03-31 21:45:00</td>\n",
       "      <td>526185.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-04-30 12:00:00</td>\n",
       "      <td>2013-04-07 21:45:00</td>\n",
       "      <td>493065.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-05-31 12:00:00</td>\n",
       "      <td>2013-04-14 21:45:00</td>\n",
       "      <td>458505.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2012-06-30 12:00:00</td>\n",
       "      <td>2013-04-21 21:45:00</td>\n",
       "      <td>425385.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2012-07-31 12:00:00</td>\n",
       "      <td>2013-04-28 21:45:00</td>\n",
       "      <td>390825.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Start_date            End_date  diff_seconds\n",
       "0 2012-01-31 12:00:00 2013-03-17 21:45:00      592425.0\n",
       "1 2012-02-29 12:00:00 2013-03-24 21:45:00      560745.0\n",
       "2 2012-03-31 12:00:00 2013-03-31 21:45:00      526185.0\n",
       "3 2012-04-30 12:00:00 2013-04-07 21:45:00      493065.0\n",
       "4 2012-05-31 12:00:00 2013-04-14 21:45:00      458505.0\n",
       "5 2012-06-30 12:00:00 2013-04-21 21:45:00      425385.0\n",
       "6 2012-07-31 12:00:00 2013-04-28 21:45:00      390825.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's calculate the time elapsed in minutes\n",
    "\n",
    "df['diff_seconds'] = df['End_date'] - df['Start_date']\n",
    "df['diff_seconds']=df['diff_seconds']/np.timedelta64(1,'m')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more details visit [this article](http://www.datasciencemadesimple.com/difference-two-timestamps-seconds-minutes-hours-pandas-python-2/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Work with different timezones\n",
    "\n",
    "In the next few cells, we will see how to work with timestamps that are in different time zones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-01 09:00:00+02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-01 10:00:00+02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-01 11:00:00+02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-01 09:00:00-05:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-01 10:00:00-05:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-01 11:00:00-05:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        time\n",
       "0  2014-08-01 09:00:00+02:00\n",
       "1  2014-08-01 10:00:00+02:00\n",
       "2  2014-08-01 11:00:00+02:00\n",
       "0  2014-08-01 09:00:00-05:00\n",
       "1  2014-08-01 10:00:00-05:00\n",
       "2  2014-08-01 11:00:00-05:00"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first, let's create a toy dataframe with some timestamps in different time zones\n",
    "\n",
    "df = pd.DataFrame()\n",
    "\n",
    "df['time'] = pd.concat([\n",
    "    pd.Series(\n",
    "        pd.date_range(\n",
    "            start='2014-08-01 09:00', freq='H', periods=3,\n",
    "            tz='Europe/Berlin')),\n",
    "    pd.Series(\n",
    "        pd.date_range(\n",
    "            start='2014-08-01 09:00', freq='H', periods=3, tz='US/Central'))\n",
    "    ], axis=0)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the different timezones indicated by the +2 and -5, respect to the meridian."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>time_utc</th>\n",
       "      <th>time_london</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-01 09:00:00+02:00</td>\n",
       "      <td>2014-08-01 07:00:00+00:00</td>\n",
       "      <td>2014-08-01 08:00:00+01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-01 10:00:00+02:00</td>\n",
       "      <td>2014-08-01 08:00:00+00:00</td>\n",
       "      <td>2014-08-01 09:00:00+01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-01 11:00:00+02:00</td>\n",
       "      <td>2014-08-01 09:00:00+00:00</td>\n",
       "      <td>2014-08-01 10:00:00+01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-01 09:00:00-05:00</td>\n",
       "      <td>2014-08-01 14:00:00+00:00</td>\n",
       "      <td>2014-08-01 15:00:00+01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-01 10:00:00-05:00</td>\n",
       "      <td>2014-08-01 15:00:00+00:00</td>\n",
       "      <td>2014-08-01 16:00:00+01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-01 11:00:00-05:00</td>\n",
       "      <td>2014-08-01 16:00:00+00:00</td>\n",
       "      <td>2014-08-01 17:00:00+01:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        time                  time_utc  \\\n",
       "0  2014-08-01 09:00:00+02:00 2014-08-01 07:00:00+00:00   \n",
       "1  2014-08-01 10:00:00+02:00 2014-08-01 08:00:00+00:00   \n",
       "2  2014-08-01 11:00:00+02:00 2014-08-01 09:00:00+00:00   \n",
       "0  2014-08-01 09:00:00-05:00 2014-08-01 14:00:00+00:00   \n",
       "1  2014-08-01 10:00:00-05:00 2014-08-01 15:00:00+00:00   \n",
       "2  2014-08-01 11:00:00-05:00 2014-08-01 16:00:00+00:00   \n",
       "\n",
       "                time_london  \n",
       "0 2014-08-01 08:00:00+01:00  \n",
       "1 2014-08-01 09:00:00+01:00  \n",
       "2 2014-08-01 10:00:00+01:00  \n",
       "0 2014-08-01 15:00:00+01:00  \n",
       "1 2014-08-01 16:00:00+01:00  \n",
       "2 2014-08-01 17:00:00+01:00  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to work with different time zones, first we unify the timezone to the central one\n",
    "# setting utc = True\n",
    "\n",
    "df['time_utc'] = pd.to_datetime(df['time'], utc=True)\n",
    "\n",
    "# next we change all timestamps to the desired timezone, eg Europe/London\n",
    "# in this example\n",
    "\n",
    "df['time_london'] = df['time_utc'].dt.tz_convert('Europe/London')\n",
    "\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For feature engineering, we can of course, set all timezones to the central, utc=True, and work with that to extract time elapsed etc. The additional timezone encoding is mostly for human readability.\n",
    "\n",
    "I hope you enjoyed this section, and see you in the next one!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "feml",
   "language": "python",
   "name": "feml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}