step_1_parse_trajectories.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "This notebook is used to parse raw gps data. In some Apps trip information starts and ends being collected when the user inform so and trip mode is also informed by the user. This implies several issues like users travelling on modes different from what they informed or trips lasting several hours because users forgot to turn off the app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import geopandas as gpd\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "import numpy as np\n",
    "import math\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import and Format Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload the data all at once\n",
    "import glob\n",
    "import errno\n",
    "path  = '.data/' # replace with data path\n",
    "files = glob.glob(path + '*.geojson')\n",
    "list_file=[]\n",
    "for i in files[0:20]:\n",
    "    a=gpd.read_file(i)\n",
    "    list_file.append(a)\n",
    "data_app=pd.concat(list_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1767000, 5), (1922959, 5))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check that there are no duplicates after concatenating all the files\n",
    "data=data_app.drop_duplicates(subset=['activity_id','user_id','date'], keep='first')\n",
    "data.shape,data_biko.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the index to have it ordered \n",
    "data.reset_index(inplace=True)\n",
    "data.drop('index',inplace=True,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sol\\Anaconda3\\envs\\osmnx\\lib\\site-packages\\pandas\\core\\indexing.py:362: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  self.obj[key] = _infer_fill_value(value)\n",
      "C:\\Users\\sol\\Anaconda3\\envs\\osmnx\\lib\\site-packages\\pandas\\core\\indexing.py:543: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  self.obj[item] = s\n"
     ]
    }
   ],
   "source": [
    "# Unpack geometry into lat an long\n",
    "data.loc[:,'lat']=data['geometry'].apply(lambda x: x.y)\n",
    "data.loc[:,'lon']=data['geometry'].apply(lambda x: x.x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put the date into timestamp format\n",
    "data.loc[:,'date_1']=data['date'].apply(lambda x: datetime.strptime(x,'%Y-%m-%dT%H:%M:%S'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort the data to be in chronologycal order from the oldest to the newest date\n",
    "data.sort_values(['activity_id','date_1'],inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to calculate the delta seconds from the start of an activity id to each point recorded\n",
    "def total_second(arr):\n",
    "    return (arr-arr.min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the total_seconds function to create a list of cumulative times\n",
    "cum= data[['date_1']].groupby(data['activity_id']).transform(total_second)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "cum.rename(columns={'date_1':'delta_seconds'},inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a column where we see the seconds from the start of the trip for each recorded point\n",
    "merge=pd.merge(data,cum,left_index=True,right_index=True,how='outer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=merge.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the function total_seconds (notice that this is not our function delta_second) to go from timestamp to float number\n",
    "data['delta_seconds']=data['delta_seconds'].apply(lambda x: x.total_seconds())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parse Trajectories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "This program extracts \"stay points\" from a sequence of traces of an anonymous individual user with spatiotemporal inputs include:\n",
    "* user1--time stamp (in second), lon, lat (in degree)\n",
    "* par(1)-- roaming distance threshold (in km)\n",
    "* par(2)-- stay time threshold (in second).\n",
    "\n",
    "The algorithm was first proposed by R. Hariharan and K. Toyama (2004) in “Project Lachesis: parsing and modeling location histories.” The program here is a slightly revised version from R. Hariharan and K. Toyama(2004) By Shan Jiang (shanjang@mit.edu), May 2013. For more advanced algorithms treating phone data for a similar purpose, please refer to Jiang, S., G. Fiore, Y. Yang, J. Ferreira, E. Frazzoli, and M. C. González.(2013). \"A Review of Urban Computing for Mobile Phone Traces: Current Methods, Challenges and Opportunities.\" Proceedings of the ACM SIGKDD International Workshop on Urban Computing. Chicago, IL, USA.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We took the above described code and modified for our purpose in which we were more interested in identifying moving points rather than stay point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting parameter of spatial and temporal thresholds;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 1st is spatial parameter in km;\n",
    "* 2nd is temporal parameter in seconds\n",
    "\n",
    "Please try with different parameters and compare differences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Stay points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stay(user1):\n",
    "    user1=user1.reset_index()\n",
    "    par=[0.1,300]\n",
    "    maxt=len(user1['delta_seconds']) #Number of records in the activity\n",
    "    diam=par[0]\n",
    "    dur=par[1]\n",
    "    stay_ind=[0]*maxt\n",
    "\n",
    "#     Calculate the delta distance and time between two consequtive points\n",
    "    user1.loc[1:,'dest_lat']=user1['lat'].tolist()[:-1]\n",
    "    user1.loc[1:,'dest_lon']=user1['lon'].tolist()[:-1]\n",
    "    user1.loc[1:,'time_dest']=user1['delta_seconds'].tolist()[:-1]\n",
    "    user1.loc[1:,'d_t']=user1['delta_seconds']-user1['time_dest']\n",
    "    dT=user1['d_t'].to_list()\n",
    "    user1.loc[1:,'dist']=user1.apply(lambda row:pos2dist(row['lat'],row['lon'],row['dest_lat'],row['dest_lon']),axis=1)\n",
    "    dist=user1['dist'].tolist()\n",
    "\n",
    "# Detect the stays\n",
    "    i=0;\n",
    "    k=0;\n",
    "    move=1;\n",
    "    start_move=0;\n",
    "    stop=0;\n",
    "    while i<=maxt-2:\n",
    "        if dist[i]>diam:\n",
    "            i=i+1; \n",
    "        else:            \n",
    "            start_stay=i;\n",
    "            stop_move=start_stay-1;\n",
    "            end_stay=i+1;\n",
    "#             roaming distance\n",
    "            for j in range (i+2,maxt):\n",
    "                lat1 = user1['lat'][i];\n",
    "                lon1 = user1['lon'][i];\n",
    "                lat2 = user1['lat'][j];\n",
    "                lon2 = user1['lon'][j];\n",
    "                dist_ij = pos2dist(lat1,lon1,lat2,lon2);\n",
    "                if dist_ij>diam:\n",
    "                    end_stay=j-1;\n",
    "                    break\n",
    "                if j==maxt-1: # Cos in python we have -1 positions \n",
    "                    end_stay=j\n",
    "#             time duration\n",
    "            d_t=user1['delta_seconds'][end_stay]-user1['delta_seconds'][start_stay]\n",
    "            if d_t>=dur:\n",
    "                if start_stay==0:\n",
    "                    start_move=end_stay-1\n",
    "                else:                \n",
    "                    end_move=start_stay-1\n",
    "                    stay_ind[start_move:start_stay-1]=[move]*(start_stay-1-start_move)\n",
    "                    move=move+1;\n",
    "                    stop=stop+1;\n",
    "                    start_move=end_stay+1;\n",
    "                i=end_stay+1;\n",
    "                stop=stop+1;\n",
    "            else:\n",
    "                i=i+1;\n",
    "    if stop==0:\n",
    "        stay_ind=[1]*maxt;\n",
    "    return stay_ind "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pos2dist (lat1,lon1,lat2,lon2):\n",
    "    if (math.isnan(lat1)|math.isnan(lat2)|math.isnan(lon1)|math.isnan(lon2)):\n",
    "#         print ('nan')\n",
    "        dist=-1.0\n",
    "    elif ((lat1==lat2)&(lon1==lon2)):\n",
    "        dist=0.0\n",
    "    else:\n",
    "        R_aver = 6374;\n",
    "        lat1 = math.radians(lat1)\n",
    "        lon1 = math.radians(lon1)\n",
    "        lat2 = math.radians(lat2)\n",
    "        lon2 = math.radians(lon2)\n",
    "        aux=(math.cos(lat1)*math.cos(lat2)*math.cos(lon1-lon2) + math.sin(lat1)*math.sin(lat2))\n",
    "        if aux>=1:\n",
    "            dist=0.0\n",
    "        else:\n",
    "            dist = R_aver * math.acos(aux);\n",
    "    return dist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Speeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "def speed(leg):\n",
    "    DT=[]\n",
    "    DR=[]\n",
    "    speed=[]\n",
    "    Time=leg['delta_seconds'].apply(lambda x: x/3600) #time in hour for input in secs\n",
    "    for i in range(len(leg['delta_seconds'])-1):\n",
    "        DT.append(Time[i+1]-Time[i])\n",
    "        lat1 = leg['lat'][i]\n",
    "        lat2 = leg['lat'][i+1]\n",
    "        lon1 = leg['lon'][i]\n",
    "        lon2 = leg['lon'][i+1]\n",
    "        DR.append(pos2dist(lat1,lon1,lat2,lon2)) #output in km\n",
    "        sp=DR[i]/DT[i]\n",
    "        speed.append(sp)\n",
    "        speed_filter=pd.DataFrame(speed)\n",
    "        speed_filter.columns=['speed']        \n",
    "        mean_speed=sum(speed)/len(speed)\n",
    "        mean_speed_filt=(speed_filter[speed_filter['speed']<=30]['speed']).mean()\n",
    "    return mean_speed, mean_speed_filt,speed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "stays=data.groupby(['activity_id']).apply(stay)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# backup=stays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Transform the result into the data frame\n",
    "# a_2=a\n",
    "# a_3=a_2.reset_index()\n",
    "# a_3.rename(columns={0:'stays'},inplace=True)\n",
    "# a_3.set_index('activity_id',inplace=True)\n",
    "# s = a_3.apply(lambda x: pd.Series(x['stays']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "# s.reset_index().to_csv('stays.csv')\n",
    "# data.loc[:,'stays']=s.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See the result as a dataframe\n",
    "df=stays.reset_index()\n",
    "df.rename(columns={0:'stays'},inplace=True)\n",
    "df.set_index('activity_id',inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Break the list so that each element is a new row\n",
    "stays_series=df.apply(lambda x: pd.Series(x['stays']),axis=1).stack().reset_index(level=1, drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "stays_df=stays_series.reset_index()\n",
    "stays_df.rename(columns={0:'stays'},inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finaly, we add the result to our dataframe\n",
    "final_data=pd.merge(data,stays_df,left_on='activity_id',right_on='activity_id',how='outer')\n",
    "# Or\n",
    "# data.loc[:,'stays']=stays_df['stays'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export results to be used in step 2\n",
    "data.to_csv('data_parsed.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}