sensor_featureEngineer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering for Predicting Sign Up Rate\n",
    "Summary:   \n",
    "- The original data set is based on records not users. It is necessary to transfer and engineer features based on users. The reason is simple: each user can have multiple records. Once they sign-up successfully then they are converted. Notably, I can't include behavior special (Demo behavior) for sign-up users as it leaks information that users with this behavior is sign-up user.\n",
    "- The features I choose are:    \n",
    "total visit times per week   \n",
    "pageview times  \n",
    "btnClick times   \n",
    "index_leave times   \n",
    "about_leave times   \n",
    "courses_leave times   \n",
    "courses_play_leave times   \n",
    "latest_utm_term (I will extract key words from user searching query as it indicates the purpose)     \n",
    "latest_utm_medium (It indicates how user find Sensor website)    \n",
    "average_page_stay_time (I use median to replace NaN value as mean is biased by heavy users)  \n",
    "- Rebuild sign-up labels for each user (0 for non sign-up and 1 for sign-up)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. I want to get the total visit times per week for distinct users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/LiangTan/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py:2728: DtypeWarning: Columns (3,39,41,44,46,47,49,57,58,62,63,65) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "distinct_id\n",
       "00007ef910b6c9911f1b89d01a09aa3fc862f4a9     6\n",
       "000a216b72eff19bd0d5e17b9e676dd6ad9a38ac     1\n",
       "000c46a27ef69fa22b56d253a9c72773338a1686    22\n",
       "000ed1dcd942969b458c5b308937c6389c08f999     4\n",
       "00111feff544ef5280a4c7064a362a9ea59c9389     1\n",
       "Name: tot_visits_per_week, dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data = pd.read_csv('sensor_data_remove_missing.csv', sep='\\t')\n",
    "tot_visits_per_week = raw_data.groupby(['distinct_id'])['time'].count().rename(\"tot_visits_per_week\")\n",
    "tot_visits_per_week[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['nocache', 'distinct_id', 'event', 'jssdk_error', 'time', 'type',\n",
       "       'lib_lib', 'lib_method', 'lib_version', 'browser', 'browser_version',\n",
       "       'ip', 'is_first_day', 'is_first_time', 'latest_referrer',\n",
       "       'latest_referrer_host', 'latest_utm_campaign', 'latest_utm_content',\n",
       "       'latest_utm_medium', 'latest_utm_source', 'latest_utm_term',\n",
       "       'property_lib', 'property_lib_version', 'model', 'os', 'os_version',\n",
       "       'referrer', 'referrer_host', 'screen_height', 'screen_width', 'title',\n",
       "       'url', 'url_path', 'utm_campaign', 'utm_content', 'utm_medium',\n",
       "       'utm_source', 'utm_term', 'latest_ch', 'session_from_url',\n",
       "       'session_referrer', 'session_referrer_domain', 'session_referrer_host',\n",
       "       'ch', 'company', 'contact', 'email', 'from_url', 'info', 'isMsg',\n",
       "       'isSuccess', 'name', 'page', 'pagePosition', 'pageStayTime', 'pageUrl',\n",
       "       'project_name', 'referrHostUrl', 'referrerUrl', 'requestBtn', 'result',\n",
       "       'session_page_url', 'siteUrl', 'site_url', 'url_path.1',\n",
       "       'verification_code'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "$pageview                32620\n",
       "btnClick                 13866\n",
       "index_leave              10394\n",
       "demo_leave                3411\n",
       "about_leave               1032\n",
       "courses_leave              906\n",
       "formSubmit                 791\n",
       "courses_play_leave         747\n",
       "click_send_cellphone       600\n",
       "verify_cellphone_code      563\n",
       "clickSubmit                513\n",
       "page_close                 230\n",
       "Name: event, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data['event'].value_counts(dropna = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. I want to get the pageview times, btnClick times, index_leave times, about_leave times, courses_leave times and courses_play_leave times for each user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>event</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>distinct_id</th>\n",
       "      <th>event</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">00007ef910b6c9911f1b89d01a09aa3fc862f4a9</th>\n",
       "      <th>$pageview</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>btnClick</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>click_send_cellphone</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>formSubmit</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>verify_cellphone_code</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                event\n",
       "distinct_id                              event                       \n",
       "00007ef910b6c9911f1b89d01a09aa3fc862f4a9 $pageview                  1\n",
       "                                         btnClick                   1\n",
       "                                         click_send_cellphone       2\n",
       "                                         formSubmit                 1\n",
       "                                         verify_cellphone_code      1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users = list(tot_visits_per_week.index)\n",
    "event_count = raw_data.groupby(['distinct_id', 'event'])['event'].count()\n",
    "pd.DataFrame(event_count).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pageview_times = list()\n",
    "btnClick_times = list()\n",
    "index_leave_times = list()\n",
    "about_leave_times = list()\n",
    "courses_leave_times = list()\n",
    "courses_play_leave_times = list() \n",
    "for user in users:\n",
    "    if ('$pageview' in event_count[user]):\n",
    "        pageview_times.append(event_count[user]['$pageview'])\n",
    "    else:\n",
    "        pageview_times.append(0)\n",
    "    if ('btnClick' in event_count[user]):\n",
    "        btnClick_times.append(event_count[user]['btnClick'])\n",
    "    else:\n",
    "        btnClick_times.append(0)\n",
    "    if ('index_leave' in event_count[user]):\n",
    "        index_leave_times.append(event_count[user]['index_leave'])\n",
    "    else:\n",
    "        index_leave_times.append(0)\n",
    "    if ('about_leave' in event_count[user]):\n",
    "        about_leave_times.append(event_count[user]['about_leave'])\n",
    "    else:\n",
    "        about_leave_times.append(0)\n",
    "    if ('courses_leave' in event_count[user]):\n",
    "        courses_leave_times.append(event_count[user]['courses_leave'])\n",
    "    else:\n",
    "        courses_leave_times.append(0)\n",
    "    if ('courses_play_leave' in event_count[user]):\n",
    "        courses_play_leave_times.append(event_count[user]['courses_play_leave'])\n",
    "    else:\n",
    "        courses_play_leave_times.append(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. I want to get latest_utm_term features for each user. It indicates users' purpose for visiting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I think the latest_utm_term reflects the key purpose of users. To make it simpler, I will group and transform the terms into vector based on key words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sensor_term = list()\n",
    "persona_term = list()\n",
    "game_term = list()\n",
    "finance_term = list()\n",
    "web_term = list()\n",
    "client_term = list()\n",
    "ecom_term = list()\n",
    "ad_term = list()\n",
    "app_term = list()\n",
    "ret_term = list()\n",
    "pic_term = list()\n",
    "data_term = list()\n",
    "other_term = list()\n",
    "null_term = list()\n",
    "for user in users:\n",
    "    otherflag = 0\n",
    "    temp = ''.join(list(str(raw_data[raw_data['distinct_id'] == user]['latest_utm_term'])))\n",
    "    if ('NaN' in temp):\n",
    "        null_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        null_term.append(0)\n",
    "    if ('神策' in temp or 'sensor' in temp or 'Sensor' in temp):\n",
    "        sensor_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        sensor_term.append(0)\n",
    "    if ('画像' in temp):\n",
    "        persona_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        persona_term.append(0)\n",
    "    if ('游戏' in temp or '手游' in temp):\n",
    "        game_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        game_term.append(0)\n",
    "    if ('金融' in temp):\n",
    "        finance_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        finance_term.append(0)\n",
    "    if ('网页' in temp or '网站' in temp):\n",
    "        web_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        web_term.append(0)\n",
    "    if ('用户' in temp or '客户' in temp):\n",
    "        client_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        client_term.append(0)\n",
    "    if ('电商' in temp):\n",
    "        ecom_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        ecom_term.append(0)\n",
    "    if ('广告' in temp):\n",
    "        ad_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        ad_term.append(0)\n",
    "    if ('app' in temp or 'App' in temp):\n",
    "        app_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        app_term.append(0)\n",
    "    if ('留存' in temp):\n",
    "        ret_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        ret_term.append(0)\n",
    "    if ('图' in temp or '可视' in temp):\n",
    "        pic_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        pic_term.append(0)\n",
    "    if ('数据' in temp or '分析' in temp):\n",
    "        data_term.append(1)\n",
    "        otherflag = 1\n",
    "    else:\n",
    "        data_term.append(0)\n",
    "    if (otherflag == 0):\n",
    "        other_term.append(1)\n",
    "    else:\n",
    "        other_term.append(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. I want to get latest_utm_medium features for each user. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "cpc                   34623\n",
       "NaN                   25982\n",
       "mcpc                   3255\n",
       "mfeed                   934\n",
       "default                 538\n",
       "answer                  133\n",
       "banner                   67\n",
       "hao.caibaojian.com       33\n",
       "production               27\n",
       "sidonghui                24\n",
       "toutiao.io               20\n",
       "referral                 15\n",
       "youjian                   8\n",
       "lixinya                   6\n",
       "customer_m                5\n",
       "sidashidai                2\n",
       "edm                       1\n",
       "Name: latest_utm_medium, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data['latest_utm_medium'].value_counts(dropna = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "cpc_medium = list()\n",
    "mcpc_medium = list()\n",
    "mfeed_medium = list()\n",
    "default_medium = list()\n",
    "answer_medium = list()\n",
    "banner_medium = list()\n",
    "caibaojian_medium = list()\n",
    "production_medium = list()\n",
    "sidonghui_medium = list()\n",
    "toutiao_medium = list()\n",
    "referral_medium = list()\n",
    "youjian_medium = list()\n",
    "lixinya_medium = list()\n",
    "customer_medium = list()\n",
    "sidashidai_medium = list()\n",
    "edm_medium = list()\n",
    "null_medium = list()\n",
    "for user in users:\n",
    "    temp = set(raw_data[raw_data['distinct_id'] == user]['latest_utm_medium'])\n",
    "    if (np.nan in temp):\n",
    "        null_medium.append(1)\n",
    "    else:\n",
    "        null_medium.append(0)\n",
    "    if ('cpc' in temp):\n",
    "        cpc_medium.append(1)\n",
    "    else:\n",
    "        cpc_medium.append(0)\n",
    "    if ('mcpc' in temp):\n",
    "        mcpc_medium.append(1)\n",
    "    else:\n",
    "        mcpc_medium.append(0)\n",
    "    if ('mfeed' in temp):\n",
    "        mfeed_medium.append(1)\n",
    "    else:\n",
    "        mfeed_medium.append(0)\n",
    "    if ('default' in temp):\n",
    "        default_medium.append(1)\n",
    "    else:\n",
    "        default_medium.append(0)\n",
    "    if ('answer' in temp):\n",
    "        answer_medium.append(1)\n",
    "    else:\n",
    "        answer_medium.append(0)\n",
    "    if ('banner' in temp):\n",
    "        banner_medium.append(1)\n",
    "    else:\n",
    "        banner_medium.append(0)\n",
    "    if ('hao.caibaojian.com' in temp):\n",
    "        caibaojian_medium.append(1)\n",
    "    else:\n",
    "        caibaojian_medium.append(0)\n",
    "    if ('production' in temp):\n",
    "        production_medium.append(1)\n",
    "    else:\n",
    "        production_medium.append(0)\n",
    "    if ('sidonghui' in temp):\n",
    "        sidonghui_medium.append(1)\n",
    "    else:\n",
    "        sidonghui_medium.append(0)\n",
    "    if ('toutiao.io' in temp):\n",
    "        toutiao_medium.append(1)\n",
    "    else:\n",
    "        toutiao_medium.append(0)\n",
    "    if ('referral' in temp):\n",
    "        referral_medium.append(1)\n",
    "    else:\n",
    "        referral_medium.append(0)\n",
    "    if ('lixinya' in temp):\n",
    "        lixinya_medium.append(1)\n",
    "    else:\n",
    "        lixinya_medium.append(0)\n",
    "    if ('customer_m' in temp):\n",
    "        customer_medium.append(1)\n",
    "    else:\n",
    "        customer_medium.append(0)\n",
    "    if ('sidashidai' in temp):\n",
    "        sidashidai_medium.append(1)\n",
    "    else:\n",
    "        sidashidai_medium.append(0)\n",
    "    if ('edm' in temp):\n",
    "        edm_medium.append(1)\n",
    "    else:\n",
    "        edm_medium.append(0)\n",
    "    if ('youjian' in temp):\n",
    "        youjian_medium.append(1)\n",
    "    else:\n",
    "        youjian_medium.append(0)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. I want to get average page stay time features for each user. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "distinct_id\n",
       "00007ef910b6c9911f1b89d01a09aa3fc862f4a9              NaN\n",
       "000a216b72eff19bd0d5e17b9e676dd6ad9a38ac    921142.371000\n",
       "000c46a27ef69fa22b56d253a9c72773338a1686      1742.216571\n",
       "000ed1dcd942969b458c5b308937c6389c08f999        30.680000\n",
       "00111feff544ef5280a4c7064a362a9ea59c9389              NaN\n",
       "Name: pageStayTime, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_page_stay_time = raw_data.groupby(['distinct_id'])['pageStayTime'].mean()\n",
    "average_page_stay_time[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7860.027331669838, 32.362500000000004)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.nanmean(average_page_stay_time), np.nanmedian(average_page_stay_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "distinct_id\n",
       "00007ef910b6c9911f1b89d01a09aa3fc862f4a9        32.362500\n",
       "000a216b72eff19bd0d5e17b9e676dd6ad9a38ac    921142.371000\n",
       "000c46a27ef69fa22b56d253a9c72773338a1686      1742.216571\n",
       "000ed1dcd942969b458c5b308937c6389c08f999        30.680000\n",
       "00111feff544ef5280a4c7064a362a9ea59c9389        32.362500\n",
       "Name: pageStayTime, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_page_stay_time = average_page_stay_time.fillna(32.3625)\n",
    "average_page_stay_time[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. I need to get signup label (conversion label) for each user. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "isSuccess = list()\n",
    "for user in users:\n",
    "    temp = set(raw_data[raw_data['distinct_id'] == user]['isSuccess'])\n",
    "    if (True in temp):\n",
    "        isSuccess.append(1)\n",
    "    else:\n",
    "        isSuccess.append(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Make data set for model building"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_engineered = pd.DataFrame(\n",
    "    {'distinct_id': users,\n",
    "     'tot_visits_per_week': list(tot_visits_per_week),\n",
    "     'pageview_times': pageview_times,\n",
    "     'btnClick_times': btnClick_times,\n",
    "     'index_leave_times': index_leave_times,\n",
    "     'about_leave_times': about_leave_times,\n",
    "     'courses_leave_times': courses_leave_times,\n",
    "     'courses_play_leave_times': courses_play_leave_times,\n",
    "     'sensor_term': sensor_term,\n",
    "     'persona_term': persona_term,\n",
    "     'game_term': game_term,\n",
    "     'finance_term': finance_term,\n",
    "     'web_term': web_term,\n",
    "     'client_term': client_term,\n",
    "     'ecom_term': ecom_term,\n",
    "     'ad_term': ad_term,\n",
    "     'app_term': app_term,\n",
    "     'ret_term': ret_term,\n",
    "     'pic_term': pic_term,\n",
    "     'data_term': data_term,\n",
    "     'other_term': other_term,\n",
    "     'null_term': null_term,\n",
    "     'cpc_medium': cpc_medium,\n",
    "     'mcpc_medium': mcpc_medium,\n",
    "     'mfeed_medium': mfeed_medium,\n",
    "     'default_medium': default_medium,\n",
    "     'answer_medium': answer_medium,\n",
    "     'banner_medium': banner_medium,\n",
    "     'caibaojian_medium': caibaojian_medium,\n",
    "     'production_medium': production_medium,\n",
    "     'sidonghui_medium': sidonghui_medium,\n",
    "     'toutiao_medium': toutiao_medium,\n",
    "     'referral_medium': referral_medium,\n",
    "     'youjian_medium': youjian_medium,\n",
    "     'lixinya_medium': lixinya_medium,\n",
    "     'customer_medium': customer_medium,\n",
    "     'sidashidai_medium': sidashidai_medium,\n",
    "     'edm_medium': edm_medium,\n",
    "     'null_medium': null_medium,\n",
    "     'average_page_stay_time': list(average_page_stay_time),\n",
    "     'isSuccess': isSuccess\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_engineered = data_engineered[[\n",
    "     'distinct_id',\n",
    "     'tot_visits_per_week',\n",
    "     'pageview_times',\n",
    "     'btnClick_times',\n",
    "     'index_leave_times',\n",
    "     'about_leave_times',\n",
    "     'courses_leave_times',\n",
    "     'courses_play_leave_times',\n",
    "     'sensor_term',\n",
    "     'persona_term',\n",
    "     'game_term',\n",
    "     'finance_term',\n",
    "     'web_term',\n",
    "     'client_term',\n",
    "     'ecom_term',\n",
    "     'ad_term',\n",
    "     'app_term',\n",
    "     'ret_term',\n",
    "     'pic_term',\n",
    "     'data_term',\n",
    "     'other_term',\n",
    "     'null_term',\n",
    "     'cpc_medium',\n",
    "     'mcpc_medium',\n",
    "     'mfeed_medium',\n",
    "     'default_medium',\n",
    "     'answer_medium',\n",
    "     'banner_medium',\n",
    "     'caibaojian_medium',\n",
    "     'production_medium',\n",
    "     'sidonghui_medium',\n",
    "     'toutiao_medium',\n",
    "     'referral_medium',\n",
    "     'youjian_medium',\n",
    "     'lixinya_medium',\n",
    "     'customer_medium',\n",
    "     'sidashidai_medium',\n",
    "     'edm_medium',\n",
    "     'null_medium',\n",
    "     'average_page_stay_time',\n",
    "     'isSuccess'\n",
    "]]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>distinct_id</th>\n",
       "      <th>tot_visits_per_week</th>\n",
       "      <th>pageview_times</th>\n",
       "      <th>btnClick_times</th>\n",
       "      <th>index_leave_times</th>\n",
       "      <th>about_leave_times</th>\n",
       "      <th>courses_leave_times</th>\n",
       "      <th>courses_play_leave_times</th>\n",
       "      <th>sensor_term</th>\n",
       "      <th>persona_term</th>\n",
       "      <th>...</th>\n",
       "      <th>toutiao_medium</th>\n",
       "      <th>referral_medium</th>\n",
       "      <th>youjian_medium</th>\n",
       "      <th>lixinya_medium</th>\n",
       "      <th>customer_medium</th>\n",
       "      <th>sidashidai_medium</th>\n",
       "      <th>edm_medium</th>\n",
       "      <th>null_medium</th>\n",
       "      <th>average_page_stay_time</th>\n",
       "      <th>isSuccess</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00007ef910b6c9911f1b89d01a09aa3fc862f4a9</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>32.362500</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000a216b72eff19bd0d5e17b9e676dd6ad9a38ac</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>921142.371000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>000c46a27ef69fa22b56d253a9c72773338a1686</td>\n",
       "      <td>22</td>\n",
       "      <td>6</td>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1742.216571</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>000ed1dcd942969b458c5b308937c6389c08f999</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>30.680000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00111feff544ef5280a4c7064a362a9ea59c9389</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>32.362500</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                distinct_id  tot_visits_per_week  \\\n",
       "0  00007ef910b6c9911f1b89d01a09aa3fc862f4a9                    6   \n",
       "1  000a216b72eff19bd0d5e17b9e676dd6ad9a38ac                    1   \n",
       "2  000c46a27ef69fa22b56d253a9c72773338a1686                   22   \n",
       "3  000ed1dcd942969b458c5b308937c6389c08f999                    4   \n",
       "4  00111feff544ef5280a4c7064a362a9ea59c9389                    1   \n",
       "\n",
       "   pageview_times  btnClick_times  index_leave_times  about_leave_times  \\\n",
       "0               1               1                  0                  0   \n",
       "1               0               0                  1                  0   \n",
       "2               6               9                  6                  0   \n",
       "3               2               1                  1                  0   \n",
       "4               1               0                  0                  0   \n",
       "\n",
       "   courses_leave_times  courses_play_leave_times  sensor_term  persona_term  \\\n",
       "0                    0                         0            1             0   \n",
       "1                    0                         0            0             1   \n",
       "2                    1                         0            0             0   \n",
       "3                    0                         0            0             0   \n",
       "4                    0                         0            0             1   \n",
       "\n",
       "     ...      toutiao_medium  referral_medium  youjian_medium  lixinya_medium  \\\n",
       "0    ...                   0                0               0               0   \n",
       "1    ...                   0                0               0               0   \n",
       "2    ...                   0                0               0               0   \n",
       "3    ...                   0                0               0               0   \n",
       "4    ...                   0                0               0               0   \n",
       "\n",
       "   customer_medium  sidashidai_medium  edm_medium  null_medium  \\\n",
       "0                0                  0           0            0   \n",
       "1                0                  0           0            0   \n",
       "2                0                  0           0            1   \n",
       "3                0                  0           0            1   \n",
       "4                0                  0           0            0   \n",
       "\n",
       "   average_page_stay_time  isSuccess  \n",
       "0               32.362500          0  \n",
       "1           921142.371000          0  \n",
       "2             1742.216571          0  \n",
       "3               30.680000          0  \n",
       "4               32.362500          0  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_engineered.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_engineered.to_csv('sensor_data_engineered.csv', sep='\\t', encoding='utf-8', index = False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python(conda)",
   "language": "python",
   "name": "base"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}