07 - Automating Privacy in Pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "68219600",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import scipy.stats as stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b53d611e",
   "metadata": {},
   "outputs": [],
   "source": [
    "order_data = pd.read_csv('data/orders.csv')\n",
    "browser_data = pd.read_csv('data/browsing.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04d513c2",
   "metadata": {},
   "source": [
    "### Our Plan of Action\n",
    "\n",
    "- Remove personal identifiers with the exception of the user ID, which is hashed so that if marketing team comes back with specific questions, we can try to help answer them\n",
    "- If the user session came in with a tagged campaign, retain campaign information; otherwise leave blank\n",
    "- Retain billing city and state, as this was specifically asked for by the marketing department to effectively answer questions for their campaigns\n",
    "- Join order data and aggregate order quantity (number of orders) and order value (sum of quantities) per user\n",
    "- Bound outliers that have exceptionally large or small amounts of customer value (if needed, you can create a written summary of these for marketing review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c838bf20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_number</th>\n",
       "      <th>user_id</th>\n",
       "      <th>email</th>\n",
       "      <th>street_address</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>num_items</th>\n",
       "      <th>total_price</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>754</td>\n",
       "      <td>roy50</td>\n",
       "      <td>margaretwalker@example.com</td>\n",
       "      <td>8222 Jared Plains Suite 489</td>\n",
       "      <td>Millerfurt</td>\n",
       "      <td>Delaware</td>\n",
       "      <td>4</td>\n",
       "      <td>162</td>\n",
       "      <td>2022-07-31T13:53:42.176704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>371</td>\n",
       "      <td>handerson</td>\n",
       "      <td>james63@example.org</td>\n",
       "      <td>39972 Isabella Haven Suite 670</td>\n",
       "      <td>Farrellport</td>\n",
       "      <td>Oklahoma</td>\n",
       "      <td>3</td>\n",
       "      <td>42</td>\n",
       "      <td>2022-08-03T17:54:12.176704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>376</td>\n",
       "      <td>william55</td>\n",
       "      <td>sotopatricia@example.net</td>\n",
       "      <td>4418 Raymond Club</td>\n",
       "      <td>West Andreafort</td>\n",
       "      <td>Illinois</td>\n",
       "      <td>8</td>\n",
       "      <td>198</td>\n",
       "      <td>2022-08-05T18:54:45.176704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>431</td>\n",
       "      <td>uking</td>\n",
       "      <td>cponce@example.com</td>\n",
       "      <td>369 Angela Green Suite 649</td>\n",
       "      <td>Whitemouth</td>\n",
       "      <td>Tennessee</td>\n",
       "      <td>6</td>\n",
       "      <td>90</td>\n",
       "      <td>2022-08-06T20:55:01.176704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>713</td>\n",
       "      <td>tiffany94</td>\n",
       "      <td>kingelizabeth@example.net</td>\n",
       "      <td>3030 Kerr Streets Apt. 695</td>\n",
       "      <td>Hamptonhaven</td>\n",
       "      <td>Indiana</td>\n",
       "      <td>5</td>\n",
       "      <td>62</td>\n",
       "      <td>2022-08-09T22:55:50.176704</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   order_number    user_id                       email  \\\n",
       "0           754      roy50  margaretwalker@example.com   \n",
       "1           371  handerson         james63@example.org   \n",
       "2           376  william55    sotopatricia@example.net   \n",
       "3           431      uking          cponce@example.com   \n",
       "4           713  tiffany94   kingelizabeth@example.net   \n",
       "\n",
       "                   street_address             city      state  num_items  \\\n",
       "0     8222 Jared Plains Suite 489       Millerfurt   Delaware          4   \n",
       "1  39972 Isabella Haven Suite 670      Farrellport   Oklahoma          3   \n",
       "2               4418 Raymond Club  West Andreafort   Illinois          8   \n",
       "3      369 Angela Green Suite 649       Whitemouth  Tennessee          6   \n",
       "4      3030 Kerr Streets Apt. 695     Hamptonhaven    Indiana          5   \n",
       "\n",
       "   total_price                   timestamp  \n",
       "0          162  2022-07-31T13:53:42.176704  \n",
       "1           42  2022-08-03T17:54:12.176704  \n",
       "2          198  2022-08-05T18:54:45.176704  \n",
       "3           90  2022-08-06T20:55:01.176704  \n",
       "4           62  2022-08-09T22:55:50.176704  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "order_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "acfe06e9",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_number</th>\n",
       "      <th>inbound_uri</th>\n",
       "      <th>browser_agent</th>\n",
       "      <th>ip</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9</td>\n",
       "      <td>weight-likely</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; PPC Mac OS X 10_11_8; ...</td>\n",
       "      <td>199.252.41.214</td>\n",
       "      <td>2022-07-31T13:53:49.548290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>441</td>\n",
       "      <td>position-report</td>\n",
       "      <td>Opera/9.44.(X11; Linux x86_64; gv-GB) Presto/2...</td>\n",
       "      <td>179.25.93.144</td>\n",
       "      <td>2022-08-03T17:54:25.548290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72</td>\n",
       "      <td>tough-age-growth</td>\n",
       "      <td>Mozilla/5.0 (Windows; U; Windows NT 5.0) Apple...</td>\n",
       "      <td>125.124.68.154</td>\n",
       "      <td>2022-08-05T18:55:27.548290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>168</td>\n",
       "      <td>way-right-century</td>\n",
       "      <td>Mozilla/5.0 (compatible; MSIE 6.0; Windows 98;...</td>\n",
       "      <td>22.43.48.185</td>\n",
       "      <td>2022-08-08T20:55:59.548290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>539</td>\n",
       "      <td>event-might-society</td>\n",
       "      <td>Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like ...</td>\n",
       "      <td>30.219.140.88</td>\n",
       "      <td>2022-08-10T23:57:01.548290</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   order_number          inbound_uri  \\\n",
       "0             9        weight-likely   \n",
       "1           441      position-report   \n",
       "2            72     tough-age-growth   \n",
       "3           168    way-right-century   \n",
       "4           539  event-might-society   \n",
       "\n",
       "                                       browser_agent              ip  \\\n",
       "0  Mozilla/5.0 (Macintosh; PPC Mac OS X 10_11_8; ...  199.252.41.214   \n",
       "1  Opera/9.44.(X11; Linux x86_64; gv-GB) Presto/2...   179.25.93.144   \n",
       "2  Mozilla/5.0 (Windows; U; Windows NT 5.0) Apple...  125.124.68.154   \n",
       "3  Mozilla/5.0 (compatible; MSIE 6.0; Windows 98;...    22.43.48.185   \n",
       "4  Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like ...   30.219.140.88   \n",
       "\n",
       "                    timestamp  \n",
       "0  2022-07-31T13:53:49.548290  \n",
       "1  2022-08-03T17:54:25.548290  \n",
       "2  2022-08-05T18:55:27.548290  \n",
       "3  2022-08-08T20:55:59.548290  \n",
       "4  2022-08-10T23:57:01.548290  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "browser_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c829ed2",
   "metadata": {},
   "source": [
    "### First we drop the unnecessary PII columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d09a97fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "browser_data = browser_data.drop(['browser_agent', 'ip'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b85ec592",
   "metadata": {},
   "outputs": [],
   "source": [
    "order_data = order_data.drop(['email', 'street_address'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1e083f5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "browser_and_orders = browser_data.merge(order_data, how='inner', on=['order_number'], \n",
    "                                        suffixes=('__browser','__order'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "142c0847",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_number</th>\n",
       "      <th>inbound_uri</th>\n",
       "      <th>timestamp__browser</th>\n",
       "      <th>user_id</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>num_items</th>\n",
       "      <th>total_price</th>\n",
       "      <th>timestamp__order</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9</td>\n",
       "      <td>weight-likely</td>\n",
       "      <td>2022-07-31T13:53:49.548290</td>\n",
       "      <td>qgeorge</td>\n",
       "      <td>New Beth</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>2</td>\n",
       "      <td>198</td>\n",
       "      <td>2024-02-17T09:17:44.176704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9</td>\n",
       "      <td>weight-likely</td>\n",
       "      <td>2022-07-31T13:53:49.548290</td>\n",
       "      <td>ilarsen</td>\n",
       "      <td>West Drewfort</td>\n",
       "      <td>Texas</td>\n",
       "      <td>4</td>\n",
       "      <td>162</td>\n",
       "      <td>2030-02-05T06:27:23.176704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>weight-likely</td>\n",
       "      <td>2022-07-31T13:53:49.548290</td>\n",
       "      <td>kelliott</td>\n",
       "      <td>Conniemouth</td>\n",
       "      <td>Georgia</td>\n",
       "      <td>3</td>\n",
       "      <td>106</td>\n",
       "      <td>2035-02-11T03:07:40.176704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>weight-likely</td>\n",
       "      <td>2022-07-31T13:53:49.548290</td>\n",
       "      <td>angela76</td>\n",
       "      <td>Brownville</td>\n",
       "      <td>South Dakota</td>\n",
       "      <td>6</td>\n",
       "      <td>22</td>\n",
       "      <td>2043-03-24T10:33:55.176704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9</td>\n",
       "      <td>weight-likely</td>\n",
       "      <td>2022-07-31T13:53:49.548290</td>\n",
       "      <td>adrianhughes</td>\n",
       "      <td>Murphyhaven</td>\n",
       "      <td>Arizona</td>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>2058-07-13T21:12:02.176704</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   order_number    inbound_uri          timestamp__browser       user_id  \\\n",
       "0             9  weight-likely  2022-07-31T13:53:49.548290       qgeorge   \n",
       "1             9  weight-likely  2022-07-31T13:53:49.548290       ilarsen   \n",
       "2             9  weight-likely  2022-07-31T13:53:49.548290      kelliott   \n",
       "3             9  weight-likely  2022-07-31T13:53:49.548290      angela76   \n",
       "4             9  weight-likely  2022-07-31T13:53:49.548290  adrianhughes   \n",
       "\n",
       "            city         state  num_items  total_price  \\\n",
       "0       New Beth      Virginia          2          198   \n",
       "1  West Drewfort         Texas          4          162   \n",
       "2    Conniemouth       Georgia          3          106   \n",
       "3     Brownville  South Dakota          6           22   \n",
       "4    Murphyhaven       Arizona          3           30   \n",
       "\n",
       "             timestamp__order  \n",
       "0  2024-02-17T09:17:44.176704  \n",
       "1  2030-02-05T06:27:23.176704  \n",
       "2  2035-02-11T03:07:40.176704  \n",
       "3  2043-03-24T10:33:55.176704  \n",
       "4  2058-07-13T21:12:02.176704  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "browser_and_orders.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b8d0e0d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "098115f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_by_order = browser_and_orders.groupby('order_number').agg({'user_id':'first', \n",
    "                                                'city':'first', \n",
    "                                                'state':'first', \n",
    "                                                'num_items':'sum',\n",
    "                                                'total_price': 'sum',\n",
    "                                                'timestamp__order': 'first',\n",
    "                                                'inbound_uri': 'first'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9b89073f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>num_items</th>\n",
       "      <th>total_price</th>\n",
       "      <th>timestamp__order</th>\n",
       "      <th>inbound_uri</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>order_number</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aaron21</td>\n",
       "      <td>Walkerville</td>\n",
       "      <td>Delaware</td>\n",
       "      <td>605</td>\n",
       "      <td>9108</td>\n",
       "      <td>2045-06-23T14:05:41.176704</td>\n",
       "      <td>bag-fast-difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>jessicajacobs</td>\n",
       "      <td>North Adam</td>\n",
       "      <td>Georgia</td>\n",
       "      <td>252</td>\n",
       "      <td>4860</td>\n",
       "      <td>2025-10-06T02:48:18.176704</td>\n",
       "      <td>those-color-degree</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>rcombs</td>\n",
       "      <td>West Jaredberg</td>\n",
       "      <td>Mississippi</td>\n",
       "      <td>305</td>\n",
       "      <td>4750</td>\n",
       "      <td>2046-11-03T19:15:02.176704</td>\n",
       "      <td>third-wish-spring</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>brandon98</td>\n",
       "      <td>Ryanchester</td>\n",
       "      <td>Alaska</td>\n",
       "      <td>440</td>\n",
       "      <td>6402</td>\n",
       "      <td>2038-10-10T12:43:40.176704</td>\n",
       "      <td>produce-might-a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>desireegonzales</td>\n",
       "      <td>Thomasshire</td>\n",
       "      <td>Arizona</td>\n",
       "      <td>680</td>\n",
       "      <td>13420</td>\n",
       "      <td>2033-05-02T14:28:03.176704</td>\n",
       "      <td>whole-activity-new</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      user_id            city        state  num_items  \\\n",
       "order_number                                                            \n",
       "0                     aaron21     Walkerville     Delaware        605   \n",
       "1               jessicajacobs      North Adam      Georgia        252   \n",
       "2                      rcombs  West Jaredberg  Mississippi        305   \n",
       "3                   brandon98     Ryanchester       Alaska        440   \n",
       "4             desireegonzales     Thomasshire      Arizona        680   \n",
       "\n",
       "              total_price            timestamp__order         inbound_uri  \n",
       "order_number                                                               \n",
       "0                    9108  2045-06-23T14:05:41.176704  bag-fast-difficult  \n",
       "1                    4860  2025-10-06T02:48:18.176704  those-color-degree  \n",
       "2                    4750  2046-11-03T19:15:02.176704   third-wish-spring  \n",
       "3                    6402  2038-10-10T12:43:40.176704     produce-might-a  \n",
       "4                   13420  2033-05-02T14:28:03.176704  whole-activity-new  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_by_order.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e5e5677",
   "metadata": {},
   "source": [
    "### Encrypting the User ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "16c1a4f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_by_order.user_id.map(lambda x: len(x)).max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "407ccfcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import string, random\n",
    "\n",
    "def genkey(length: int):\n",
    "    \"\"\"Generate key.\"\"\"\n",
    "    return \"\".join(random.choice(string.ascii_lowercase + \n",
    "                                 string.ascii_uppercase + \n",
    "                                 string.digits + \n",
    "                                 '[@_!#$%^&*()<>?/\\|}{~:]') for _ in range(length))\n",
    "\n",
    "def xor_strings(s, t):\n",
    "    \"\"\"xor two strings together.\"\"\"\n",
    "    if isinstance(s, str):\n",
    "        # Text strings contain single characters\n",
    "        return \"\".join(chr(ord(a) ^ ord(b)) for a, b in zip(s, t))\n",
    "    else:\n",
    "        # Bytes objects contain integer values in the range 0-255\n",
    "        return bytes([a ^ b for a, b in zip(s, t)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "74aa9fbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_secret_key = genkey(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "cf7046ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "def xor_username(key, string):\n",
    "    if len(string) < 20:\n",
    "        string += ' ' * (20 - len(string))\n",
    "    return xor_strings(string, key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3b2387ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_by_order['e_user_id'] = summary_by_order.user_id.map(lambda x: xor_username(my_secret_key, x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1faebb61",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_by_order = summary_by_order.drop(['user_id'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "51042a80",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'jessicajacobs       '"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xor_username(my_secret_key, summary_by_order['e_user_id'][1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b792a65",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "50ae7e54",
   "metadata": {},
   "source": [
    "### Key Safety\n",
    "\n",
    "What happens if we change or lose the key? :/ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "397f84ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "backup = my_secret_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "45f618f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_secret_key = genkey(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0c821394",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'nX@_4{rwiiOqM\\x1d\\x0c\\r?.;@'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xor_username(my_secret_key, summary_by_order['e_user_id'][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1e9b9bb",
   "metadata": {},
   "source": [
    "### Finding Bounds and Outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "4e0baf62",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88F64QAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAiwUlEQVR4nO3de3DU1f3/8dcm2WyIYRPCJYGSAB2UiAjWILDVtgqBlDJ4ITNaZFpKGa1tYIS0KrQqYNsh0g54mYBOizD9I2JxCha5ZoKEUhOESJSITbWFQoWECibhYpaVnO8f/tifay6yyeZsdvN8zOzgns/JZ9+77/0kLz/Zk4/DGGMEAABgSUy4CwAAAD0L4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVXHhLuDLmpubdfLkSfXu3VsOhyPc5QAAgKtgjNG5c+c0aNAgxcS0f26j24WPkydPKiMjI9xlAACADjhx4oQGDx7c7pxuFz569+4t6fPi3W53h/fj8/m0a9cuTZkyRU6nM1TloQvQq8hBryIL/Yoc0dCrxsZGZWRk+H+Ot6fbhY8rv2pxu92dDh+JiYlyu90R28iegl5FDnoVWehX5IimXl3NRyb4wCkAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKyKC3cBiGxDF21tdfxY4TTLlQAAIgVnPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYFRfuAhCdhi7a2ur4scJplisBAHQ3nPkAAABWBRU+li5dKofDEXDLysryb29qalJ+fr769u2rpKQk5eXlqa6uLuRFAwCAyBX0mY8bbrhBp06d8t/27dvn37Zw4UJt2bJFGzduVFlZmU6ePKkZM2aEtGAAABDZgv7MR1xcnNLT01uMNzQ0aO3atSouLtbEiRMlSevWrdP111+viooKTZgwofPVAgCAiBd0+Pjggw80aNAgJSQkyOPxaPny5crMzFRlZaV8Pp9ycnL8c7OyspSZmany8vI2w4fX65XX6/Xfb2xslCT5fD75fL5gy/O78rWd2Qe+mivWBDW/tX7Qq8hBryIL/Yoc0dCrYGp3GGOu+qfH9u3bdf78eY0YMUKnTp3SsmXL9NFHH6m6ulpbtmzRnDlzAoKEJI0bN0533HGHnn766Vb3uXTpUi1btqzFeHFxsRITE6/6iQAAgPC5ePGi7r//fjU0NMjtdrc7N6jw8WX19fUaMmSIVq5cqV69enUofLR25iMjI0Mff/zxVxbfHp/Pp5KSEk2ePFlOp7PD+0H7Ri3d2el9HPrVRHoVITiuIgv9ihzR0KvGxkb169fvqsJHp/7OR0pKiq677jp9+OGHmjx5si5duqT6+nqlpKT459TV1bX6GZErXC6XXC5Xi3Gn0xmSBoRqP2id97Kj0/u40h96FTnoVWShX5EjknsVTN2d+jsf58+f17/+9S8NHDhQ2dnZcjqdKi0t9W+vqanR8ePH5fF4OvMwAAAgigR15uMXv/iFpk+friFDhujkyZNasmSJYmNjNXPmTCUnJ2vu3LkqKChQamqq3G635s+fL4/Hw0oXAADgF1T4+O9//6uZM2fqzJkz6t+/v2677TZVVFSof//+kqRVq1YpJiZGeXl58nq9ys3N1erVq7ukcAAAEJmCCh8bNmxod3tCQoKKiopUVFTUqaIAAED04touAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCqU39eHT3L0EVbw10CACAKcOYDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYFVcuAtAeA1dtLXF2LHCaWGopKXWapO6T30AgI7hzAcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArGKpbZQJxfLUtvYBAEAocOYDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWdCh+FhYVyOBxasGCBf6ypqUn5+fnq27evkpKSlJeXp7q6us7WCQAAokSHw8eBAwf04osvavTo0QHjCxcu1JYtW7Rx40aVlZXp5MmTmjFjRqcLBQAA0aFD4eP8+fOaNWuW/vCHP6hPnz7+8YaGBq1du1YrV67UxIkTlZ2drXXr1unNN99URUVFyIoGAACRq0PhIz8/X9OmTVNOTk7AeGVlpXw+X8B4VlaWMjMzVV5e3rlKAQBAVAj62i4bNmzQ22+/rQMHDrTYVltbq/j4eKWkpASMp6Wlqba2ttX9eb1eeb1e//3GxkZJks/nk8/nC7Y8vytf25l9RCJXrGl1vK3Xoa35NrXVq2CfC7peTz2uIhX9ihzR0Ktgag8qfJw4cUIPP/ywSkpKlJCQEHRhrVm+fLmWLVvWYnzXrl1KTEzs9P5LSko6vY9IsmJc6+Pbtm0Lar5NV3r05V4F+1xgT087riId/YockdyrixcvXvVchzHmqv/Xd/PmzbrnnnsUGxvrH7t8+bIcDodiYmK0c+dO5eTk6JNPPgk4+zFkyBAtWLBACxcubLHP1s58ZGRk6OOPP5bb7b7qJ/JlPp9PJSUlmjx5spxOZ4f3E2lGLd3Z6nj10tyg5tt06FcTW+1VsM8FXa+nHleRin5FjmjoVWNjo/r166eGhoav/Pkd1JmPSZMm6fDhwwFjc+bMUVZWlh577DFlZGTI6XSqtLRUeXl5kqSamhodP35cHo+n1X26XC65XK4W406nMyQNCNV+IoX3sqPV8bZeg7bm23Slti/3KtjnAnt62nEV6ehX5IjkXgVTd1Dho3fv3ho1alTA2DXXXKO+ffv6x+fOnauCggKlpqbK7XZr/vz58ng8mjBhQjAPBQAAolTQHzj9KqtWrVJMTIzy8vLk9XqVm5ur1atXh/phAABAhOp0+NizZ0/A/YSEBBUVFamoqKizuwYAAFEo5Gc+0D0NXbQ13CUAACCJC8sBAADLCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCqW2kawaFk+O2rpTq0Y9/m/V/Pn3lt73scKp3VFaQCALsCZDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYFRfuAoBQGLpoa6vjxwqnWa4EAPBVOPMBAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKtYaouoxhJcAOh+OPMBAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArAoqfKxZs0ajR4+W2+2W2+2Wx+PR9u3b/dubmpqUn5+vvn37KikpSXl5eaqrqwt50QAAIHIFFT4GDx6swsJCVVZW6uDBg5o4caLuuusuvffee5KkhQsXasuWLdq4caPKysp08uRJzZgxo0sKBwAAkSkumMnTp08PuP/b3/5Wa9asUUVFhQYPHqy1a9equLhYEydOlCStW7dO119/vSoqKjRhwoTQVQ0AACJWUOHjiy5fvqyNGzfqwoUL8ng8qqyslM/nU05Ojn9OVlaWMjMzVV5e3mb48Hq98nq9/vuNjY2SJJ/PJ5/P19Hy/F/bmX10d65YE+4SQsIVYwL+tSGa3xddqSccV9GEfkWOaOhVMLUHHT4OHz4sj8ejpqYmJSUladOmTRo5cqSqqqoUHx+vlJSUgPlpaWmqra1tc3/Lly/XsmXLWozv2rVLiYmJwZbXQklJSaf30V2tGBfuCkLr12ObrT3Wtm3brD1WNIrm4yoa0a/IEcm9unjx4lXPDTp8jBgxQlVVVWpoaNCrr76q2bNnq6ysLNjd+C1evFgFBQX++42NjcrIyNCUKVPkdrs7vF+fz6eSkhJNnjxZTqezw/vpzkYt3RnuEkLCFWP067HNeuJgjLzNjrDVUb00N2yPHSl6wnEVTehX5IiGXl35zcXVCDp8xMfHa/jw4ZKk7OxsHThwQM8++6zuu+8+Xbp0SfX19QFnP+rq6pSent7m/lwul1wuV4txp9MZkgaEaj/dkfdy+H5QdwVvsyOszyla3yddIZqPq2hEvyJHJPcqmLo7/Xc+mpub5fV6lZ2dLafTqdLSUv+2mpoaHT9+XB6Pp7MPAwAAokRQZz4WL16sqVOnKjMzU+fOnVNxcbH27NmjnTt3Kjk5WXPnzlVBQYFSU1Pldrs1f/58eTweVroAAAC/oMLH6dOn9cMf/lCnTp1ScnKyRo8erZ07d2ry5MmSpFWrVikmJkZ5eXnyer3Kzc3V6tWru6RwAAAQmYIKH2vXrm13e0JCgoqKilRUVNSpogAAQPTi2i4AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsKrDV7VF6A1dtDXcJQAA0OU48wEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCKC8sB/09bF/Y7VjjNciUAEN048wEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAq+LCXUBPNXTR1nCXgE5oq3/HCqdZrgQAIg9nPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFUttga/AsmgACC3OfAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsCip8LF++XLfccot69+6tAQMG6O6771ZNTU3AnKamJuXn56tv375KSkpSXl6e6urqQlo0AACIXEGFj7KyMuXn56uiokIlJSXy+XyaMmWKLly44J+zcOFCbdmyRRs3blRZWZlOnjypGTNmhLxwAAAQmYL68+o7duwIuL9+/XoNGDBAlZWV+va3v62GhgatXbtWxcXFmjhxoiRp3bp1uv7661VRUaEJEyaErnIAABCROnVtl4aGBklSamqqJKmyslI+n085OTn+OVlZWcrMzFR5eXmr4cPr9crr9frvNzY2SpJ8Pp98Pl+Ha7vytZ3ZR1dyxZpwl9BtuGJMwL+RrLu+30Klux9XCES/Ikc09CqY2h3GmA59x29ubtadd96p+vp67du3T5JUXFysOXPmBIQJSRo3bpzuuOMOPf300y32s3TpUi1btqzFeHFxsRITEztSGgAAsOzixYu6//771dDQILfb3e7cDp/5yM/PV3V1tT94dNTixYtVUFDgv9/Y2KiMjAxNmTLlK4tvj8/nU0lJiSZPniyn09mpGrvCqKU7w11Ct+GKMfr12GY9cTBG3mZHuMvpEtVLc8NdQkh09+MKgehX5IiGXl35zcXV6FD4mDdvnl5//XXt3btXgwcP9o+np6fr0qVLqq+vV0pKin+8rq5O6enpre7L5XLJ5XK1GHc6nSFpQKj2E2rey9H5Q7YzvM2OqH1duuN7sDO663GF1tGvyBHJvQqm7qBWuxhjNG/ePG3atEm7d+/WsGHDArZnZ2fL6XSqtLTUP1ZTU6Pjx4/L4/EE81AAACBKBXXmIz8/X8XFxXrttdfUu3dv1dbWSpKSk5PVq1cvJScna+7cuSooKFBqaqrcbrfmz58vj8fDShcAACApyPCxZs0aSdLtt98eML5u3Tr96Ec/kiStWrVKMTExysvLk9frVW5urlavXh2SYgEAQOQLKnxczcKYhIQEFRUVqaioqMNFAQCA6MW1XQAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYFdWE5BG/ooq3hLgHdQGvvg2OF08JQCQCEH2c+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVVzVNkS4ei0AAFeHMx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCpWuwDdTDArp44VTuvCSgCga3DmAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBUXluuAYC78BQAAAnHmAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWsdQWiEJtLQc/VjjNciUA0BJnPgAAgFVBh4+9e/dq+vTpGjRokBwOhzZv3hyw3RijJ598UgMHDlSvXr2Uk5OjDz74IFT1AgCACBd0+Lhw4YLGjBmjoqKiVrevWLFCzz33nF544QXt379f11xzjXJzc9XU1NTpYgEAQOQL+jMfU6dO1dSpU1vdZozRM888o8cff1x33XWXJOlPf/qT0tLStHnzZn3/+9/vXLUAACDihfQDp0ePHlVtba1ycnL8Y8nJyRo/frzKy8tbDR9er1der9d/v7GxUZLk8/nk8/k6XMuVr+3MPtriijUh32dP5ooxAf/2FG29N4N5fwW7j84eD115XCH06FfkiIZeBVO7wxjT4e/4DodDmzZt0t133y1JevPNN3Xrrbfq5MmTGjhwoH/evffeK4fDoVdeeaXFPpYuXaply5a1GC8uLlZiYmJHSwMAABZdvHhR999/vxoaGuR2u9udG/altosXL1ZBQYH/fmNjozIyMjRlypSvLL49Pp9PJSUlmjx5spxOZyhK9Ru1dGdI99fTuWKMfj22WU8cjJG32RHucqypXprb6ngw76+u3EdruvK4QujRr8gRDb268puLqxHS8JGeni5JqqurCzjzUVdXp5tuuqnVr3G5XHK5XC3GnU5nSBoQqv18kfdyz/kBaZO32dGjXtu23pfBvAZduY+v+ppI/QbZE9GvyBHJvQqm7pD+nY9hw4YpPT1dpaWl/rHGxkbt379fHo8nlA8FAAAiVNBnPs6fP68PP/zQf//o0aOqqqpSamqqMjMztWDBAv3mN7/Rtddeq2HDhumJJ57QoEGD/J8LAQAAPVvQ4ePgwYO64447/PevfF5j9uzZWr9+vR599FFduHBBDz74oOrr63Xbbbdpx44dSkhICF3VAAAgYgUdPm6//Xa1t0DG4XDoqaee0lNPPdWpwgAAQHQK+2oXoKdq6+Jv0aS158jF7QBwYTkAAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBWrXYAerq1VN121KiXYVT6sjgGiD2c+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAVS22BCNYTLk4HIPpw5gMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAVF5ZrBxftQk/W2vvfFWu0Ypw0aulOeS87/OPHCqfZLK1NbR2z3aU+AJ/jzAcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMCqHndhOS48BYQeF2G0g+9fiBac+QAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVvW41S5t4dP6QPcUimOztX0Es0KEVSZAaHHmAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWsdQWQI/UVUt4pehZxhuO2jq7LBqBuuv7izMfAADAqi4LH0VFRRo6dKgSEhI0fvx4vfXWW131UAAAIIJ0Sfh45ZVXVFBQoCVLlujtt9/WmDFjlJubq9OnT3fFwwEAgAjSJeFj5cqVeuCBBzRnzhyNHDlSL7zwghITE/XSSy91xcMBAIAIEvIPnF66dEmVlZVavHixfywmJkY5OTkqLy9vMd/r9crr9frvNzQ0SJLOnj0rn8/X4Tp8Pp8uXryoM2fOyOl0+sfjPrvQ4X2ia8Q1G1282Kw4X4wuNzvCXQ7aQa+uzpkzZ656blvfk1rbRzBzpba/D3ZFbaHS2mN25eN1F53tVVts9vDcuXOSJGPMV082IfbRRx8ZSebNN98MGH/kkUfMuHHjWsxfsmSJkcSNGzdu3Lhxi4LbiRMnvjIrhH2p7eLFi1VQUOC/39zcrLNnz6pv375yODr+f1aNjY3KyMjQiRMn5Ha7Q1Equgi9ihz0KrLQr8gRDb0yxujcuXMaNGjQV84Nefjo16+fYmNjVVdXFzBeV1en9PT0FvNdLpdcLlfAWEpKSsjqcbvdEdvInoZeRQ56FVnoV+SI9F4lJydf1byQf+A0Pj5e2dnZKi0t9Y81NzertLRUHo8n1A8HAAAiTJf82qWgoECzZ8/W2LFjNW7cOD3zzDO6cOGC5syZ0xUPBwAAIkiXhI/77rtP//vf//Tkk0+qtrZWN910k3bs2KG0tLSueLhWuVwuLVmypMWvdND90KvIQa8iC/2KHD2tVw5jrmZNDAAAQGhwbRcAAGAV4QMAAFhF+AAAAFYRPgAAgFVRGz6Kioo0dOhQJSQkaPz48XrrrbfCXVJU2bt3r6ZPn65BgwbJ4XBo8+bNAduNMXryySc1cOBA9erVSzk5Ofrggw8C5pw9e1azZs2S2+1WSkqK5s6dq/PnzwfMeffdd/Wtb31LCQkJysjI0IoVK1rUsnHjRmVlZSkhIUE33nijtm3bFvLnG8mWL1+uW265Rb1799aAAQN09913q6amJmBOU1OT8vPz1bdvXyUlJSkvL6/FHwo8fvy4pk2bpsTERA0YMECPPPKIPvvss4A5e/bs0c033yyXy6Xhw4dr/fr1Lerh2GzbmjVrNHr0aP8fmvJ4PNq+fbt/O33qvgoLC+VwOLRgwQL/GP1qR0gu6NLNbNiwwcTHx5uXXnrJvPfee+aBBx4wKSkppq6uLtylRY1t27aZX/3qV+Yvf/mLkWQ2bdoUsL2wsNAkJyebzZs3m3feecfceeedZtiwYebTTz/1z/nud79rxowZYyoqKszf/vY3M3z4cDNz5kz/9oaGBpOWlmZmzZplqqurzcsvv2x69eplXnzxRf+cv//97yY2NtasWLHCHDlyxDz++OPG6XSaw4cPd/lrEClyc3PNunXrTHV1tamqqjLf+973TGZmpjl//rx/zkMPPWQyMjJMaWmpOXjwoJkwYYL55je/6d/+2WefmVGjRpmcnBxz6NAhs23bNtOvXz+zePFi/5x///vfJjEx0RQUFJgjR46Y559/3sTGxpodO3b453Bstu+vf/2r2bp1q/nnP/9pampqzC9/+UvjdDpNdXW1MYY+dVdvvfWWGTp0qBk9erR5+OGH/eP0q21RGT7GjRtn8vPz/fcvX75sBg0aZJYvXx7GqqLXl8NHc3OzSU9PN7/73e/8Y/X19cblcpmXX37ZGGPMkSNHjCRz4MAB/5zt27cbh8NhPvroI2OMMatXrzZ9+vQxXq/XP+exxx4zI0aM8N+/9957zbRp0wLqGT9+vPnJT34S0ucYTU6fPm0kmbKyMmPM571xOp1m48aN/jnvv/++kWTKy8uNMZ+HzZiYGFNbW+ufs2bNGuN2u/39efTRR80NN9wQ8Fj33Xefyc3N9d/n2Axenz59zB//+Ef61E2dO3fOXHvttaakpMR85zvf8YcP+tW+qPu1y6VLl1RZWamcnBz/WExMjHJyclReXh7GynqOo0ePqra2NqAHycnJGj9+vL8H5eXlSklJ0dixY/1zcnJyFBMTo/379/vnfPvb31Z8fLx/Tm5urmpqavTJJ5/453zxca7Moddta2hokCSlpqZKkiorK+Xz+QJex6ysLGVmZgb068Ybbwz4Q4G5ublqbGzUe++955/TXi84NoNz+fJlbdiwQRcuXJDH46FP3VR+fr6mTZvW4jWlX+0L+1VtQ+3jjz/W5cuXW/w11bS0NP3jH/8IU1U9S21trSS12oMr22prazVgwICA7XFxcUpNTQ2YM2zYsBb7uLKtT58+qq2tbfdxEKi5uVkLFizQrbfeqlGjRkn6/LWMj49vcUHHL/ertdf5yrb25jQ2NurTTz/VJ598wrF5FQ4fPiyPx6OmpiYlJSVp06ZNGjlypKqqquhTN7Nhwwa9/fbbOnDgQIttHFfti7rwAaBt+fn5qq6u1r59+8JdCtowYsQIVVVVqaGhQa+++qpmz56tsrKycJeFLzlx4oQefvhhlZSUKCEhIdzlRJyo+7VLv379FBsb2+ITxXV1dUpPTw9TVT3Llde5vR6kp6fr9OnTAds/++wznT17NmBOa/v44mO0NYdetzRv3jy9/vrreuONNzR48GD/eHp6ui5duqT6+vqA+V/uV0d74Xa71atXL47NqxQfH6/hw4crOztby5cv15gxY/Tss8/Sp26msrJSp0+f1s0336y4uDjFxcWprKxMzz33nOLi4pSWlka/2hF14SM+Pl7Z2dkqLS31jzU3N6u0tFQejyeMlfUcw4YNU3p6ekAPGhsbtX//fn8PPB6P6uvrVVlZ6Z+ze/duNTc3a/z48f45e/fulc/n888pKSnRiBEj1KdPH/+cLz7OlTn0+v8zxmjevHnatGmTdu/e3eJXWdnZ2XI6nQGvY01NjY4fPx7Qr8OHDwcExpKSErndbo0cOdI/p71ecGx2THNzs7xeL33qZiZNmqTDhw+rqqrKfxs7dqxmzZrl/2/61Y5wf+K1K2zYsMG4XC6zfv16c+TIEfPggw+alJSUgE8Uo3POnTtnDh06ZA4dOmQkmZUrV5pDhw6Z//znP8aYz5fapqSkmNdee828++675q677mp1qe03vvENs3//frNv3z5z7bXXBiy1ra+vN2lpaeYHP/iBqa6uNhs2bDCJiYktltrGxcWZ3//+9+b99983S5YsYantl/z0pz81ycnJZs+ePebUqVP+28WLF/1zHnroIZOZmWl2795tDh48aDwej/F4PP7tV5YETpkyxVRVVZkdO3aY/v37t7ok8JFHHjHvv/++KSoqanVJIMdm2xYtWmTKysrM0aNHzbvvvmsWLVpkHA6H2bVrlzGGPnV3X1ztYgz9ak9Uhg9jjHn++edNZmamiY+PN+PGjTMVFRXhLimqvPHGG0ZSi9vs2bONMZ8vt33iiSdMWlqacblcZtKkSaampiZgH2fOnDEzZ840SUlJxu12mzlz5phz584FzHnnnXfMbbfdZlwul/na175mCgsLW9Ty5z//2Vx33XUmPj7e3HDDDWbr1q1d9rwjUWt9kmTWrVvnn/Ppp5+an/3sZ6ZPnz4mMTHR3HPPPebUqVMB+zl27JiZOnWq6dWrl+nXr5/5+c9/bnw+X8CcN954w9x0000mPj7efP3rXw94jCs4Ntv24x//2AwZMsTEx8eb/v37m0mTJvmDhzH0qbv7cvigX21zGGNMeM65AACAnijqPvMBAAC6N8IHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAq/4PHK9JYad/f/UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "summary_by_order.total_price.hist(bins=80)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "479358b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88F64QAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAfWklEQVR4nO3dfVBU5/n/8c9ClkUiYDBRpIKah5akVpNgMMS09QGljmPjhDG2pq2hTtJm0EbpQ6TTBEybSJw22iZETWJJO1PGNO2Y1NhIGBpxMpFEyTijycTGVgdHIuZJQByX/cH5/dG63yBUWVius7u8XzM7em7OnnNdsODHm73P8TiO4wgAAMBInNsFAACA4YXwAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFOXuV3Ahbq7u9Xc3Kzk5GR5PB63ywEAAP3gOI7a29uVkZGhuLiLz21EXPhobm5WZmam22UAAIABOH78uMaPH3/RfSIufCQnJ0v6T/EpKSkDPk4gENBrr72mefPmyev1hqu8iBHL/dFbdIrl3qTY7o/eolOk9dbW1qbMzMzgv+MXE3Hh4/yvWlJSUgYdPpKSkpSSkhIRX5Rwi+X+6C06xXJvUmz3R2/RKVJ7689bJnjDKQAAMEX4AAAApggfAADAFOEDAACYInwAAABThA8AAGCK8AEAAEwRPgAAgCnCBwAAMEX4AAAApggfAADAFOEDAACYInwAAABThA8AAGDqMrcLQGyYuGZnj+1jFQtcqgQAEOmY+QAAAKYIHwAAwBThAwAAmCJ8AAAAU4QPAABgivABAABMET4AAIApwgcAADBF+AAAAKYIHwAAwBThAwAAmCJ8AAAAU4QPAABgivABAABMET4AAIApwgcAADBF+AAAAKYIHwAAwBThAwAAmCJ8AAAAU4QPAABgivABAABMET4AAIApwgcAADBF+AAAAKYIHwAAwBThAwAAmCJ8AAAAU4QPAABgivABAABMET4AAIApwgcAADBF+AAAAKYIHwAAwBThAwAAmCJ8AAAAU4QPAABg6jK3C8DwMXHNzh7bxyoWuFQJAMBNzHwAAABThA8AAGCK8AEAAEwRPgAAgCnCBwAAMEX4AAAApggfAADAFOEDAACYInwAAABThA8AAGBqUOGjoqJCHo9Hq1atCo6dO3dOxcXFGj16tEaOHKnCwkK1tLQMtk4AABAjBhw+9u3bpy1btmjKlCk9xlevXq0dO3boxRdfVH19vZqbm3XnnXcOulAAABAbBhQ+zpw5o7vvvlvPPvusrrjiiuB4a2urtm7dqieeeEKzZ89WTk6Oqqqq9Oabb6qhoSFsRQMAgOg1oLvaFhcXa8GCBcrPz9evfvWr4HhjY6MCgYDy8/ODY9nZ2crKytLevXt166239jqW3++X3+8Pbre1tUmSAoGAAoHAQMoLPv/zf8aaSOvPF+/02O6rrv7s8/nxSOktnOgtesVyf/QWnSKtt1Dq8DiO41x6t/+zbds2Pfroo9q3b58SExM1c+ZM3Xjjjdq4caOqq6tVVFTUI0xIUm5urmbNmqXHH3+81/HKy8u1du3aXuPV1dVKSkoKpTQAAOCSs2fPaunSpWptbVVKSspF9w1p5uP48eN64IEHVFtbq8TExEEVeV5paalKSkqC221tbcrMzNS8efMuWfzFBAIB1dbWau7cufJ6veEoNaJEWn+Ty2t6bB8qLxjQPlLk9RZO9Ba9Yrk/eotOkdbb+d9c9EdI4aOxsVGnTp3SzTffHBzr6urSnj179NRTT6mmpkadnZ06ffq0Ro0aFdynpaVF6enpfR7T5/PJ5/P1Gvd6vWH5ZIbrOJEqUvrzd3l6bPdVU3/2ufDjkdDbUKC36BXL/dFbdIqU3kKpIaTwMWfOHB08eLDHWFFRkbKzs/Xggw8qMzNTXq9XdXV1KiwslCQdPnxYTU1NysvLC+VUAAAgRoUUPpKTkzV58uQeY5dffrlGjx4dHF++fLlKSkqUlpamlJQUrVy5Unl5eX2+2RQAAAw/A1rtcjEbNmxQXFycCgsL5ff7VVBQoKeffjrcpwEAAFFq0OFj9+7dPbYTExNVWVmpysrKwR4aAADEoLDPfAD9NXHNzl5jxyoWuFAJAMASN5YDAACmCB8AAMAU4QMAAJgifAAAAFOEDwAAYIrwAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFOEDwAAYIrwAQAATF3mdgEI3cQ1O+WLd7Q+V5pcXiN/l2dIb0U/cc3OHtuWt72/8NzW5wcAhB8zHwAAwBThAwAAmCJ8AAAAU4QPAABgivABAABMET4AAIApwgcAADDFdT4Qsr6uvTGQfQAAwxMzHwAAwBThAwAAmCJ8AAAAU4QPAABgivABAABMET4AAIApltoOcxcuieV29QCAocbMBwAAMEX4AAAApggfAADAFOEDAACYInwAAABThA8AAGCK8AEAAExxnY9hhNvcAwAiATMfAADAFOEDAACYInwAAABThA8AAGCK8AEAAEwRPgAAgCmW2iLqXLhk+FjFApcqAQAMBDMfAADAFOEDAACYInwAAABThA8AAGCK8AEAAEwRPgAAgCnCBwAAMMV1PmLYhdfDGKrnAAAQCmY+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFOEDwAAYIrwAQAATBE+AACAqZDCx6ZNmzRlyhSlpKQoJSVFeXl5evXVV4MfP3funIqLizV69GiNHDlShYWFamlpCXvRAAAgeoUUPsaPH6+Kigo1NjZq//79mj17tu644w69++67kqTVq1drx44devHFF1VfX6/m5mbdeeedQ1I4AACITiHd22XhwoU9th999FFt2rRJDQ0NGj9+vLZu3arq6mrNnj1bklRVVaXrr79eDQ0NuvXWW8NXNQAAiFoDfs9HV1eXtm3bpo6ODuXl5amxsVGBQED5+fnBfbKzs5WVlaW9e/eGpVgAABD9Qr6r7cGDB5WXl6dz585p5MiR2r59u2644QYdOHBACQkJGjVqVI/9x44dq5MnT/7P4/n9fvn9/uB2W1ubJCkQCCgQCIRaXtD55w7mGJHKF+/IF+f85+///bOvPn3xjmld4fD5r3sgEOhXD9H0NY7l12Us9ybFdn/0Fp0irbdQ6vA4jhPSv1CdnZ1qampSa2ur/vKXv+i5555TfX29Dhw4oKKioh5BQpJyc3M1a9YsPf74430er7y8XGvXru01Xl1draSkpFBKAwAALjl79qyWLl2q1tZWpaSkXHTfkMPHhfLz83XNNddoyZIlmjNnjj777LMesx8TJkzQqlWrtHr16j6f39fMR2Zmpj7++ONLFn8xgUBAtbW1mjt3rrxe74CPE4kml9fIF+fol9O69dD+OPm7PTpUXtDnftHmUHlBj6/dTY/+o1/PiRax/LqM5d6k2O6P3qJTpPXW1tamK6+8sl/hI+Rfu1you7tbfr9fOTk58nq9qqurU2FhoSTp8OHDampqUl5e3v98vs/nk8/n6zXu9XrD8skM13Eiib/L839/7/bI3+Xps8fP7xctPt+H1+vtVw/R+PWNxdflebHcmxTb/dFbdIqU3kKpIaTwUVpaqvnz5ysrK0vt7e2qrq7W7t27VVNTo9TUVC1fvlwlJSVKS0tTSkqKVq5cqby8PFa6AACAoJDCx6lTp/S9731PH374oVJTUzVlyhTV1NRo7ty5kqQNGzYoLi5OhYWF8vv9Kigo0NNPPz0khQMAgOgUUvjYunXrRT+emJioyspKVVZWDqooAAAQu7i3CwAAMDXoN5wCbpu4ZmevsWMVC1yoBADQH8x8AAAAU4QPAABgivABAABMET4AAIApwgcAADBF+AAAAKYIHwAAwBTX+YgRfV3rIhpNXLNTvnhH63PP35U3+m6OBwC4OGY+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFMstcWw0J+lyMcqFhhUAgBg5gMAAJgifAAAAFOEDwAAYIrwAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFOEDwAAYIrwAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFOEDwAAYIrwAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFOEDwAAYIrwAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFOEDwAAYIrwAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJgifAAAAFOEDwAAYIrwAQAATF3mdgHD3cQ1Oy+5z7GKBQaVxJb+fF778xw+9wAQfsx8AAAAU4QPAABgKqTwsW7dOt1yyy1KTk7WmDFjtGjRIh0+fLjHPufOnVNxcbFGjx6tkSNHqrCwUC0tLWEtGgAARK+Qwkd9fb2Ki4vV0NCg2tpaBQIBzZs3Tx0dHcF9Vq9erR07dujFF19UfX29mpubdeedd4a9cAAAEJ1CesPprl27emw///zzGjNmjBobG/W1r31Nra2t2rp1q6qrqzV79mxJUlVVla6//no1NDTo1ltvDV/lAAAgKg1qtUtra6skKS0tTZLU2NioQCCg/Pz84D7Z2dnKysrS3r17+wwffr9ffr8/uN3W1iZJCgQCCgQCA67t/HMHcwwLvnjnkvtc2IMv3pEv7j/PO/9nLImk3sL9+omW1+VAxHJvUmz3R2/RKdJ6C6UOj+M4A/oJ393drW9+85s6ffq03njjDUlSdXW1ioqKeoQJScrNzdWsWbP0+OOP9zpOeXm51q5d22u8urpaSUlJAykNAAAYO3v2rJYuXarW1lalpKRcdN8Bz3wUFxfr0KFDweAxUKWlpSopKQlut7W1KTMzU/Pmzbtk8RcTCARUW1uruXPnyuv1DqrGoTS5vOaS+xwqL+j1HF+co19O69ZD++Pk7/YMVXmuiKTeLvzcD1a0vC4HIpZ7k2K7P3qLTpHW2/nfXPTHgMLHihUr9Morr2jPnj0aP358cDw9PV2dnZ06ffq0Ro0aFRxvaWlRenp6n8fy+Xzy+Xy9xr1eb1g+meE6zlDxd136H9cL6//8c/zdnn4dIxpFQm9D9dqJ9NflYMRyb1Js90dv0SlSegulhpBWuziOoxUrVmj79u36xz/+oUmTJvX4eE5Ojrxer+rq6oJjhw8fVlNTk/Ly8kI5FQAAiFEhzXwUFxerurpaL7/8spKTk3Xy5ElJUmpqqkaMGKHU1FQtX75cJSUlSktLU0pKilauXKm8vDxWugAAAEkhho9NmzZJkmbOnNljvKqqSvfcc48kacOGDYqLi1NhYaH8fr8KCgr09NNPh6VYAAAQ/UIKH/1ZGJOYmKjKykpVVlYOuCgAABC7uLcLAAAwNaiLjAHDzcQ1O3uNHatYEPI+ADCcMfMBAABMET4AAIApwgcAADBF+AAAAKYIHwAAwBThAwAAmCJ8AAAAU4QPAABgivABAABMET4AAIApwgcAADBF+AAAAKYIHwAAwBThAwAAmLrM7QKGk75utY7IxtcMAMKPmQ8AAGCK8AEAAEwRPgAAgCnCBwAAMEX4AAAApggfAADAFEttowDLPQEAsYSZDwAAYIrwAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJjiOh/AIHEdFgAIDTMfAADAFOEDAACYInwAAABThA8AAGCK8AEAAEwRPgAAgCmW2gIRqq8lvMcqFrhQCQCEFzMfAADAFOEDAACYInwAAABThA8AAGCK8AEAAEwRPgAAgCnCBwAAMMV1PgAD56/Z4Yt3tD7X5WIAwGXMfAAAAFOEDwAAYIrwAQAATBE+AACAKcIHAAAwRfgAAACmWGobJn3d/hwAAPTGzAcAADBF+AAAAKYIHwAAwBThAwAAmCJ8AAAAU4QPAABgivABAABMcZ0PwAWTy2vk7/L0GDtWscClagDAFjMfAADAFOEDAACYCjl87NmzRwsXLlRGRoY8Ho9eeumlHh93HEcPP/ywxo0bpxEjRig/P18ffPBBuOoFAABRLuTw0dHRoalTp6qysrLPj69fv16/+93vtHnzZr311lu6/PLLVVBQoHPnzg26WAAAEP1CfsPp/PnzNX/+/D4/5jiONm7cqF/84he64447JEl//OMfNXbsWL300kv61re+NbhqAQBA1AvrapejR4/q5MmTys/PD46lpqZq+vTp2rt3b5/hw+/3y+/3B7fb2tokSYFAQIFAYMC1nH/uYI4RCl+8Y3Ke4PninB5/xpLh2tuFr9W+XlNWr+eBsP6esxbL/dFbdIq03kKpw+M4zoB/wns8Hm3fvl2LFi2SJL355puaMWOGmpubNW7cuOB+d911lzwej1544YVexygvL9fatWt7jVdXVyspKWmgpQEAAENnz57V0qVL1draqpSUlIvu6/p1PkpLS1VSUhLcbmtrU2ZmpubNm3fJ4i8mEAiotrZWc+fOldfrDUepFzW5vGbIz/F5vjhHv5zWrYf2x8nf7bn0E6LIcO3tUHlBj+2+XlMX7hNJrL/nrMVyf/QWnSKtt/O/ueiPsIaP9PR0SVJLS0uPmY+WlhbdeOONfT7H5/PJ5/P1Gvd6vWH5ZIbrOJdy4QWjrPi7Pa6de6gNt94ufJ321Xsk/IC5FKvvObfEcn/0Fp0ipbdQagjrdT4mTZqk9PR01dXVBcfa2tr01ltvKS8vL5ynAgAAUSrkmY8zZ87oyJEjwe2jR4/qwIEDSktLU1ZWllatWqVf/epXuu666zRp0iQ99NBDysjICL4vBAAADG8hh4/9+/dr1qxZwe3z79dYtmyZnn/+ef3sZz9TR0eH7rvvPp0+fVq33367du3apcTExPBVDQAAolbI4WPmzJm62AIZj8ejRx55RI888sigCgMAALGJe7sAAABTri+1jQYT1+zsNcbtz+GGC1+LvA4BRCNmPgAAgCnCBwAAMEX4AAAApggfAADAFOEDAACYInwAAABTLLUdoL6W3wKRYCCvTcsluyxdB8DMBwAAMEX4AAAApggfAADAFOEDAACYInwAAABThA8AAGCK8AEAAExxnQ8gQnDtGADDBTMfAADAFOEDAACYInwAAABThA8AAGCK8AEAAEwRPgAAgCmW2gJRLFzLc7nNPQBLzHwAAABThA8AAGCK8AEAAEwRPgAAgCnCBwAAMEX4AAAApggfAADAFNf5ANAvA70WSH+uRXLhPgO9xki4jgNgaDHzAQAATBE+AACAKcIHAAAwRfgAAACmCB8AAMAU4QMAAJhiqS2APoW6RNYX72h9rjS5vEb+Lk/Yz82yWSB2MPMBAABMET4AAIApwgcAADBF+AAAAKYIHwAAwBThAwAAmCJ8AAAAU8PuOh/cchsAAHcx8wEAAEwRPgAAgCnCBwAAMEX4AAAApggfAADAFOEDAACYGnZLbQFEJzeXyU9cs1O+eEfrc6XJ5TXyd3lcX6bPZQMQzZj5AAAApggfAADAFOEDAACYInwAAABThA8AAGCK8AEAAEwRPgAAgKlhf52PC9fKA4gO/fneHcrv76G6zobbP5P6Or/1NVXcOnescvtr2hdmPgAAgCnCBwAAMDVk4aOyslITJ05UYmKipk+frrfffnuoTgUAAKLIkISPF154QSUlJSorK9M777yjqVOnqqCgQKdOnRqK0wEAgCgyJOHjiSee0L333quioiLdcMMN2rx5s5KSkvT73/9+KE4HAACiSNhXu3R2dqqxsVGlpaXBsbi4OOXn52vv3r299vf7/fL7/cHt1tZWSdKnn36qQCAw4DoCgYDOnj2rTz75RF6vNzh+2f/rGPAxI8ll3Y7Onu3WZYE4dXV73C4nrOgtOsVyb9Kl+/vkk0/Cc54B/owazPk///Oyr/OHq7f+uPD8gz33//q3IBb0tzerr2l7e7skyXGcS+/shNmJEyccSc6bb77ZY/ynP/2pk5ub22v/srIyRxIPHjx48ODBIwYex48fv2RWcP06H6WlpSopKQlud3d369NPP9Xo0aPl8Qz8f09tbW3KzMzU8ePHlZKSEo5SI0os90dv0SmWe5Niuz96i06R1pvjOGpvb1dGRsYl9w17+LjyyisVHx+vlpaWHuMtLS1KT0/vtb/P55PP5+sxNmrUqLDVk5KSEhFflKESy/3RW3SK5d6k2O6P3qJTJPWWmprar/3C/obThIQE5eTkqK6uLjjW3d2turo65eXlhft0AAAgygzJr11KSkq0bNkyTZs2Tbm5udq4caM6OjpUVFQ0FKcDAABRZEjCx5IlS/TRRx/p4Ycf1smTJ3XjjTdq165dGjt27FCcrk8+n09lZWW9fqUTK2K5P3qLTrHcmxTb/dFbdIrm3jyO0581MQAAAOHBvV0AAIApwgcAADBF+AAAAKYIHwAAwNSwCB/Hjh3T8uXLNWnSJI0YMULXXHONysrK1NnZ6XZpYfHoo4/qtttuU1JSUlgv0OaGyspKTZw4UYmJiZo+fbrefvttt0sKiz179mjhwoXKyMiQx+PRSy+95HZJYbNu3TrdcsstSk5O1pgxY7Ro0SIdPnzY7bLCYtOmTZoyZUrwIk55eXl69dVX3S5rSFRUVMjj8WjVqlVulxIW5eXl8ng8PR7Z2dlulxU2J06c0He+8x2NHj1aI0aM0Fe+8hXt37/f7bL6bViEj/fff1/d3d3asmWL3n33XW3YsEGbN2/Wz3/+c7dLC4vOzk4tXrxY999/v9ulDMoLL7ygkpISlZWV6Z133tHUqVNVUFCgU6dOuV3aoHV0dGjq1KmqrKx0u5Swq6+vV3FxsRoaGlRbW6tAIKB58+apoyP6b+I4fvx4VVRUqLGxUfv379fs2bN1xx136N1333W7tLDat2+ftmzZoilTprhdSlh9+ctf1ocffhh8vPHGG26XFBafffaZZsyYIa/Xq1dffVXvvfeefvOb3+iKK65wu7T+C8/t5KLP+vXrnUmTJrldRlhVVVU5qampbpcxYLm5uU5xcXFwu6ury8nIyHDWrVvnYlXhJ8nZvn2722UMmVOnTjmSnPr6erdLGRJXXHGF89xzz7ldRti0t7c71113nVNbW+t8/etfdx544AG3SwqLsrIyZ+rUqW6XMSQefPBB5/bbb3e7jEEZFjMffWltbVVaWprbZeC/Ojs71djYqPz8/OBYXFyc8vPztXfvXhcrQ6haW1slKea+v7q6urRt2zZ1dHTE1K0iiouLtWDBgh7fe7Higw8+UEZGhq6++mrdfffdampqcruksPjb3/6madOmafHixRozZoxuuukmPfvss26XFZJhGT6OHDmiJ598Uj/4wQ/cLgX/9fHHH6urq6vXVXDHjh2rkydPulQVQtXd3a1Vq1ZpxowZmjx5stvlhMXBgwc1cuRI+Xw+/fCHP9T27dt1ww03uF1WWGzbtk3vvPOO1q1b53YpYTd9+nQ9//zz2rVrlzZt2qSjR4/qq1/9qtrb290ubdD+/e9/a9OmTbruuutUU1Oj+++/Xz/60Y/0hz/8we3S+i2qw8eaNWt6vaHowsf777/f4zknTpzQN77xDS1evFj33nuvS5Vf2kB6A9xWXFysQ4cOadu2bW6XEjZf+tKXdODAAb311lu6//77tWzZMr333ntulzVox48f1wMPPKA//elPSkxMdLucsJs/f74WL16sKVOmqKCgQH//+991+vRp/fnPf3a7tEHr7u7WzTffrMcee0w33XST7rvvPt17773avHmz26X125Dc28XKj3/8Y91zzz0X3efqq68O/r25uVmzZs3SbbfdpmeeeWaIqxucUHuLdldeeaXi4+PV0tLSY7ylpUXp6ekuVYVQrFixQq+88or27Nmj8ePHu11O2CQkJOjaa6+VJOXk5Gjfvn367W9/qy1btrhc2eA0Njbq1KlTuvnmm4NjXV1d2rNnj5566in5/X7Fx8e7WGF4jRo1Sl/84hd15MgRt0sZtHHjxvWafbv++uv117/+1aWKQhfV4eOqq67SVVdd1a99T5w4oVmzZiknJ0dVVVWKi4vsSZ9QeosFCQkJysnJUV1dnRYtWiTpP+m+rq5OK1ascLc4XJTjOFq5cqW2b9+u3bt3a9KkSW6XNKS6u7vl9/vdLmPQ5syZo4MHD/YYKyoqUnZ2th588MGYCh6SdObMGf3rX//Sd7/7XbdLGbQZM2b0Ws7+z3/+UxMmTHCpotBFdfjorxMnTmjmzJmaMGGCfv3rX+ujjz4KfiwW/lfd1NSkTz/9VE1NTerq6tKBAwckSddee61GjhzpbnEhKCkp0bJlyzRt2jTl5uZq48aN6ujoUFFRkdulDdqZM2d6/I/r6NGjOnDggNLS0pSVleViZYNXXFys6upqvfzyy0pOTg6+Ryc1NVUjRoxwubrBKS0t1fz585WVlaX29nZVV1dr9+7dqqmpcbu0QUtOTu71vpzLL79co0ePjon36/zkJz/RwoULNWHCBDU3N6usrEzx8fH69re/7XZpg7Z69Wrddttteuyxx3TXXXfp7bff1jPPPBPxM/o9uL3cxkJVVZUjqc9HLFi2bFmfvb3++utulxayJ5980snKynISEhKc3Nxcp6Ghwe2SwuL111/v82u0bNkyt0sbtP/1vVVVVeV2aYP2/e9/35kwYYKTkJDgXHXVVc6cOXOc1157ze2yhkwsLbVdsmSJM27cOCchIcH5whe+4CxZssQ5cuSI22WFzY4dO5zJkyc7Pp/Pyc7Odp555hm3SwqJx3EcxzLsAACA4S2y3/gAAABiDuEDAACYInwAAABThA8AAGCK8AEAAEwRPgAAgCnCBwAAMEX4AAAApggfAADAFOEDAACYInwAAABThA8AAGDq/wOf0PIBjGVyvAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "summary_by_order['z-score'] = stats.zscore(list(summary_by_order.total_price))\n",
    "\n",
    "summary_by_order['z-score'].hist(bins=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "79e27250",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "city                7\n",
       "state               7\n",
       "num_items           7\n",
       "total_price         7\n",
       "timestamp__order    7\n",
       "inbound_uri         7\n",
       "e_user_id           7\n",
       "z-score             7\n",
       "dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_by_order[summary_by_order.total_price > 27000].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "087cffa9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "city                5\n",
       "state               5\n",
       "num_items           5\n",
       "total_price         5\n",
       "timestamp__order    5\n",
       "inbound_uri         5\n",
       "e_user_id           5\n",
       "z-score             5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_by_order[summary_by_order.total_price < 1500].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "18cea16c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_order_outliers(total):\n",
    "    if total > 27000 or total < 1500:\n",
    "        # NOTE: we want to watch this over time if we rerun the report and make sure it is fitting.\n",
    "        # We could also normalize the data and use Interquartile Range (IQR) or Z-Scores \n",
    "        return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8c1e323b",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_by_order['remove'] = summary_by_order.total_price.map(remove_order_outliers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1f98c0c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "city                1000\n",
       "state               1000\n",
       "num_items           1000\n",
       "total_price         1000\n",
       "timestamp__order    1000\n",
       "inbound_uri         1000\n",
       "e_user_id           1000\n",
       "z-score             1000\n",
       "remove              1000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_by_order.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "0520e4a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_by_order = summary_by_order[summary_by_order['remove'] == False]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "828ac8d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_by_order = summary_by_order.drop(['remove'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0224a542",
   "metadata": {},
   "source": [
    "### Getting started with Great Expectations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "f6934e11",
   "metadata": {},
   "outputs": [],
   "source": [
    "import great_expectations as ge\n",
    "context = ge.get_context()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5b60254a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ge_df = ge.from_pandas(summary_by_order)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "22198399",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "  \"exception_info\": {\n",
       "    \"raised_exception\": false,\n",
       "    \"exception_traceback\": null,\n",
       "    \"exception_message\": null\n",
       "  },\n",
       "  \"meta\": {},\n",
       "  \"result\": {\n",
       "    \"element_count\": 988,\n",
       "    \"missing_count\": 0,\n",
       "    \"missing_percent\": 0.0,\n",
       "    \"unexpected_count\": 0,\n",
       "    \"unexpected_percent\": 0.0,\n",
       "    \"unexpected_percent_total\": 0.0,\n",
       "    \"unexpected_percent_nonmissing\": 0.0,\n",
       "    \"partial_unexpected_list\": []\n",
       "  },\n",
       "  \"success\": true\n",
       "}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ge_df.expect_column_values_to_be_between('total_price', 1500, 27000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b8210b0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "  \"expectation_suite_name\": \"default\",\n",
       "  \"meta\": {\n",
       "    \"great_expectations_version\": \"0.15.49\"\n",
       "  },\n",
       "  \"data_asset_type\": \"Dataset\",\n",
       "  \"ge_cloud_id\": null,\n",
       "  \"expectations\": [\n",
       "    {\n",
       "      \"meta\": {},\n",
       "      \"expectation_type\": \"expect_column_values_to_be_between\",\n",
       "      \"kwargs\": {\n",
       "        \"column\": \"total_price\",\n",
       "        \"min_value\": 1500,\n",
       "        \"max_value\": 27000\n",
       "      }\n",
       "    }\n",
       "  ]\n",
       "}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ge_df.get_expectation_suite(discard_failed_expectations=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e4f8887f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open(\"order_summary_for_sharing_expecation_file.json\", \"w\") as my_file:    \n",
    "    my_file.write(        \n",
    "        json.dumps(ge_df.get_expectation_suite().to_json_dict())\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "152463a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>num_items</th>\n",
       "      <th>total_price</th>\n",
       "      <th>timestamp__order</th>\n",
       "      <th>inbound_uri</th>\n",
       "      <th>e_user_id</th>\n",
       "      <th>z-score</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>order_number</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Walkerville</td>\n",
       "      <td>Delaware</td>\n",
       "      <td>605</td>\n",
       "      <td>9108</td>\n",
       "      <td>2045-06-23T14:05:41.176704</td>\n",
       "      <td>bag-fast-difficult</td>\n",
       "      <td>]\u001c",
       "\b\u001b[konEyONmNFS\u0017Fc\u0004</td>\n",
       "      <td>-0.238824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>North Adam</td>\n",
       "      <td>Georgia</td>\n",
       "      <td>252</td>\n",
       "      <td>4860</td>\n",
       "      <td>2025-10-06T02:48:18.176704</td>\n",
       "      <td>those-color-degree</td>\n",
       "      <td>V\u0018\\t\u0007\\:?$\u0004:\u0000\f",
       "&gt;NFS\u0017Fc\u0004</td>\n",
       "      <td>-1.063402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>West Jaredberg</td>\n",
       "      <td>Mississippi</td>\n",
       "      <td>305</td>\n",
       "      <td>4750</td>\n",
       "      <td>2046-11-03T19:15:02.176704</td>\n",
       "      <td>third-wish-spring</td>\n",
       "      <td>N\u001e",
       "\u0015\u0019W*~nEyONmNFS\u0017Fc\u0004</td>\n",
       "      <td>-1.084755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ryanchester</td>\n",
       "      <td>Alaska</td>\n",
       "      <td>440</td>\n",
       "      <td>6402</td>\n",
       "      <td>2038-10-10T12:43:40.176704</td>\n",
       "      <td>produce-might-a</td>\n",
       "      <td>^\u000f\u001b\u001aQ60w]yONmNFS\u0017Fc\u0004</td>\n",
       "      <td>-0.764085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thomasshire</td>\n",
       "      <td>Arizona</td>\n",
       "      <td>680</td>\n",
       "      <td>13420</td>\n",
       "      <td>2033-05-02T14:28:03.176704</td>\n",
       "      <td>whole-activity-new</td>\n",
       "      <td>X\u0018\\t\u001d",
       "G&lt;;)\\n7\u0015\u000f!\u000b",
       "\u0015S\u0017Fc\u0004</td>\n",
       "      <td>0.598178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>Ryanland</td>\n",
       "      <td>Idaho</td>\n",
       "      <td>161</td>\n",
       "      <td>6986</td>\n",
       "      <td>2029-02-28T09:00:44.176704</td>\n",
       "      <td>affect-coach</td>\n",
       "      <td>]\u0011\u001f\f",
       "\u0005`~nEyONmNFS\u0017Fc\u0004</td>\n",
       "      <td>-0.650725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>Jeremymouth</td>\n",
       "      <td>California</td>\n",
       "      <td>528</td>\n",
       "      <td>7896</td>\n",
       "      <td>2040-09-11T21:51:02.176704</td>\n",
       "      <td>film-and-nature</td>\n",
       "      <td>Q\u0012\u0015\u0010L:6/\u00175\u0000\u001a9\u000b",
       "FS\u0017Fc\u0004</td>\n",
       "      <td>-0.474085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>West Kennethtown</td>\n",
       "      <td>New Hampshire</td>\n",
       "      <td>1008</td>\n",
       "      <td>22144</td>\n",
       "      <td>2023-03-19T16:55:31.176704</td>\n",
       "      <td>if-follow-large</td>\n",
       "      <td>_\u0016\u001f\u001aAy~nEyONmNFS\u0017Fc\u0004</td>\n",
       "      <td>2.291592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>Courtneymouth</td>\n",
       "      <td>New Jersey</td>\n",
       "      <td>306</td>\n",
       "      <td>5274</td>\n",
       "      <td>2035-11-03T17:16:09.176704</td>\n",
       "      <td>growth-close-field</td>\n",
       "      <td>]\u0013\u001e",
       "\u0006P8? \u0001+\u000e\\n(NFS\u0017Fc\u0004</td>\n",
       "      <td>-0.983041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>South Pamela</td>\n",
       "      <td>South Dakota</td>\n",
       "      <td>378</td>\n",
       "      <td>5868</td>\n",
       "      <td>2022-11-14T17:21:31.176704</td>\n",
       "      <td>consumer-region-ago</td>\n",
       "      <td>D\u001f\u001b\u0006W&lt;,nEyONmNFS\u0017Fc\u0004</td>\n",
       "      <td>-0.867740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>988 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          city          state  num_items  total_price  \\\n",
       "order_number                                                            \n",
       "0                  Walkerville       Delaware        605         9108   \n",
       "1                   North Adam        Georgia        252         4860   \n",
       "2               West Jaredberg    Mississippi        305         4750   \n",
       "3                  Ryanchester         Alaska        440         6402   \n",
       "4                  Thomasshire        Arizona        680        13420   \n",
       "...                        ...            ...        ...          ...   \n",
       "995                   Ryanland          Idaho        161         6986   \n",
       "996                Jeremymouth     California        528         7896   \n",
       "997           West Kennethtown  New Hampshire       1008        22144   \n",
       "998              Courtneymouth     New Jersey        306         5274   \n",
       "999               South Pamela   South Dakota        378         5868   \n",
       "\n",
       "                        timestamp__order          inbound_uri  \\\n",
       "order_number                                                    \n",
       "0             2045-06-23T14:05:41.176704   bag-fast-difficult   \n",
       "1             2025-10-06T02:48:18.176704   those-color-degree   \n",
       "2             2046-11-03T19:15:02.176704    third-wish-spring   \n",
       "3             2038-10-10T12:43:40.176704      produce-might-a   \n",
       "4             2033-05-02T14:28:03.176704   whole-activity-new   \n",
       "...                                  ...                  ...   \n",
       "995           2029-02-28T09:00:44.176704         affect-coach   \n",
       "996           2040-09-11T21:51:02.176704      film-and-nature   \n",
       "997           2023-03-19T16:55:31.176704      if-follow-large   \n",
       "998           2035-11-03T17:16:09.176704   growth-close-field   \n",
       "999           2022-11-14T17:21:31.176704  consumer-region-ago   \n",
       "\n",
       "                           e_user_id   z-score  \n",
       "order_number                                    \n",
       "0               ]\n",
       "\b\u001b[konEyONmNFS\u0017Fc\u0004 -0.238824  \n",
       "1              V\u0018\\t\u0007\\:?$\u0004:\u0000\n",
       ">NFS\u0017Fc\u0004 -1.063402  \n",
       "2               N\n",
       "\u0015\u0019W*~nEyONmNFS\u0017Fc\u0004 -1.084755  \n",
       "3               ^\u000f\u001b\u001aQ60w]yONmNFS\u0017Fc\u0004 -0.764085  \n",
       "4             X\u0018\\t\n",
       "G<;)\\n7\u0015\u000f!\n",
       "\u0015S\u0017Fc\u0004  0.598178  \n",
       "...                              ...       ...  \n",
       "995             ]\u0011\u001f\n",
       "\u0005`~nEyONmNFS\u0017Fc\u0004 -0.650725  \n",
       "996             Q\u0012\u0015\u0010L:6/\u00175\u0000\u001a9\n",
       "FS\u0017Fc\u0004 -0.474085  \n",
       "997             _\u0016\u001f\u001aAy~nEyONmNFS\u0017Fc\u0004  2.291592  \n",
       "998            ]\u0013\n",
       "\u0006P8? \u0001+\u000e\\n(NFS\u0017Fc\u0004 -0.983041  \n",
       "999             D\u001f\u001b\u0006W<,nEyONmNFS\u0017Fc\u0004 -0.867740  \n",
       "\n",
       "[988 rows x 8 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_by_order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "1aa7c622",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_by_order.to_csv('data/order_summary_for_sharing.csv', escapechar='\\\\')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0d16950",
   "metadata": {},
   "source": [
    "### Challenges\n",
    "\n",
    "- Document the above pipeline and its privacy constraints in a text and graphic form, as if you would share it with fellow stakeholders. What is important for them to know? How can you communicate it to them concisely?\n",
    "- How would you test to ensure that differential privacy mechanism was used? How could you validate and audit pipelines that use anonymization?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0cea6e7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}