T300683-01-onboarding-funnel-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8c9128c6",
   "metadata": {},
   "source": [
    "# Add an Image Funnel Analysis\n",
    "\n",
    "The phabricator task for this work is [T300683](https://phabricator.wikimedia.org/T300683)\n",
    "\n",
    "We're doing a similar analysis as we did for Add a Link. In the process, we'll apply some of our learnings from that analysis. Things to keep in mind are:\n",
    "\n",
    "* Carefully define the steps of the funnel in such a way that it translates into a straightforward SQL query.\n",
    "* Figure out if there are certain questions that are answered by calculating user proportions (e.g. the overarching key question of task completion is not necessarily covered by the funnel, but should instead allow for users to start the task multiple times.\n",
    "* The funnel does not end with a completed edit, but with the open question of whether the edit was reverted.\n",
    "* The number of users on mobile vs desktop was unexpected. Turns out that it might be because desktop users are asked to set topics and tasks to activate the module, whereas mobile users have immediate access to random tasks. We might therefore want to consider starting the funnel earlier (e.g. Visits Homepage) so that we can understand the path to clicking a task.\n",
    "\n",
    "From the Add a Link discussion, we get a lot of notes about possible paths and such, \"how many users do X?\", \"how many do X then Y?\", etc… These are always interesting, but will be either outside the scope of this analysis, or need to be answered with specific queries.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7eb96658",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from collections import defaultdict\n",
    "\n",
    "from wmfdata import spark, mariadb\n",
    "\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2bc5a746",
   "metadata": {},
   "outputs": [],
   "source": [
    "## We'll gather data from December 2021 and January 2022, as we per the data gathering\n",
    "## time have complete data for both months.\n",
    "\n",
    "start_date = dt.date(2021, 12, 1)\n",
    "end_date = dt.date(2022, 1, 31)\n",
    "\n",
    "## List of wikis that we're gathering data from:\n",
    "wikis = ['arwiki', 'bnwiki', 'cswiki']\n",
    "\n",
    "## Lists of known users to ignore (e.g. test accounts and experienced users)\n",
    "known_users = defaultdict(set)\n",
    "known_users['cswiki'].update([14, 127629, 303170, 342147, 349875, 44133, 100304, 307410, 439792, 444907,\n",
    "                              454862, 456272, 454003, 454846, 92295, 387915, 398470, 416764, 44751, 132801,\n",
    "                              137787, 138342, 268033, 275298, 317739, 320225, 328302, 339583, 341191,\n",
    "                              357559, 392634, 398626, 404765, 420805, 429109, 443890, 448195, 448438,\n",
    "                              453220, 453628, 453645, 453662, 453663, 453664, 440694, 427497, 272273,\n",
    "                              458025, 458487, 458049, 59563, 118067, 188859, 191908, 314640, 390445,\n",
    "                              451069, 459434, 460802, 460885, 79895, 448735, 453176, 467557, 467745,\n",
    "                              468502, 468583, 468603, 474052, 475184, 475185, 475187, 475188, 294174,\n",
    "                              402906, 298011])\n",
    "\n",
    "known_users['kowiki'].update([303170, 342147, 349875, 189097, 362732, 384066, 416362, 38759, 495265,\n",
    "                              515553, 537326, 566963, 567409, 416360, 414929, 470932, 472019, 485036,\n",
    "                              532123, 558423, 571587, 575553, 576758, 360703, 561281, 595100, 595105,\n",
    "                              595610, 596025, 596651, 596652, 596653, 596654, 596655, 596993, 942,\n",
    "                              13810, 536529])\n",
    "\n",
    "known_users['viwiki'].update([451842, 628512, 628513, 680081, 680083, 680084, 680085, 680086, 355424,\n",
    "                              387563, 443216, 682713, 659235, 700934, 705406, 707272, 707303, 707681, 585762])\n",
    "\n",
    "known_users['arwiki'].update([237660, 272774, 775023, 1175449, 1186377, 1506091, 1515147, 1538902,\n",
    "                              1568858, 1681813, 1683215, 1699418, 1699419, 1699425, 1740419, 1759328, 1763990])\n",
    "\n",
    "## Grab the user IDs of known test accounts so they can be added to the exclusion list\n",
    "\n",
    "def get_known_users(wiki):\n",
    "    '''\n",
    "    Get user IDs of known test accounts and return a set of them.\n",
    "    '''\n",
    "    \n",
    "    username_patterns = [\"MMiller\", \"Zilant\", \"Roan\", \"KHarlan\", \"MWang\", \"SBtest\",\n",
    "                         \"Cloud\", \"Rho2019\", \"Test\"]\n",
    "\n",
    "    known_user_query = '''\n",
    "SELECT user_id\n",
    "FROM user\n",
    "WHERE user_name LIKE \"{name_pattern}%\"\n",
    "    '''\n",
    "    \n",
    "    known_users = set()\n",
    "    \n",
    "    for u_pattern in username_patterns:\n",
    "        new_known = mariadb.run(known_user_query.format(\n",
    "            name_pattern = u_pattern), wiki)\n",
    "        known_users = known_users | set(new_known['user_id'])\n",
    "\n",
    "    return(known_users)\n",
    "        \n",
    "for wiki in wikis:\n",
    "    known_users[wiki] = known_users[wiki] | get_known_users(wiki)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9f11891",
   "metadata": {},
   "source": [
    "## Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "754cab83",
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_known_users_sql(kd, wiki_column, user_column):\n",
    "    '''\n",
    "    Based on the dictionary `kd` mapping wiki names to sets of user IDs of known users,\n",
    "    create a SQL expression to exclude users based on the name of the wiki matching `wiki_column`\n",
    "    and the user ID not matching `user_column`\n",
    "    '''\n",
    "    \n",
    "    wiki_exp = '''({w_column} = '{wiki}' AND {u_column} NOT IN ({id_list}))'''\n",
    "    \n",
    "    expressions = list()\n",
    "\n",
    "    ## Iteratively build the expression for each wiki\n",
    "    for wiki_name, wiki_users in kd.items():\n",
    "        expressions.append(wiki_exp.format(\n",
    "            w_column = wiki_column,\n",
    "            wiki = wiki_name,\n",
    "            u_column = user_column,\n",
    "            id_list = ','.join([str(u) for u in wiki_users])\n",
    "        ))\n",
    "    \n",
    "    ## We then join all the expressions with an OR, and we're done.\n",
    "    return(' OR '.join(expressions))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "04187ebb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_partition_statement(start_ts, end_ts, prefix = ''):\n",
    "    '''\n",
    "    This takes the two timestamps and creates a statement that selects\n",
    "    partitions based on `year`, `month`, and `day` in order to make our\n",
    "    data gathering not use excessive amounts of data. It assumes that\n",
    "    `start_ts` and `end_ts` are either in the same year, or if spanning\n",
    "    a year boundary are within a month apart.\n",
    "    This assumption simplifies the code and output a lot.\n",
    "    \n",
    "    An optional prefix can be set to enable selecting partitions for\n",
    "    multiple tables with different aliases.\n",
    "    \n",
    "    :param start_ts: start timestamp\n",
    "    :type start_ts: datetime.datetime\n",
    "    \n",
    "    :param end_ts: end timestamp\n",
    "    :type end_ts: datetime.datetime\n",
    "    \n",
    "    :param prefix: prefix to use in front of partition clauses, \".\" is added automatically\n",
    "    :type prefix: str\n",
    "    '''\n",
    "    \n",
    "    if prefix:\n",
    "        prefix = f'{prefix}.' # adds \".\" after the prefix\n",
    "    \n",
    "    # there are three cases:\n",
    "    # 1: month and year are the same, output a \"BETWEEN\" statement with the days\n",
    "    # 2: the years are the same, and the months differ by 1: output a statement for each month\n",
    "    # 3: the years are the same: create a list of statements from start_ts.month to end_ts.month,\n",
    "    #    return them OR'ed together\n",
    "    # 4: the years differ by 1, start_ts is December and end_ts is January, do the same as #2\n",
    "    # 5: anything else, raise an exception because this isn't implemented yet.\n",
    "    \n",
    "    if start_ts.year == end_ts.year and start_ts.month == end_ts.month:\n",
    "        return(f'''{prefix}year = {start_ts.year}\n",
    "AND {prefix}month = {start_ts.month}\n",
    "AND {prefix}day BETWEEN {start_ts.day} AND {end_ts.day}''')\n",
    "    elif start_ts.year == end_ts.year and (end_ts.month - start_ts.month) == 1:\n",
    "        return(f'''\n",
    "(\n",
    "    ({prefix}year = {start_ts.year}\n",
    "     AND {prefix}month = {start_ts.month}\n",
    "     AND {prefix}day >= {start_ts.day})\n",
    " OR ({prefix}year = {end_ts.year}\n",
    "     AND {prefix}month = {end_ts.month}\n",
    "     AND {prefix}day <= {end_ts.day})\n",
    ")''')\n",
    "    elif start_ts.year == end_ts.year:\n",
    "        # do the start month as a list\n",
    "        parts = [f'''({prefix}year = {start_ts.year}\n",
    "     AND {prefix}month = {start_ts.month}\n",
    "     AND {prefix}day >= {start_ts.day})''']\n",
    "        # for month +1 to end month, add each month\n",
    "        for m in range(start_ts.month+1, end_ts.month):\n",
    "            parts.append(f'''({prefix}year = {start_ts.year}\n",
    "            AND {prefix}month = {m})''')\n",
    "        # then append the end month and return a parenthesis OR'ed together of all of it\n",
    "        parts.append(f'''({prefix}year = {end_ts.year}\n",
    "     AND {prefix}month = {end_ts.month}\n",
    "     AND {prefix}day <= {end_ts.day})''')\n",
    "        return('({})'.format(\n",
    "            '\\nOR\\n'.join(parts)\n",
    "        ))\n",
    "    elif (end_ts.year - start_ts.year) == 1 and start_ts.month == 12 and end_ts.month == 1:\n",
    "        return(f'''\n",
    "(\n",
    "    ({prefix}year = {start_ts.year}\n",
    "     AND {prefix}month = {start_ts.month}\n",
    "     AND {prefix}day >= {start_ts.day})\n",
    " OR ({prefix}year = {end_ts.year}\n",
    "     AND {prefix}month = {end_ts.month}\n",
    "     AND {prefix}day <= {end_ts.day})\n",
    ")''')\n",
    "    else:\n",
    "        raise Exception('Difference between start and end timestamps is not implemented. See code for details.')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "aa8f3e46",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_variant_data(wikis, variant_property = 'growthexperiments-homepage-variant'):\n",
    "    '''\n",
    "    Connects to the given wikis and queries for the value of the user property that\n",
    "    defines what experiment variant the users are in. This can later be used to\n",
    "    filter out all users who are not in a specific variant\n",
    "    (e.g. \"imagerecommendation\" for Add an Image)\n",
    "    '''\n",
    "    \n",
    "    variant_query = f'''\n",
    "    SELECT\n",
    "      DATABASE() AS wiki,\n",
    "      up_user AS user_id,\n",
    "      up_value AS variant_name\n",
    "    FROM user_properties\n",
    "    WHERE up_property = \"{variant_property}\"\n",
    "    '''\n",
    "    \n",
    "    return(mariadb.run(variant_query, wikis))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94524640",
   "metadata": {},
   "source": [
    "## Funnel Definition\n",
    "\n",
    "Similarly as we did for Add a Link, we want to understand what happens the first time a user encounters the Add an Image task. This is because it's the only time we're guaranteed that they'll see the onboarding screen and thus help us learn their behaviour related to it.\n",
    "\n",
    "One thing we miss with this approach is that users might click a task only to find that they weren't interested in it and go back to find another, thus this first encounter isn't the session where they're actually working on the task. We could mitigate that by identifying the session where they take a certain action, but that again would mean that they could've seen onboarding previously and skipped it. In other words, that means we're back to \"users do X (in any session) followed by doing Y (in any session) within some amount of time\", and that's not a funnel analysis.\n",
    "\n",
    "```\n",
    "Step 1: Homepage visit: HomepageVisit entry + at least one HomepageModule impression entry.\n",
    "Step 2: Add an Image task click\n",
    "Step 3: Add an Image impression\n",
    "Step 4: Onboarding impression\n",
    "Step 5.1: Skipping onboarding, move to \"On task\"\n",
    "Step 5.2: Moving to Onboarding Step 2\n",
    "Step 6: Onboarding Step 2 impression\n",
    "Step 6.1: Skipping onboarding, move to \"On task\"\n",
    "Step 6.2: Moving to Onboarding Step 3\n",
    "Step 7: Onboarding Step 3 impression\n",
    "Step 7.1: Skipping onboarding, move to \"On task\"\n",
    "Step 7.2: Moving to onboarding Step 4\n",
    "Step 8: Onboarding Step 4 impression\n",
    "Step 8.1: Clicking \"Get started\", completing onboarding, move to \"On task\"\n",
    "Step 9: On task\n",
    "```\n",
    "\n",
    "I'd like to figure out a way to flag users clicking the \"Don't show this again\" checkbox. The state of that will be captured in 5.1, 6.1, 7.1, and 8.1.\n",
    "\n",
    "**Note:** None of the \"Moving to Onboarding Step…\" events can currently be part of the funnel, because the \"back\" and \"next\" buttons in the onboarding dialogue was not instrumented (filed as [T301486](https://phabricator.wikimedia.org/T301486)). We therefore use \"Onboarding Step… impression\" as the next step, and note that we won't be able to understand if we have a loss of users/data between clicking \"next\" and the impression of the next onboarding screen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ec1ce50b",
   "metadata": {},
   "outputs": [],
   "source": [
    "onboarding_funnel_query = '''\n",
    "WITH hp_visits AS (\n",
    "    SELECT\n",
    "        -- HomepageVisit is the authoritative source here, and we're grouping by\n",
    "        -- homepage_pageview_token to deduplicate the multiple module impressions\n",
    "        hpv.event.homepage_pageview_token,\n",
    "        FIRST_VALUE(hpv.wiki) AS wiki,\n",
    "        FIRST_VALUE(hpv.event.user_id) AS user_id,\n",
    "        FIRST_VALUE(hpv.event.is_mobile) AS is_mobile,\n",
    "        FIRST_VALUE(hpv.dt) AS visit_dt,\n",
    "        FIRST_VALUE(1) AS homepage_visit,\n",
    "        FIRST_VALUE(IF(ssac.event.userid IS NOT NULL, 1, 0)) AS is_newcomer,\n",
    "        FIRST_VALUE(IF(unix_timestamp(hpv.dt, \"yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'\") -\n",
    "                        unix_timestamp(ssac.dt, \"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\") < 60*60*24, 1, 0))\n",
    "            AS is_24hr_visit\n",
    "    FROM event.homepagevisit AS hpv\n",
    "    JOIN event.homepagemodule AS hpm\n",
    "    ON hpv.event.homepage_pageview_token = hpm.event.homepage_pageview_token\n",
    "    LEFT JOIN event.serversideaccountcreation AS ssac\n",
    "    ON hpv.wiki = ssac.wiki\n",
    "    AND hpv.event.user_id = ssac.event.userid\n",
    "    WHERE {hpv_partition_statement}\n",
    "    AND hpv.wiki IN ({wiki_list})\n",
    "    AND {hpv_known_user_id_expression}\n",
    "    AND {hpm_partition_statement}\n",
    "    AND {ssac_partition_statement}\n",
    "    AND ssac.wiki IN ({wiki_list})\n",
    "    AND hpv.event.is_mobile = true\n",
    "    AND hpm.event.action = \"impression\"\n",
    "    GROUP BY hpv.event.homepage_pageview_token\n",
    "),\n",
    "newcomer_tasks AS (\n",
    "    -- grab unique task token/task type data from newcomer tasks\n",
    "    SELECT\n",
    "        DISTINCT event.newcomer_task_token, event.task_type, event.page_id\n",
    "    FROM event.newcomertask\n",
    "    WHERE {partition_statement}\n",
    "),\n",
    "addimage_task_clicks AS (\n",
    "    -- clicks to Add an Image tasks\n",
    "    SELECT\n",
    "        event.homepage_pageview_token,\n",
    "        dt AS click_dt,\n",
    "        row_number() OVER (PARTITION BY hpm.wiki, hpm.event.user_id ORDER BY hpm.dt) AS click_number\n",
    "    FROM hp_visits AS hpv\n",
    "    JOIN event.homepagemodule AS hpm\n",
    "    ON hpv.homepage_pageview_token = hpm.event.homepage_pageview_token\n",
    "    JOIN newcomer_tasks AS nt\n",
    "    ON str_to_map(hpm.event.action_data, \";\", \"=\")[\"newcomerTaskToken\"] = nt.newcomer_task_token\n",
    "    WHERE {partition_statement}\n",
    "    AND hpm.wiki IN ({wiki_list})\n",
    "    AND event.action IN (\"se-task-click\", \"se-edit-button-click\")\n",
    "    AND nt.task_type = \"image-recommendation\"\n",
    "    AND dt > hpv.visit_dt\n",
    "),\n",
    "first_task_click AS (\n",
    "    SELECT\n",
    "        *\n",
    "    FROM addimage_task_clicks\n",
    "    WHERE click_number = 1\n",
    "),\n",
    "nosuggestion_impression AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN first_task_click AS ftc\n",
    "    ON stimg.homepage_pageview_token = ftc.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"nosuggestions_dialog\"\n",
    "    AND action = \"impression\"\n",
    "    AND stimg.dt > ftc.click_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "addimage_impression AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN first_task_click AS ftc\n",
    "    ON stimg.homepage_pageview_token = ftc.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface IN (\"machinesuggestions_mode\", \"recommendedimagetoolbar_dialog\")\n",
    "    AND action = \"impression\"\n",
    "    AND stimg.dt > ftc.click_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "onb_step1_impression AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN addimage_impression AS ai\n",
    "    ON stimg.homepage_pageview_token = ai.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"onboarding_step_1_dialog\"\n",
    "    AND action = \"impression\"\n",
    "    -- not limiting by timestamp because I'm unsure if it always occurs after\n",
    "    -- the first impression of the interface\n",
    "    GROUP BY stimg.homepage_pageview_token \n",
    "),\n",
    "onb_step1_skip AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN onb_step1_impression AS onsi\n",
    "    ON stimg.homepage_pageview_token = onsi.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"onboarding_step_1_dialog\"\n",
    "    AND action = \"skip_all\"\n",
    "    AND stimg.dt > onsi.event_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "onb_step2_impression AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN onb_step1_impression AS onsi\n",
    "    ON stimg.homepage_pageview_token = onsi.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"onboarding_step_2_dialog\"\n",
    "    AND action = \"impression\"\n",
    "    AND stimg.dt > onsi.event_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "onb_step2_skip AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN onb_step2_impression AS onsi\n",
    "    ON stimg.homepage_pageview_token = onsi.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"onboarding_step_2_dialog\"\n",
    "    AND action = \"skip_all\"\n",
    "    AND stimg.dt > onsi.event_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "onb_step3_impression AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN onb_step2_impression AS onsi\n",
    "    ON stimg.homepage_pageview_token = onsi.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"onboarding_step_3_dialog\"\n",
    "    AND action = \"impression\"\n",
    "    AND stimg.dt > onsi.event_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "onb_step3_skip AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN onb_step3_impression AS onsi\n",
    "    ON stimg.homepage_pageview_token = onsi.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"onboarding_step_3_dialog\"\n",
    "    AND action = \"skip_all\"\n",
    "    AND stimg.dt > onsi.event_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "onb_step4_impression AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN onb_step3_impression AS onsi\n",
    "    ON stimg.homepage_pageview_token = onsi.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"onboarding_step_4_dialog\"\n",
    "    AND action = \"impression\"\n",
    "    AND stimg.dt > onsi.event_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "onb_step4_getstarted AS (\n",
    "    SELECT\n",
    "        stimg.homepage_pageview_token,\n",
    "        MIN(dt) AS event_dt\n",
    "    FROM event.mediawiki_structured_task_article_image_suggestion_interaction AS stimg\n",
    "    JOIN onb_step4_impression AS onsi\n",
    "    ON stimg.homepage_pageview_token = onsi.homepage_pageview_token\n",
    "    WHERE {partition_statement}\n",
    "    AND active_interface = \"onboarding_step_4_dialog\"\n",
    "    AND action = \"get_started\"\n",
    "    AND stimg.dt > onsi.event_dt\n",
    "    GROUP BY stimg.homepage_pageview_token\n",
    "),\n",
    "on_task AS (\n",
    "    SELECT\n",
    "        homepage_pageview_token,\n",
    "        MIN(event_dt) AS event_dt\n",
    "    FROM (\n",
    "        SELECT homepage_pageview_token, event_dt\n",
    "        FROM onb_step1_skip\n",
    "        UNION ALL\n",
    "        SELECT homepage_pageview_token, event_dt\n",
    "        FROM onb_step2_skip\n",
    "        UNION ALL\n",
    "        SELECT homepage_pageview_token, event_dt\n",
    "        FROM onb_step3_skip\n",
    "        UNION ALL\n",
    "        SELECT homepage_pageview_token, event_dt\n",
    "        FROM onb_step4_getstarted\n",
    "    ) AS ontask_events\n",
    "    GROUP BY homepage_pageview_token\n",
    ")\n",
    "SELECT\n",
    "    hpv.*,\n",
    "    ftc.click_dt,\n",
    "    ftc.click_number,\n",
    "    IF(nosuggestion_impression.homepage_pageview_token IS NOT NULL, 1, 0) AS nosuggestion_impression,\n",
    "    nosuggestion_impression.event_dt AS nosuggestion_impression_dt,\n",
    "    IF(addimage_impression.homepage_pageview_token IS NOT NULL, 1, 0) AS addimage_impression,\n",
    "    addimage_impression.event_dt AS addimage_impression_dt,\n",
    "    IF(onb_step1_impression.homepage_pageview_token IS NOT NULL, 1, 0) AS onboarding_step1_impression,\n",
    "    onb_step1_impression.event_dt AS onboarding_step1_impression_dt,\n",
    "    IF(onb_step1_skip.homepage_pageview_token IS NOT NULL, 1, 0) AS onboarding_step1_skipall,\n",
    "    onb_step1_skip.event_dt AS onboarding_step1_skipall_dt,\n",
    "    IF(onb_step2_impression.homepage_pageview_token IS NOT NULL, 1, 0) AS onboarding_step2_impression,\n",
    "    onb_step2_impression.event_dt AS onboarding_step2_impression_dt,\n",
    "    IF(onb_step2_skip.homepage_pageview_token IS NOT NULL, 1, 0) AS onboarding_step2_skipall,\n",
    "    onb_step2_skip.event_dt AS onboarding_step2_skipall_dt,\n",
    "    IF(onb_step3_impression.homepage_pageview_token IS NOT NULL, 1, 0) AS onboarding_step3_impression,\n",
    "    onb_step3_impression.event_dt AS onboarding_step3_impression_dt,\n",
    "    IF(onb_step3_skip.homepage_pageview_token IS NOT NULL, 1, 0) AS onboarding_step3_skipall,\n",
    "    onb_step3_skip.event_dt AS onboarding_step3_skipall_dt,\n",
    "    IF(onb_step4_impression.homepage_pageview_token IS NOT NULL, 1, 0) AS onboarding_step4_impression,\n",
    "    onb_step4_impression.event_dt AS onboarding_step4_impression_dt,\n",
    "    IF(onb_step4_getstarted.homepage_pageview_token IS NOT NULL, 1, 0) AS onboarding_step4_getstarted,\n",
    "    onb_step4_getstarted.event_dt AS onboarding_step4_getstarted_dt,\n",
    "    IF(on_task.homepage_pageview_token IS NOT NULL, 1, 0) AS on_task,\n",
    "    on_task.event_dt AS on_task_dt\n",
    "FROM hp_visits AS hpv\n",
    "LEFT JOIN first_task_click AS ftc\n",
    "ON hpv.homepage_pageview_token = ftc.homepage_pageview_token\n",
    "LEFT JOIN nosuggestion_impression\n",
    "ON hpv.homepage_pageview_token = nosuggestion_impression.homepage_pageview_token\n",
    "LEFT JOIN addimage_impression\n",
    "ON hpv.homepage_pageview_token = addimage_impression.homepage_pageview_token\n",
    "LEFT JOIN onb_step1_impression\n",
    "ON hpv.homepage_pageview_token = onb_step1_impression.homepage_pageview_token\n",
    "LEFT JOIN onb_step1_skip\n",
    "ON hpv.homepage_pageview_token = onb_step1_skip.homepage_pageview_token\n",
    "LEFT JOIN onb_step2_impression\n",
    "ON hpv.homepage_pageview_token = onb_step2_impression.homepage_pageview_token\n",
    "LEFT JOIN onb_step2_skip\n",
    "ON hpv.homepage_pageview_token = onb_step2_skip.homepage_pageview_token\n",
    "LEFT JOIN onb_step3_impression\n",
    "ON hpv.homepage_pageview_token = onb_step3_impression.homepage_pageview_token\n",
    "LEFT JOIN onb_step3_skip\n",
    "ON hpv.homepage_pageview_token = onb_step3_skip.homepage_pageview_token\n",
    "LEFT JOIN onb_step4_impression\n",
    "ON hpv.homepage_pageview_token = onb_step4_impression.homepage_pageview_token\n",
    "LEFT JOIN onb_step4_getstarted\n",
    "ON hpv.homepage_pageview_token = onb_step4_getstarted.homepage_pageview_token\n",
    "LEFT JOIN on_task\n",
    "ON hpv.homepage_pageview_token = on_task.homepage_pageview_token\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a324ae54",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PySpark executors will use /usr/lib/anaconda-wmf/bin/python3.\n"
     ]
    }
   ],
   "source": [
    "mobile_onb_funnel_data = spark.run(\n",
    "    onboarding_funnel_query.format(\n",
    "        wiki_list = ','.join(['\"{}\"'.format(w) for w in wikis]),\n",
    "        hpv_known_user_id_expression = make_known_users_sql(known_users, 'hpv.wiki', 'hpv.event.user_id'),\n",
    "        hpv_partition_statement = make_partition_statement(start_date, end_date, 'hpv'),\n",
    "        hpm_partition_statement = make_partition_statement(start_date, end_date, 'hpm'),\n",
    "        ssac_partition_statement = make_partition_statement(start_date, end_date, 'ssac'),\n",
    "        partition_statement = make_partition_statement(start_date, end_date),\n",
    "    ), session_type = 'yarn-large'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93074035",
   "metadata": {},
   "outputs": [],
   "source": [
    "mobile_onb_funnel_data[mobile_onb_funnel_data.duplicated(subset = ['homepage_pageview_token'], keep = False)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "98b19169",
   "metadata": {},
   "outputs": [],
   "source": [
    "variant_data = get_variant_data(wikis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1285f1fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "variant_data['variant_name'] = variant_data['variant_name'].apply(lambda v: v.decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2323dad7",
   "metadata": {},
   "outputs": [],
   "source": [
    "variant_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3e2e795b",
   "metadata": {},
   "outputs": [],
   "source": [
    "mobile_onb_funnel_data = mobile_onb_funnel_data.merge(variant_data,\n",
    "                                                     on = ['wiki', 'user_id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db31cb77",
   "metadata": {},
   "source": [
    "## Discarding Invalid Sessions\n",
    "\n",
    "Later investigation of task completion based on whether users skipped or completed onboarding revealed that we have some invalid sessions where users did both. We'll discard those sessions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "244947c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "mobile_onb_funnel_data.loc[\n",
    "    ((mobile_onb_funnel_data['onboarding_step1_skipall'] == 1) |\n",
    "     (mobile_onb_funnel_data['onboarding_step2_skipall'] == 1) |\n",
    "     (mobile_onb_funnel_data['onboarding_step3_skipall'] == 1)\n",
    "    ) &\n",
    "    (mobile_onb_funnel_data['onboarding_step4_getstarted'] == 1),\n",
    "    'homepage_pageview_token'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e9bce4b",
   "metadata": {},
   "source": [
    "# Homepage Visit to Onboarding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0d352821",
   "metadata": {},
   "outputs": [],
   "source": [
    "def addimage_mobile_newcomers(df):\n",
    "    return(df.loc[\n",
    "        (df['variant_name'] == 'imagerecommendation') &\n",
    "        (df['is_newcomer'] == 1) &\n",
    "        (df['is_24hr_visit'] == 1) &\n",
    "        (df['is_mobile'] == True)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3dfa4010",
   "metadata": {},
   "outputs": [],
   "source": [
    "def round_perc(x, y, prec = 1):\n",
    "    return(round(100.0 * x / y, prec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "46d736b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def round_perc_df(df_x, df_y, prec = 1):\n",
    "    return(round(100.0 * len(df_x) / len(df_y), prec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "94dee787",
   "metadata": {},
   "outputs": [],
   "source": [
    "mob_newcomers = addimage_mobile_newcomers(mobile_onb_funnel_data.loc[\n",
    "    ~mobile_onb_funnel_data['homepage_pageview_token'].isin(\n",
    "        mobile_onb_funnel_data.loc[\n",
    "            ((mobile_onb_funnel_data['onboarding_step1_skipall'] == 1) |\n",
    "             (mobile_onb_funnel_data['onboarding_step2_skipall'] == 1) |\n",
    "             (mobile_onb_funnel_data['onboarding_step3_skipall'] == 1)\n",
    "            ) &\n",
    "            (mobile_onb_funnel_data['onboarding_step4_getstarted'] == 1),\n",
    "            'homepage_pageview_token'\n",
    "        ]\n",
    "    )\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a081483",
   "metadata": {},
   "source": [
    "Number of visitors to the Homepage is the whole dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "9e1fb77f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15031"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa20690b",
   "metadata": {},
   "source": [
    "Number of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "928b6be1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1490"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[mob_newcomers['click_number'] > 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "587e78ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.9"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[mob_newcomers['click_number'] > 0],\n",
    "    mob_newcomers\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af6e9ec",
   "metadata": {},
   "source": [
    "Number of users who got the no suggestions dialogue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "2cb75a94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "123acf74",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "09b998c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.4"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[mob_newcomers['click_number'] > 0]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4ea2d50",
   "metadata": {},
   "source": [
    "Proportion out of all visitors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "719b0c55",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59f57576",
   "metadata": {},
   "source": [
    "Number of users who saw onboarding Step 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "48961737",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1214"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "]) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de4c5d52",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "95253152",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "81.5"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0)  &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[mob_newcomers['click_number'] > 0]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "430fa8dd",
   "metadata": {},
   "source": [
    "Proportion out of all visitors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "efc4979e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.1"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c585723e",
   "metadata": {},
   "source": [
    "Bounces are the ones who are left:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "70c16929",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "240"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    len(mob_newcomers.loc[(mob_newcomers['click_number'] > 0)]) -\n",
    "    len(mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 1)\n",
    "    ]) -\n",
    "    len(mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ]) \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05aaa748",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "2fd0f422",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16.1"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    (\n",
    "        len(mob_newcomers.loc[(mob_newcomers['click_number'] > 0)]) -\n",
    "        len(mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 1)\n",
    "        ]) -\n",
    "        len(mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "        ]) \n",
    "    ),\n",
    "    len(mob_newcomers.loc[mob_newcomers['click_number'] > 0])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758fa329",
   "metadata": {},
   "source": [
    "Proportion out of all visitors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "1ab26339",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.6"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    (\n",
    "        len(mob_newcomers.loc[(mob_newcomers['click_number'] > 0)]) -\n",
    "        len(mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 1)\n",
    "        ]) -\n",
    "        len(mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "        ]) \n",
    "    ),\n",
    "    len(mob_newcomers)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "049d962c",
   "metadata": {},
   "source": [
    "## Onboarding Step 1\n",
    "\n",
    "Just a reminder of how many users saw Onboarding Step 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "83f11437",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1214"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bea0f385",
   "metadata": {},
   "source": [
    "How many of them clicked \"Skip all\" at this stage?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "009b44a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "357"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step1_skipall'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cbe9c08",
   "metadata": {},
   "source": [
    "Proportion out of users who saw onboarding step 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "41430305",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.4"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de47c8d",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "589187ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24.6"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "883d5626",
   "metadata": {},
   "source": [
    "Number of users who clicked \"next\" (meaning they didn't click \"skip all\" *and* saw step 2):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "22b968d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "763"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9063d7b",
   "metadata": {},
   "source": [
    "Proportion out of users who saw onboarding step 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "6b1939c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "62.9"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17907cfa",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "c60b2b73",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "52.5"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "795cd8a8",
   "metadata": {},
   "source": [
    "Users who bounced are those who didn't do either of the two preceeding actions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "260fdee9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "94"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e3968df",
   "metadata": {},
   "source": [
    "Proportion out of users who saw onboarding step 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "31d8fa34",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.7"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "        ]\n",
    "    ),\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5560a807",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "745c2f1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.5"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "        ]\n",
    "    ),\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ab7dabb",
   "metadata": {},
   "source": [
    "Number of users who saw Step 2 (after not skipping step 1), and who there clicked \"Skip all\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "2264e48d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "71"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step2_skipall'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ce84772",
   "metadata": {},
   "source": [
    "Proportion out of users who saw Step 2 (without skipping on Step 1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "7e4e1a81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.3"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe858d3",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "1b9dc0bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.9"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9558c11",
   "metadata": {},
   "source": [
    "Number of users who saw Step 3 (after not skipping step 1 or 2):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "6297677e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "683"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd21fb81",
   "metadata": {},
   "source": [
    "Proportion out of users who saw Step 2 (without skipping on Step 1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "718c059d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "89.5"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0dd014c",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "fc429754",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "47.0"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27403ac9",
   "metadata": {},
   "source": [
    "Users who bounced are those who didn't do either of the two preceeding actions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "d3ae6eeb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65832628",
   "metadata": {},
   "source": [
    "Proportion out of users who saw onboarding step 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "5979d535",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "        ]\n",
    "    ),\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56b897ac",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "dda5db74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "        ]\n",
    "    ),\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f833d5",
   "metadata": {},
   "source": [
    "Number of users on Step 3 (after not skipping step 1 or 2), who then clicked \"Skip all\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "f4d6fa79",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "37"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step3_skipall'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfe3eb55",
   "metadata": {},
   "source": [
    "Proportion out of users who saw Step 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "6b1d32cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.4"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d391f4",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "abbba906",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.5"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "884770b5",
   "metadata": {},
   "source": [
    "Number of users on Step 3 who clicked \"Next\" (meaning they didn't skip and saw Step 4):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "469457f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "643"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52cc7947",
   "metadata": {},
   "source": [
    "Proportion out of users who saw Step 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "d841cc1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "94.1"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26f3ee7c",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "82a6a392",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44.2"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7f637aa",
   "metadata": {},
   "source": [
    "Users who bounced are those who didn't do either of the two preceeding actions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "cb97ce2f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step3_skipall'] == 1)\n",
    "        ]    \n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd81a01b",
   "metadata": {},
   "source": [
    "Proportion out of users who saw onboarding step 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "43e9794d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step3_skipall'] == 1)\n",
    "        ]    \n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "        ]\n",
    "    ),\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7be78d16",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "ed254691",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1)\n",
    "        ]\n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step3_skipall'] == 1)\n",
    "        ]    \n",
    "    ) -\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "            (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "            (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "            (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "        ]\n",
    "    ),\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a39ff101",
   "metadata": {},
   "source": [
    "Number of users on Step 4 (after not skipping step 1, 2, or 3), who then completed onboarding by clicking \"Get started\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "fc59d2a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "635"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "    (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "    (mob_newcomers['onboarding_step4_impression'] == 1) &\n",
    "    (mob_newcomers['onboarding_step4_getstarted'] == 1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4e11d2a",
   "metadata": {},
   "source": [
    "Proportion out of users who saw Step 4:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "de893477",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "98.8"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step4_getstarted'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59514914",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "7b6d01bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43.7"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step4_getstarted'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "    (mob_newcomers['click_number'] > 0) &\n",
    "    (mob_newcomers['nosuggestion_impression'] == 0)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01fcf1fb",
   "metadata": {},
   "source": [
    "Number of users on Step 4 who bounced are those who reached that step but didn't click \"Get started\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "366ef1dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    len(mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "    ]) -\n",
    "    len(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step4_getstarted'] == 1)\n",
    "    ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6f18ccc",
   "metadata": {},
   "source": [
    "Proportion out of users who saw onboarding Step 4:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "a1fadb0b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    len(mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "    ]) -\n",
    "    len(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step4_getstarted'] == 1)\n",
    "    ]\n",
    "    ),\n",
    "    len(mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "    ])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a7d0e2a",
   "metadata": {},
   "source": [
    "Proportion out of users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "fea83515",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc(\n",
    "    len(mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1)\n",
    "    ]) -\n",
    "    len(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step4_getstarted'] == 1)\n",
    "    ]\n",
    "    ),\n",
    "    len(\n",
    "        mob_newcomers.loc[\n",
    "            (mob_newcomers['click_number'] > 0) &\n",
    "            (mob_newcomers['nosuggestion_impression'] == 0)\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba336ec2",
   "metadata": {},
   "source": [
    "Number of users reaching \"On Task\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "3fab06fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1100"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['on_task'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1768ca36",
   "metadata": {},
   "source": [
    "Proportion of users reaching \"On task\" out of users who saw Step 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "f9ef346c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "90.6"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['on_task'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba5df4d7",
   "metadata": {},
   "source": [
    "Proportion of users reaching \"On task\" out of all users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "99ac1c95",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "75.7"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['on_task'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5780f5c0",
   "metadata": {},
   "source": [
    "Proportion of users skipping on Step 2 out of users who saw Step 1, and out of all users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "13c7d806",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.8"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "d96b49ba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.9"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16847ee9",
   "metadata": {},
   "source": [
    "Proportion of users skipping on Step 3 out of users who saw Step 1, and out of all users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "f0a66769",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "b386bc2a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.5"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae1c37dd",
   "metadata": {},
   "source": [
    "Proportion of users clicking \"Get started\" on Step 4 out of users who saw Step 1, and out of all users who clicked a task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "2eb76881",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "52.3"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step4_getstarted'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "b6602d8e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43.7"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round_perc_df(\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0) &\n",
    "        (mob_newcomers['onboarding_step1_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step1_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step2_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step2_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step3_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step3_skipall'] == 0) &\n",
    "        (mob_newcomers['onboarding_step4_impression'] == 1) &\n",
    "        (mob_newcomers['onboarding_step4_getstarted'] == 1)\n",
    "    ],\n",
    "    mob_newcomers.loc[\n",
    "        (mob_newcomers['click_number'] > 0) &\n",
    "        (mob_newcomers['nosuggestion_impression'] == 0)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "712400b6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}