00_q&a_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "18b1d229-353d-456b-bfda-a86100b01fbb",
   "metadata": {},
   "source": [
    "# A/B Testing Course"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5084756-43cf-4c24-adf3-f9b81d8c516e",
   "metadata": {},
   "source": [
    "## Q&A Session 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e953726-be25-4c8b-9c27-4e7773e5b26c",
   "metadata": {},
   "source": [
    "### Q1. What are recommend methods for working with dependent data?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faf0971b-234f-4f3d-a1e2-93520af43fd8",
   "metadata": {},
   "source": [
    "Dependent data themselves do not really ruin our lives, except that we cannot achieve a uniform distribution of p-values in A/A tests.  \n",
    "Thus, you can make absolutely any transformations and if your A/A tests (p-value) are distributed uniformly (uniform distribution) with the fairness of the null hypothesis, then it can be said that you have come up with a new method of working with dependent data.   \n",
    "\n",
    "Often all these methods of working with dependent data are related to somehow correcting the variance. Thus, for t-test with dependent variables, you need to carefully recalculate the variances in the denominator taking into account the covariance that has appeared. That is, if you have a sequence of actions and understand how one action depends on another, then you can calculate the covariance and make all sorts of corrections to the variance for dependencies of variables when calculating t-test statistics.  \n",
    "\n",
    "The next thing we can talk about is when you don’t have a randomized experiment. A randomized experiment is when you first allocate control, then test, influence one of them and then compare, i.e. do everything as usual. But sometimes we cannot allocate independent control and test groups ourselves, for example, when people get vaccinated (because we cannot tell one group to get vaccinated and another not), and when people have already been vaccinated themselves, those who did it and those who did not are initially different, i.e. we do not have true randomization. Here you can use such methods as *matching* - when you try to select a group A’ among group A (usually among a larger group) that was as similar as possible to group B in initial characteristics, and then compare them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a222debd-d7db-4e16-919c-3d9a9615262a",
   "metadata": {},
   "source": [
    "### Q2. Where to find advanced knowledge about A/B testing?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a03f9e2-d886-4e01-98dc-670c780f7f55",
   "metadata": {},
   "source": [
    "- articles from top companies\n",
    "- university articles on statistics   \n",
    "\n",
    "For example, CUPED scientifically is a Hilbert space of random variables. The method was known more than 50-60 years ago."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "650138ed-07d2-46ab-acf1-27486e4aa080",
   "metadata": {},
   "source": [
    "### Q3. What is the best way to master advanced A/B testing techniques?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74671b6a-711a-4e77-ad66-460b1a908522",
   "metadata": {},
   "source": [
    "The easiest and most reliable way is to get a job at a company that uses them, and most importantly, has enough data to apply them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4294c0a6-2163-4e4e-ba4f-23cc4139123f",
   "metadata": {},
   "source": [
    "### Q4. Is a sufficiently large sample size needed when working with highly skewed metric distributions so that the CLT approximates the means well? Or should non-parametrics be used instead, such as Mann-Whitney?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd2e0640-af62-4a23-a60b-6bd7a5272912",
   "metadata": {},
   "source": [
    "1. Let's consider a typical example of data with right skew (e.g. data on average check). What can be done?  \n",
    "For example, stratify based on historical data. Roll back to a period in the past and do this: one stratum - people who bought nothing or little, another stratum - everyone else. Then you will have not such a large variance in the first stratum and also in the second stratum. Thus, in each of the strata, your metric will no longer be so skewed. And you can evaluate them independently, and then take a combination of their results in proportion to the weights from the general population.\n",
    "2. Next, if there is an additional skew (outlier) to the left.\n",
    "- What can be done?  \n",
    "Remove this outlier, estimate without it, but do not forget to mention it when reporting the results. Plus, think about how it happened that this outlier appeared and what can be done to minimize the probability of its appearance when conducting subsequent tests.\n",
    "- What else can be done?   \n",
    "Let’s decompose this metric and say that zero (who did not make a purchase) is 0, and everything from zero to plus infinity (those who made a purchase) is 1. Thus, from the average check (money), you can switch to such a metric as the share of people who performed some action (made a purchase). You can also try to split not into 0 and 1, but by the number of purchases (targeted actions), for example 0, 1, 2, 3, 4 etc. In this case there will be no big skewness, but unfortunately this will not be exactly the metric we need.  \n",
    "- You can also take only those who made purchases in group A and group B and compare them with each other, and exclude those with zero average check from both samples. That is, now we will compare sales per person in group A and sales per person in group B.\n",
    "- The second and third options separately here may not be very informative, because the number of people who made a purchase may decrease, and sales per person in the same group may increase, so you will need to multiply the results of the second and third options to get the final result. \n",
    "- Bucket method - google how to apply. We take users, randomly divide them into 100 buckets, count the metric in each bucket, thus we get 100 metrics in group A and 100 metrics in group B. And in the end we compare 100 metrics of group A with 100 metrics of group B. It should converge better."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c15c6476-9f99-4c27-b987-7986274e9066",
   "metadata": {},
   "source": [
    "### Q5. What to do if users are not registered? How to approach A/B testing?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64b558ca-18f3-432f-b8aa-90aa42d447c1",
   "metadata": {},
   "source": [
    "The main thing is to run an A/A test on historical data so that the p-values are distributed evenly. If yes, then everything is fine. You have found the right approach to selecting groups for unregistered users."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca975a5-79ec-4d6c-923d-fa9b1737d279",
   "metadata": {},
   "source": [
    "### Q6. What are bucket, salt, etc. How to calculate them?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07f898f4-9829-4d1b-a522-d9686cbbc6bb",
   "metadata": {},
   "source": [
    "In A/B testing, a bucket refers to a group of users who are randomly assigned to either variant A or variant B.  \n",
    "\n",
    "Salt is a term used in A/B testing to describe the process of adding random noise to the data in order to prevent bias and ensure that the test is statistically valid.  \n",
    "\n",
    "We have a large number of existing users and we have conditionally divided them into 100 buckets. We want to conduct an experiment, if we take users from one bucket for the control group and users from another bucket for the test group, then after several years of experiments our buckets will differ significantly from each other. Therefore, double hashing is used - we take two buckets, combine them into one large bucket and take a hash function on it: \n",
    "```\n",
    "if hash(id + salt) % 2 == 0:\n",
    "    control\n",
    "else:\n",
    "    test\n",
    "```\n",
    "\n",
    "After that we get our control and test groups. If you want to conduct an experiment again on users from these same two buckets, all you need to do is take another salt. Thus, users will be divided by an orthogonal partition and sequentially conducted experiments will not affect each other.\n",
    "\n",
    "We need other buckets (first level) to conduct many experiments in parallel. They allow us to control the intersection of experiments. The second level of salt and hash is needed so that users can be re-shuffled for subsequent experiments and so that buckets do not drift apart over time in their characteristics.\n",
    "\n",
    "To get 100 first-level buckets also `id+Salt % 100`.\n",
    "\n",
    "In large companies there are thousands and hundreds of thousands of first-level buckets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6e349b9-7e86-4541-9549-988e1c1d27ec",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Q7. Should data be tested for normality before applying parametric or non-parametric tests?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "970501e0-cce6-45ff-af9e-641da1fd96a9",
   "metadata": {},
   "source": [
    "To make the t-test work formally, the following three conditions must be met:\n",
    "- The numerator is normally distributed.\n",
    "- The denominator has a chi-square distribution.\n",
    "- The numerator and denominator are independent random variables.  \n",
    "\n",
    "If Xi and Yi are independent identically distributed random variables from a normal distribution, the first condition is met. But in real life, this condition is often ignored.  \n",
    "\n",
    "The distribution of the sample does not matter, because according to the Central Limit Theorem (CLT), the distribution of the sample means will always be normally distributed, and it doesn't matter what the distribution of the original sample is. Since mathematical statistics guarantees that if you take random variables and take their mean, then it turns out that with a sufficiently large sample size, this mean will tend towards a normal distribution, i.e., the t-test requirement is automatically fulfilled.  \n",
    "\n",
    "Therefore, there is no need to check data for normality, and the choice between the Student's t-test or the Mann-Whitney test is not based on the results of the normality test."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ac29791-49b6-4e07-8290-9128c3222753",
   "metadata": {},
   "source": [
    "### Q8. How do you choose between Student's t-test or Mann-Whitney test then? If not based on normality testing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6dff727-417f-4e3a-a3e6-fa8170f737e0",
   "metadata": {},
   "source": [
    "On historical data, you should resample your sample a large number of times (10,000 times). Each time, look at the p-value and remember it. Then, you build the distribution of your p-values:\n",
    "- if it is uniform, then you can use the t-test.\n",
    "- if it is not uniform, then you either have dependent data or outliers. In this case, you should think about how to get rid of these two problems.  \n",
    "\n",
    "Your ultimate goal is to obtain a uniform distribution on A/A tests. This will guarantee that your design and allocation are correct, and you control the occurrence of type I errors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a7fe687-07be-4c66-b7d7-3639e99cd4c3",
   "metadata": {},
   "source": [
    "### Q9. There are two models, and one of them is 1% better than the other. There are no users. How can you tell if this is a real difference and not just statistical error?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a229e65-120d-414c-95c8-a9e52e8ba377",
   "metadata": {},
   "source": [
    "You bootstrap pairs (target_i, predict_i) for each model, obtaining bootstrapped loss functions. You do the same for the second model. Now you have a large number of metrics. All that remains is to construct confidence intervals for the differences and see if zero falls within it. If it does, then there is no statistically significant change, but if zero lies outside the confidence interval, then we have managed to detect a statistically significant difference."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828d2c97-897e-474a-b292-a66259c1f9bb",
   "metadata": {},
   "source": [
    "### Q10. Is it permissible to neglect the correction for multiple testing when conducting multiple tests?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77471c42-1141-45ce-8189-f887cf527381",
   "metadata": {},
   "source": [
    "It is not permissible! Especially if you are making the final decision.\n",
    "\n",
    "What can be done: run 100 experiments for 100 different variations, for example, the color of the button. Look at the uplift across them. From the 100, select the top 2 results with the highest uplift. And after that, conduct a classic A/B/C with only two treatment options.\n",
    "\n",
    "That is, multiple testing is never for decision making. As a tool for filtering hypotheses, it can be applied to identify potentially good changes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d72aa94f-02c3-4b22-9eeb-048aebcbd224",
   "metadata": {},
   "source": [
    "### Q11. Which method should be used for prioritizing hypotheses?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "137751e0-0ca4-4612-863f-fda177f5dc0d",
   "metadata": {},
   "source": [
    "For this, it is better to use product managers and product owners, rather than analysts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e054d135-31b4-495e-8ed0-69e96961df74",
   "metadata": {},
   "source": [
    "### Q12. What is a random variable and what is the realization of a random variable?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "760c5c20-e67c-4490-8db8-29c68cbc6cab",
   "metadata": {},
   "source": [
    "A random variable is a mathematical abstract object. It has a normal distribution under certain conditions of independence of observations in the sample.\n",
    "\n",
    "A realization of a random variable is what we observe in the real world. A realization of a random variable is two numbers (the means of our control and test groups). And all you can do is take the difference between these means, divided by the variance (see t-test formula above) and see how much it deviates from what is expected. That is, it is expected to be around zero. If your difference in sample means falls within 1.96*σ, then you say that I have no reason to believe that the null hypothesis is false and we should reject it.\n",
    "\n",
    "In mathematical statistics, you observe only one sample mean (a realization of a random variable) and this is where all the complexity lies. But knowing the theory, we can say from which distribution it came given that the null hypothesis is true, and see if our observed value corresponds to the expected one. If it does not correspond, then we have grounds to reject the null hypothesis of equality of two means.\n",
    "\n",
    "The sample mean is normal according to the Central Limit Theorem."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0f0094f-945e-4f65-869a-a1423d5220f9",
   "metadata": {},
   "source": [
    "### Q13. What is better to use for CTR?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c53b2e-d337-41c2-92f9-8a0fc641edaa",
   "metadata": {},
   "source": [
    "If you are calculating CTR, you need linearization.\n",
    "\n",
    "Bootstrap - to build a confidence interval.\n",
    "\n",
    "t-test - if the data is independent.\n",
    "\n",
    "Linearization - if the data is dependent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55187fcc-bd9e-4a9d-8cdb-8e4f1966642c",
   "metadata": {},
   "source": [
    "### Q14. What is the hypothesis of the Kolmogorov-Smirnov test?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c3b05b7-0d84-4b63-b8f7-9004297af2ac",
   "metadata": {},
   "source": [
    "The null hypothesis of the Kolmogorov-Smirnov test is that the two distributions being compared are identical. However, it is typically used to test whether there is a significant shift in the mean of the two distributions, so this test is not particularly useful for most tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a16337-e6b4-46b2-ac4a-26895336f49c",
   "metadata": {},
   "source": [
    "### Q15. How to deal with the fact that t-test requires a normal distribution?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cec4d52-e175-4c27-941e-54d18b5d0254",
   "metadata": {},
   "source": [
    "It is **not** necessary for t-test to have a normal distribution. And there is no need to try to make it normal either."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a2f6851-1f17-4ab2-9430-953e2f0cc3c7",
   "metadata": {},
   "source": [
    "### Q16. How to evaluate A/B/C/D/... tests?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f51832c-bc00-419e-b9df-765597b2feea",
   "metadata": {},
   "source": [
    "Simply comparing each variant to each other is not an option, as it would lead to multiple comparisons.\n",
    "\n",
    "The proper approach is to compare each variant with group A one at a time, select the best hypotheses, and then conduct a classical A/B test."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98659554-d05b-43ba-9986-70eae08ec6fe",
   "metadata": {},
   "source": [
    "### Q17. How to properly evaluate conversions: in users or in events?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d529db0-97ef-48e5-baa4-01db1f72a668",
   "metadata": {},
   "source": [
    "It is correct to evaluate both in users and in events. These are simply two different metrics. Based on the speaker's experience, it is more appropriate to evaluate conversions in events and apply linearization."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9afefb4-fce4-4998-8431-fe99b4615e59",
   "metadata": {},
   "source": [
    "### Q18. Is it necessary to do an A/A test before every test?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfae8055-5f9c-4709-97e8-1a52cb6e0b0f",
   "metadata": {},
   "source": [
    "If you have the opportunity, it is better to do so. Especially if you do not have a reliable platform for online experiments.\n",
    "\n",
    "The best way to do this is to take a piece of historical data, conduct 1000 A/A tests on it, conduct 1000 simulations of A/B tests on it (artificially adding uplift). Take another piece of historical data, do the same thing on it and repeat the process 10 times.\n",
    "\n",
    "This way you will ensure that you control the type I error (via the A/A test) and the power (via the A/B simulations) of your experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "380263b6-e3c4-4670-bca1-62280366d8e1",
   "metadata": {},
   "source": [
    "### Q19. Should A/A/B tests be conducted?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca7a7f22-3ecb-470e-b6a0-3e4553eb66ea",
   "metadata": {},
   "source": [
    "In other words, taking a sample of customers, dividing them into two control groups and one test group, and then comparing the control groups with each other and the test group with the controls. - This should never be done because such an experiment does not give you any additional knowledge. Just one control group and the standard A/B option are enough.\n",
    "\n",
    "And controlling the type I error occurs on historical data when performing the classic A/A test."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd2a9b8a-9701-48be-ba83-03ba63bdb81f",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Q20. Tell me about one-sided and two-sided criteria?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20ca2a9f-6524-4cf7-83fc-ff5c87429e49",
   "metadata": {},
   "source": [
    "Single-tailed and two-tailed tests are used to determine statistical significance in hypothesis testing.\n",
    "\n",
    "If you have a question about which one to use and are unsure, it's best to use a two-tailed test.\n",
    "\n",
    "For example, if the metric in the experimental group has decreased, and your null hypothesis is that B>A, then you won't reject the null hypothesis. You'll only reject it if the metric in the experimental group has increased.\n",
    "\n",
    "Using a single-tailed test means that you'll miss out on the possibility of detecting unsuccessful experiments (losing half the information). Therefore, it's recommended to always use a two-tailed test."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cab1483-030c-41e6-84e4-abb2741d590e",
   "metadata": {},
   "source": [
    "### Q21. What examples of unsuccessful experiments can you give?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36b955dd-2da4-4c7f-863b-6233597db85a",
   "metadata": {},
   "source": [
    "Unsuccessful experiments happen often, sometimes due to various technical and business limitations. For example, you cannot conduct a treatment in only one part of the store and not conduct it in the other part. Or, for example, we do not have two cities like Moscow.\n",
    "\n",
    "During crises, sometimes everything was good in historical data, but everything has changed in reality. In such a situation, if possible, it is better to stop, wait it out, and when the situation normalizes again, run a repeat A/B test."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c78480a-1c85-437c-847c-9430acf7e1d3",
   "metadata": {},
   "source": [
    "### Q22. What is orthogonal user segmentation?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36732a9e-6d16-4381-b759-017941c2c617",
   "metadata": {},
   "source": [
    "During parallel testing of different changes (for example, website design and promotional code email), it is important to segment users into groups orthogonally. This means that the set of users for the website design change is segmented vertically, while the set of users for the email promotion is segmented horizontally. This way, we can conduct our two different tests honestly at the same time.\n",
    "\n",
    "When you intersect your experiments, there is always a risk. But if you don't intersect, you can greatly slow down your business by constantly waiting for one experiment to end before starting a new one. Therefore, if you do intersect, do it intelligently, for example, as shown above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53edde37-9458-45d0-84c3-2426627976b1",
   "metadata": {},
   "source": [
    "### Q23.How to calculate sample size in multiple comparisons?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20c56c40-a63c-4d5a-b86f-7d58e77ff3ff",
   "metadata": {},
   "source": [
    "Calculating sample size for multiple comparisons is similar to calculating it for a single comparison, except now we use alpha prime (adjusted for Bonferroni correction for multiple comparisons) instead of alpha. Beta is not affected by multiple comparisons."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bca674f2-9cf8-4fee-b5bd-e787b35f00a2",
   "metadata": {},
   "source": [
    "### Q24. Can 20 days be removed from the middle of a 100-day experiment?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a4c87a4-3848-4754-950a-3e15b86a1f75",
   "metadata": {},
   "source": [
    "The answer, as always, is simple:\n",
    "- It should have been built into the design.\n",
    "- The design should be extensively tested on historical AA and AB tests. If the p-value distribution is uniform on AA tests, and the required power is achieved on AB tests, then it can be done.\n",
    "- It is a purely practical question from there. It may not have been accounted for in the design, and something may have broken in the middle of the experiment. In this case, as mathematicians, we should restart the experiment, but the business may not allow us to do so. Therefore, we often have no choice but to continue the experiment. In any case, we need to estimate the probability of Type 1 and Type 2 errors for such a design based on historical data, and make adjustments if necessary."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}