Grocery Website Download Page AB Testing

In this use case I am working as an analyst for a large grossery chain. One of the goals our company has, it's to drive more customers to download thier mobile app and register for the loyalty program.

One of the goals of the company discussed here is to get more users to download the mobile app and sign up for the customer loyalty program.

The company is making a link change to a button in the app store and asking if clickthrough rates will increase for the app download page?

1- Explain methodology step-by-step :

• What data do we have ?

  • IP Address: Client's IP Address
  • LoggedInFlag: Indicates whether the user has a profile on our website
  • ServerID: Belonging to the Treatment and Control group, 1: Treatment , 2-3: Control
  • VisitPageFlag: The status of the user clicking the download link

• Whats the Duration Of the Test ?

  • 1 Weak

• What's our unit of analysis

  • IP Address: Client's IP Address

• What's our metric to measure success here

  • Number clicked in download

• The size of our treatment and control groups

  • 1/3 of the treatment units and
  • 2/3 of control units.

• What's our MDE (Minimum Detectable Effect) ?

  • MDE 1%

• Apply hypothesis testing and check assumptions

  • Check Normality & Homogeneity
  • Apply tests (Shapiro, Levene Test, T-Test, Welch Test, Mann Whitney U Test)
  • Evaluate the results

Libraries and Packages :

• matplotlib
• seaborn
• pandas
• numpy
• scipy