Add a page to explain there are two types of interactions

[lukasvermeer]

Interactions between two experiments can be of one of two kinds: traffic or metric interactions.

Traffic interactions
Traffic interactions occur when one experiment treatment is causing more or less traffic to flow to another experiment.

For example, when variation 1 of experiment 1 changes the destination of a link on the homepage from the one page to another, while experiment 2 is making changes on one of those pages. In that case, the first experiment will cause more or less traffic to flow into the second experiment.

Traffic interactions are not necessarily a problem. The traffic in experiment 2 in the example above can still be equally split and the metrics can still be tracked. There is however a risk of bias, since the traffic diverted by experiment 2 might be different from the remaining traffic in some meaningful way.

Metric interactions
Metric interactions occur when the impact on a given metric for a combination of two experiments is different from what we see in either experiment in isolation. For any two experiments, there might be differences in interaction effects for different metrics.

For example, when experiment 1 shows an uplift of 5% on conversion and experiment 2 shows an uplift of 10% on conversion, then we would expect the combination to show an uplift of approximately (1,1*1,05)-1 = 15,5%. If instead we see a drop of 50% for the audience which is exposed to both tests, then we would call that a (in this case negative) interaction. At the same time, for the same experiments, we might not see any such interaction effect on CTR or other metrics.

Similar to traffic interactions, metric interactions are not necessarily a problem depending on the nature and severity of the interaction.