To evaluate the heterogeneity of a treatment or nudge, one can attempt to predict the conditional average treatment effect (CATE) using personal characteristics. For example, Künzel et al 2019 introduced metalearners, that are build on base algorithms such as random forests, Bayesian additive regression trees, or neural network to estimate the CATE. Metalearners decompose estimating the CATE into several subregression problems that can be solved with any regression or supervised learning method.
We implemented three different kinds of metalearners: S-learners, T-learners and X-learners, as described in [Künzel et al 2019]](https://doi.org/10.1073/pnas.1804597116). S-learners simply apply the regression model directly to the dataset. T-learners apply a regression model to both the nudging and control groups. X-learners are similar to T-learners, but add another model that crosses over the results of the nuding and control groups.
In addition, we propose a new, alternative method for estimating the heterogeneity effect by deriving the probability that a treatment or nudge is successfull given the personal characteristics. We do this by first matching indviduals from the treatment group to individuals in the control group. This matching is done by propensity score. The propensity score is the probability of treatment assignment, given observed baseline characteristics. The propensity score can be used to balance the treatment and control groups to make them comparable. Rosenbaum and Rubin (1983) showed that treated and untreated subjects with the same propensity scores have identical distributions for all baseline variables. In our case, the treatment group is the group that received a nudge and the control is the group that didn't.
We calculate the propensity score by logistic regression. This is a statistical model used to predict the probability that an event occurs. In logistic regression, the dependent variable is binary; in our case, we have Z=1 for the treated subjects and Z=0 for the untreated subjects. We can then derive the logistic regression model and subsequently use it to calculate the propensity score for each subject. Propensity score matching is done by nearest neighbour matching of treated and untreated subjects, so that matched subjects have similar values of the propensity score.
When we have matched subjects, we simply determine the nudge succes by evaluating whether the outcome variable had increased or decreased, depending on the aim of the nudge. Thus nudge success is a binary, 0 for failure or 1 for success, This allows us later to combine results for different studies without having to apply normalization. Using a probablistic model, we can then derive the nudge effectiveness as function of the personal characteristics. Nudge effectiveness is defined here as the probability of nudge success.
We implemented two different kinds of probablistic models: logistic regression and naive bayes. Logistic regression is a discriminitive model, meaning it learns the posterior probability directly from the tranining data. Naive Bayes is a generative model, meaning it learns the joint probability distribution and uses Bayes' Theorem to predicts the posterior probability. Typically, naive Bayes converges quicker but has a higher error than logistic regression, see Ng and Jordan 2001. Thus, while on small datasets naive Bayes may be preferable, logistic regression is likely to achieve better results as the training set size grows.
To summarize, we distinguish between probabilistic models that attempt to predict nudge effectiveness and metalearners that predict the CATE.
All models are based on scikit-learn.