From cole_haus:
GiveWell produces cost-effectiveness models of its top charities. These models take as inputs many uncertain parameters. Instead of representing those uncertain parameters with point estimates—as the cost-effectiveness analysis spreadsheet does—we can (should) represent them with probability distributions. Feeding probability distributions into the models allows us to output explicit probability distributions on the cost-effectiveness of each charity.
A common approach: Consider 'best' and 'worst' case scenarios for each parameter, and consider 'what if all goes best?' and 'what if all goes worst?', for lower and upper bounds.
But this approach is not optimal, because:
Either case ('all best' or 'all worst') is extremely unlikely, and more unlikely the more 'uncertain things there are'. (At least this is true if the random/uncertain things are independent; we can have correlated uncertainties too).
Consider: What are the chances of winning the lottery ten times in a row? What are the chances of getting hit by lightning ten times in a year?
The details of the uncertainty matter, and may matter to outcomes: Some 'uncertainties' are far more uncertain than others, or have more meaningfully 'long-tailed' distributions. Furthermore, if the uncertain events are correlated to one another, this leads to a lot more variance in the outcome.
There are reasonable ways of explicitly measuring and calibrating uncertainty over each parameter, and making this explicit is helpful
cole_haus again:
From a subjective Bayesian point of view, the best way to represent our state of knowledge on the input parameters is with a probability distribution over the values the parameter could take. For example, I could say that a negative value for increasing consumption seems very improbable to me but that a wide range of positive values seem about equally plausible. Once we specify a probability distribution, we can feed these distributions into the model and, in principle, we'll end up with a probability distribution over our results. This probability distribution on the results helps us understand the uncertainty contained in our estimates and how literally we should take them.
From a table for Malaria Consortium Seasonal malaria chemoprevention (rephrased)
| Input | Type of uncertainty | Meaning/importance |
|---|---|---|
| num LLINs distributed per person | Operational | Affects total cost for effect |
| deaths avert per protected child | Causal | How effective is core activity |
| lifespan of an LLIN | Empirical | Years of benefit per distribution |
| internal validity adjustment | Methodological | trust underlying studies? |
| pct mortal. AMF areas v trial | Empirical/historical | Affects size of problem |
| value averting child death | Moral | Det. how outcomes become value |