V2 - Removing bootstrapped confidence intervals in lieu of confidence intervals based on the standard error of mean predictions

[oliver314]

Hello!

What is the reasoning for removing the bootstrapped confidence intervals apart the performance gains?

The standard error of mean predictions gives confidence intervals which are indeed much more conservative. How did you assess them to be more realistic?

https://github.com/BCG-Gamma/facet/blob/2.0.x/RELEASE_NOTES.rst#facetsimulation

Best regards and many thanks!

A practical note that might be helpful for people in the future:
UnivariateProbabilitySimulator.simulate_feature evaluates what partitions to use - but there is no way to pass arguments such as lower_bound or upper_bound, or to use an already fitted partitioner - so one needs to subclass to implement that functionality.

[j-ittner]

Hi @oliver314 !

First of all - thanks for taking the release candidate of FACET 2 for a spin, and good catch on the partitioner arguments - we're adding an option to specify partitioner parameters like lower_bound when calling UnivariateProbabilitySimulator.simulate_feature. See PR #355. Thanks for the helpful feedback!

On the bootstrapping: the main change in FACET 2 is that we run the simulations off of a single model, fitted on the entire dataset (FACET 1 simulated across multiple models fitted on different bootstrap samples). The confidence intervals tend to be less conservative (= narrower) as a consequence of that change. Given the sampling distribution of the mean can be considered normal for any but very small samples (n < 30), using the standard error of the mean will give us the same confidence intervals as bootstrapping would, but more efficiently.