Tutorial on model assesment, selection and inference after selection

Example notebooks in R using rstanarm, rstan, bayesplot, loo, projpred.

Videos

• Model selection tutorial at StanCon 2018 Asilomar
• Regularized horseshoe talk at StanCon 2018 Asilomar (regularized horseshoe video and slides are included as RHS is a good prior when we assume that some of the covariates are irrelevant)

Slides

• Model selection tutorial slides
• Regularized horseshoe slides

Outline of the tutorial and links to notebooks

• Basics of predictive performance estimation
• When cross-validation is not needed
  • Simple model we trust - betablockers
• When cross-validation is useful
  • We don't trust the model - roaches
  • Complex model with posterior dependencies - collinear
• On accuracy of cross-validation
• Cross-validation and hierarchical models
• When cross-validation is not enough
  • large number of models - diabetes
• loo 2.0 (coming soon)
• Projection predictive model selection
  • collinear
  • diabetes
  • projpred

Additional demos added after the tutorial

• projpred examples
  • collinearity - mesquite
  • random data vs original data - candy
  • winequality-red
  • bodyfat
• See also projpred quick start vignette

See also

• Nice horseshoe example in Bayes Sparse Regression case study by Michael Betancourt

References

• Heinze G1, Wallisch C1, Dunkler D: Variable selection - A review and recommendations for the practicing statistician. Biom J. 2018 Jan 2. doi: 10.1002/bimj.201700067. Online
• Gelman, A., Hwang, J., and Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):997–1016. Preprint
• Piironen, J. and Vehtari, A. (2016), Comparison of Bayesian predictive methods for model selection, Statistics and Computing 27(3), 711–735. Online
• Piironen, J., and Vehtari, A. (2017). On the hyperprior choice for the global shrinkage parameter in the horseshoe prior. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:905-913. Online
• Piironen, J., and Vehtari, A. (2017). Sparsity information and regularization in the horseshoe and other shrinkage priors. In Electronic Journal of Statistics, 11(2):5018-5051. Online
• Piironen, J., and Vehtari, A. (2018). Iterative supervised principal components. Proceedings of the 21th International Conference on Artificial Intelligence and Statistics, accepted for publication. arXiv preprint arXiv:1710.06229
• Vehtari, A., Gelman, A., Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5):1413–1432. arXiv preprint.
• Vehtari, A., Gelman, A., Gabry, J. (2017). Pareto smoothed importance sampling. arXiv preprint.
• Vehtari, A., Mononen, T., Tolvanen, V., and Winther, O. (2016). Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models. JMLR, 17(103):1–38. Online
• Vehtari, A. and Ojanen, J.: 2012, A survey of Bayesian predictive methods for model assessment, selection and comparison, Statistics Surveys 6, 142–228. Online
• Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2017). Using stacking to average Bayesian predictive distributions. In Bayesian Analysis, doi:10.1214/17-BA1091, Online