Awesome Uncertainty in Deep learning

This repo is a collection of awesome papers, codes, books, and blogs about Uncertainty and Deep learning.

⭐ Feel free to star and fork. ⭐

If you think we missed a paper, please open a pull request or send a message on the corresponding GitHub discussion. Tell us where the article was published and when, and send us GitHub and ArXiv links if they are available.

We are also open to any ideas for improvements!

Table of Contents

• Awesome Uncertainty in Deep learning
• Papers
  • Surveys
  • Theory
  • Bayesian-Methods
  • Ensemble-Methods
  • Sampling/Dropout-based-Methods
  • Post-hoc-Methods/Auxiliary-Networks
  • Data-augmentation/Generation-based-methods
  • Output-Space-Modeling/Evidential-deep-learning
  • Deterministic-Uncertainty-Methods
  • Quantile-Regression/Predicted-Intervals
  • Conformal Predictions
  • Calibration/Evaluation-Metrics
  • Misclassification Detection & Selective Classification
  • Applications
    • Classification and Semantic-Segmentation
    • Regression
    • Anomaly-detection and Out-of-Distribution-Detection
    • Object detection
    • Domain adaptation
    • Semi-supervised
    • Natural Language Processing
    • Others
• Datasets and Benchmarks
• Libraries
  • Python
  • PyTorch
  • JAX
  • TensorFlow
• Lectures and tutorials
• Books
• Other Resources

Papers

Surveys

Conference

• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications [AISafety Workshop 2020]

Journal

• A survey of uncertainty in deep neural networks [Artificial Intelligence Review 2023] - [GitHub]
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation [TMLR2023]
• A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective [ACM2021]
• Ensemble deep learning: A review [Engineering Applications of AI 2021]
• A review of uncertainty quantification in deep learning: Techniques, applications and challenges [Information Fusion 2021]
• Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods [Machine Learning 2021]
• Predictive inference with the jackknife+ [The Annals of Statistics 2021]
• Uncertainty in big data analytics: survey, opportunities, and challenges [Journal of Big Data 2019]

Arxiv

• Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks [ArXiv2024] - [PyTorch]
• A System-Level View on Out-of-Distribution Data in Robotics [arXiv2022]
• A Survey on Uncertainty Reasoning and Quantification for Decision Making: Belief Theory Meets Deep Learning [arXiv2022]

Theory

Conference

• A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods [NeurIPS2023]
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning [ICLR2023]
• Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask? [ICLR2023]
• Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs [ICML2023] - [PyTorch]
• On Second-Order Scoring Rules for Epistemic Uncertainty Quantification [ICML2023]
• Neural Variational Gradient Descent [AABI2022]
• Top-label calibration and multiclass-to-binary reductions [ICLR2022]
• Bayesian Model Selection, the Marginal Likelihood, and Generalization [ICML2022]
• With malice towards none: Assessing uncertainty via equalized coverage [AIES 2021]
• Uncertainty in Gradient Boosting via Ensembles [ICLR2021] - [PyTorch]
• Repulsive Deep Ensembles are Bayesian [NeurIPS2021] - [PyTorch]
• Bayesian Optimization with High-Dimensional Outputs [NeurIPS2021]
• Residual Pathway Priors for Soft Equivariance Constraints [NeurIPS2021]
• Dangers of Bayesian Model Averaging under Covariate Shift [NeurIPS2021] - [TensorFlow]
• A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR Workshop2021]
• Why Are Bootstrapped Deep Ensembles Not Better? [NeurIPS Workshop]
• Deep Convolutional Networks as shallow Gaussian Processes [ICLR2019]
• On the accuracy of influence functions for measuring group effects [NeurIPS2018]
• To Trust Or Not To Trust A Classifier [NeurIPS2018] - [Python]
• Understanding Measures of Uncertainty for Adversarial Example Detection [UAI2018]

Journal

• Martingale posterior distributions [Royal Statistical Society Series B]
• A Unified Theory of Diversity in Ensemble Learning [JMLR2023]
• Multivariate Uncertainty in Deep Learning [TNNLS2021]
• A General Framework for Uncertainty Estimation in Deep Learning [RAL2020]
• Adaptive nonparametric confidence sets [Ann. Statist. 2006]

Arxiv

• Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping [arXiv2022]
• Efficient Gaussian Neural Processes for Regression [arXiv2021]
• Dense Uncertainty Estimation [arXiv2021] - [PyTorch]
• A higher-order swiss army infinitesimal jackknife [arXiv2019]

Bayesian-Methods

Conference

• Training Bayesian Neural Networks with Sparse Subspace Variational Inference [ICLR2024]
• Variational Bayesian Last Layers [ICLR2024]
• A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors [ICLR2024]
• Gradient-based Uncertainty Attribution for Explainable Bayesian Deep Learning [CVPR2023]
• Robustness to corruption in pre-trained Bayesian neural networks [ICLR2023]
• Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift [NeurIPS2023] - [PyTorch]
• Transformers Can Do Bayesian Inference [ICLR2022] - [PyTorch]
• Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picture [NeurIPS2022]
• On Batch Normalisation for Approximate Bayesian Inference [AABI2021]
• Activation-level uncertainty in deep neural networks [ICLR2021]
• Laplace Redux – Effortless Bayesian Deep Learning [NeurIPS2021] - [PyTorch]
• On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks [UAI2021]
• Learnable uncertainty under Laplace approximations [UAI2021]
• Bayesian Neural Networks with Soft Evidence [ICML Workshop2021] - [PyTorch]
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation [ECCV2020] - [PyTorch]
• How Good is the Bayes Posterior in Deep Neural Networks Really? [ICML2020]
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors [ICML2020] - [TensorFlow]
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks [ICML2020] - [PyTorch]
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [NeurIPS2020]
• A Simple Baseline for Bayesian Uncertainty in Deep Learning [NeurIPS2019] - [PyTorch] - [TorchUncertainty]
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks [ICML2018] - [TensorFlow] - [TorchUncertainty]
• Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
• A Scalable Laplace Approximation for Neural Networks [ICLR2018] - [Theano]
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning [ICML2018]
• Weight Uncertainty in Neural Networks [ICML2015]

Journal

• Analytically Tractable Hidden-States Inference in Bayesian Neural Networks [JMLR2024]
• Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification [TPAMI2023] - [PyTorch]
• Bayesian modeling of uncertainty in low-level vision [IJCV1990]

Arxiv

• Density Uncertainty Layers for Reliable Uncertainty Estimation [arXiv2023]

Ensemble-Methods

Conference

• Input-gradient space particle inference for neural network ensembles [ICLR2024]
• Fast Ensembling with Diffusion Schrödinger Bridge [ICLR2024]
• Pathologies of Predictive Diversity in Deep Ensembles [ICLR2024]
• Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization [ICML2023]
• Bayesian Posterior Approximation With Stochastic Ensembles [CVPR2023]
• Normalizing Flow Ensembles for Rich Aleatoric and Epistemic Uncertainty Modeling [AAAI2023]
• Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models [ICCV2023] - [PyTorch]
• Weighted Ensemble Self-Supervised Learning [ICLR2023]
• Agree to Disagree: Diversity through Disagreement for Better Transferability [ICLR2023] - [PyTorch]
• Packed-Ensembles for Efficient Uncertainty Estimation [ICLR2023] - [TorchUncertainty]
• Sub-Ensembles for Fast Uncertainty Estimation in Neural Networks [ICCV Workshop2023]
• Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworks [AAAI2022]
• Deep Ensembles Work, But Are They Necessary? [NeurIPS2022]
• FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation [NeurIPS2022]
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity [ICLR2022] - [PyTorch]
• On the Usefulness of Deep Ensemble Diversity for Out-of-Distribution Detection [ECCV Workshop2022]
• Masksembles for Uncertainty Estimation [CVPR2021] - [PyTorch/TensorFlow]
• Robustness via Cross-Domain Ensembles [ICCV2021] - [PyTorch]
• Uncertainty in Gradient Boosting via Ensembles [ICLR2021] - [PyTorch]
• Uncertainty Quantification and Deep Ensembles [NeurIPS2021]
• Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles [AAAI2020]
• Uncertainty in Neural Networks: Approximately Bayesian Ensembling [AISTATS 2020]
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning [ICLR2020] - [PyTorch]
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning [ICLR2020] - [TensorFlow] - [TorchUncertainty]
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification [NeurIPS2020]
• Bayesian Deep Ensembles via the Neural Tangent Kernel [NeurIPS2020]
• Diversity with Cooperation: Ensemble Methods for Few-Shot Classification [ICCV2019]
• Accurate Uncertainty Estimation and Decomposition in Ensemble Learning [NeurIPS2019]
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach [ICML2018] - [TensorFlow]
• Snapshot Ensembles: Train 1, get M for free [ICLR2017] - [TorchUncertainty]
• Simple and scalable predictive uncertainty estimation using deep ensembles [NeurIPS2017] - [TorchUncertainty]

Journal

• One Versus all for deep Neural Network for uncertainty (OVNNI) quantification [IEEE Access2021]

Arxiv

• Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting [arXiv2023]
• Deep Ensemble as a Gaussian Process Approximate Posterior [arXiv2022]
• Sequential Bayesian Neural Subnetwork Ensembles [arXiv2022]
• Confident Neural Network Regression with Bootstrapped Deep Ensembles [arXiv2022] - [TensorFlow]
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [arXiv2021]
• Deep Ensembles: A Loss Landscape Perspective [arXiv2019]
• Checkpoint ensembles: Ensemble methods from a single training process [arXiv2017] - [TorchUncertainty]

Sampling/Dropout-based-Methods

Conference

• Enabling Uncertainty Estimation in Iterative Neural Networks [ICML2024] - [Pytorch]
• Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models [CVPR2024] - [TorchUncertainty]
• Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate [AAAI2022]
• Efficient Bayesian Uncertainty Estimation for nnU-Net [MICCAI2022]
• Dropout Sampling for Robust Object Detection in Open-Set Conditions [ICRA2018]
• Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks [MIDL2018]
• Concrete Dropout [NeurIPS2017]
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning [ICML2016] - [TorchUncertainty]

Journal

• A General Framework for Uncertainty Estimation in Deep Learning [Robotics and Automation Letters2020]

Arxiv

• SoftDropConnect (SDC) – Effective and Efficient Quantification of the Network Uncertainty in Deep MR Image Analysis [arXiv2022]

Post-hoc-Methods/Auxiliary-Networks

Conference

• On the Limitations of Temperature Scaling for Distributions with Overlaps [ICLR2024]
• Post-hoc Uncertainty Learning using a Dirichlet Meta-Model [AAAI2023] - [PyTorch]
• ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models [ICCV2023]
• Out-of-Distribution Detection for Monocular Depth Estimation [ICCV2023]
• Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model [AAAI2022]
• Learning Structured Gaussians to Approximate Deep Ensembles [CVPR2022]
• Improving the reliability for confidence estimation [ECCV2022]
• Gradient-based Uncertainty for Monocular Depth Estimation [ECCV2022] - [PyTorch]
• BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks [ECCV2022] - [PyTorch]
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [ICIP2022]
• SLURP: Side Learning Uncertainty for Regression Problems [BMVC2021] - [PyTorch]
• Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [ICCV2021] - [PyTorch]
• Learning to Predict Error for MRI Reconstruction [MICCAI2021]
• A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR Workshop2021]
• Real-time uncertainty estimation in computer vision via uncertainty-aware distribution distillation [WACV2021]
• On the uncertainty of self-supervised monocular depth estimation [CVPR2020] - [PyTorch]
• Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel [ICLR2020] - [TensorFlow]
• Gradients as a Measure of Uncertainty in Neural Networks [ICIP2020]
• Learning Loss for Test-Time Augmentation [NeurIPS2020]
• Learning loss for active learning [CVPR2019] - [PyTorch] (unofficial codes)
• Addressing failure prediction by learning model confidence [NeurIPS2019] - [PyTorch]
• Structured Uncertainty Prediction Networks [CVPR2018] - [TensorFlow]
• Classification uncertainty of deep neural networks based on gradient information [IAPR Workshop2018]

Journal

• Towards More Reliable Confidence Estimation [TPAMI2023]
• Confidence Estimation via Auxiliary Models [TPAMI2021]

Arxiv

• Instance-Aware Observer Network for Out-of-Distribution Object Segmentation [arXiv2022]
• DEUP: Direct Epistemic Uncertainty Prediction [arXiv2020]
• Learning Confidence for Out-of-Distribution Detection in Neural Networks [arXiv2018]

Data-augmentation/Generation-based-methods

Conference

• Posterior Uncertainty Quantification in Neural Networks using Data Augmentation [AISTATS2024]
• Learning to Generate Training Datasets for Robust Semantic Segmentation [WACV2024]
• OpenMix: Exploring Outlier Samples for Misclassification Detection [CVPR2023] - [PyTorch]
• On the Pitfall of Mixup for Uncertainty Calibration [CVPR2023]
• Diverse, Global and Amortised Counterfactual Explanations for Uncertainty Estimates [AAAI2022]
• Out-of-distribution Detection with Implicit Outlier Transformation [ICLR2023] - [PyTorch]
• PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures [CVPR2022]
• RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy & Out-of-Distribution Robustness [NeurIPS2022] - [PyTorch]
• Towards efficient feature sharing in MIMO architectures [CVPR Workshop2022]
• Robust Semantic Segmentation with Superpixel-Mix [BMVC2021] - [PyTorch]
• MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks [ICCV2021] - [PyTorch]
• Training independent subnetworks for robust prediction [ICLR2021]
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [IJCAI2021] - [PyTorch]
• Uncertainty-aware GAN with Adaptive Loss for Robust MRI Image Enhancement [ICCV Workshop2021]
• Uncertainty-Aware Deep Classifiers using Generative Models [AAAI2020]
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation [ECCV2020] - [PyTorch]
• Detecting the Unexpected via Image Resynthesis [ICCV2019] - [PyTorch]
• Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning [ICML2020]
• Deep Anomaly Detection with Outlier Exposure [ICLR2019]
• On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks [NeurIPS2019]

Arxiv

• Reliability in Semantic Segmentation: Can We Use Synthetic Data? [arXiv2023]
• Quantifying uncertainty with GAN-based priors [arXiv2019] - [TensorFlow]

Output-Space-Modeling/Evidential-deep-learning

Conference

• Hyper Evidential Deep Learning to Quantify Composite Classification Uncertainty [ICLR2024]
• The Evidence Contraction Issue in Deep Evidential Regression: Discussion and Solution [AAAI2024]
• Discretization-Induced Dirichlet Posterior for Robust Uncertainty Quantification on Regression [AAAI2024] - [PyTorch]
• The Unreasonable Effectiveness of Deep Evidential Regression [AAAI2023] - [PyTorch] - [TorchUncertainty]
• Exploring and Exploiting Uncertainty for Incomplete Multi-View Classification [CVPR2023]
• Plausible Uncertainties for Human Pose Regression [ICCV2023] - [PyTorch]
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [ICML2023] - [PyTorch]
• Improving Evidential Deep Learning via Multi-task Learning [AAAI2022] - [PyTorch]
• An Evidential Neural Network Model for Regression Based on Random Fuzzy Numbers [BELIEF2022]
• On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks [ICLR2022] - [PyTorch]
• Natural Posterior Network: Deep Bayesian Uncertainty for Exponential Family Distributions [ICLR2022] - [PyTorch]
• Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation [NeurIPS2022]
• Fast Predictive Uncertainty for Classification with Bayesian Deep Networks [UAI2022] - [PyTorch]
• Evaluating robustness of predictive uncertainty estimation: Are Dirichlet-based models reliable? [ICML2021]
• Trustworthy multimodal regression with mixture of normal-inverse gamma distributions [NeurIPS2021]
• Misclassification Risk and Uncertainty Quantification in Deep Classifiers [WACV2021]
• Ensemble Distribution Distillation [ICLR2020]
• Conservative Uncertainty Estimation By Fitting Prior Networks [ICLR2020]
• Being Bayesian about Categorical Probability [ICML2020] - [PyTorch]
• Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts [NeurIPS2020] - [PyTorch]
• Deep Evidential Regression [NeurIPS2020] - [TensorFlow] - [TorchUncertainty]
• Noise Contrastive Priors for Functional Uncertainty [UAI2020]
• Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples [NeurIPS Workshop2020]
• Uncertainty on Asynchronous Time Event Prediction [NeurIPS2019] - [TensorFlow]
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness [NeurIPS2019]
• Quantifying Classification Uncertainty using Regularized Evidential Neural Networks [AAAI FSS2019]
• Uncertainty estimates and multi-hypotheses networks for optical flow [ECCV2018] - [TensorFlow]
• Evidential Deep Learning to Quantify Classification Uncertainty [NeurIPS2018] - [PyTorch]
• Predictive uncertainty estimation via prior networks [NeurIPS2018]
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? [NeurIPS2017]
• Estimating the Mean and Variance of the Target Probability Distribution [(ICNN1994)]

Journal

• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation [TMLR2023]
• Region-Based Evidential Deep Learning to Quantify Uncertainty and Improve Robustness of Brain Tumor Segmentation [NCA2022]
• An evidential classifier based on Dempster-Shafer theory and deep learning [Neurocomputing2021] - [TensorFlow]
• Evidential fully convolutional network for semantic segmentation [AppliedIntelligence2021] - [TensorFlow]
• Information Aware max-norm Dirichlet networks for predictive uncertainty estimation [NeuralNetworks2021]
• A neural network classifier based on Dempster-Shafer theory [IEEETransSMC2000]

Arxiv

• Evidential Uncertainty Quantification: A Variance-Based Perspective [arXiv2023]
• Effective Uncertainty Estimation with Evidential Models for Open-World Recognition [arXiv2022]
• Multivariate Deep Evidential Regression [arXiv2022]
• Regression Prior Networks [arXiv2020]
• A Variational Dirichlet Framework for Out-of-Distribution Detection [arXiv2019]
• Uncertainty estimation in deep learning with application to spoken language assessment [PhDThesis2019]
• Inhibited softmax for uncertainty estimation in neural networks [arXiv2018]
• Quantifying Intrinsic Uncertainty in Classification via Deep Dirichlet Mixture Networks [arXiv2018]

Deterministic-Uncertainty-Methods

Conference

• A Rate-Distortion View of Uncertainty Quantification [ICML2024] - [Tensorflow]
• Deep Deterministic Uncertainty: A Simple Baseline [CVPR2023] - [PyTorch]
• Gaussian Latent Representations for Uncertainty Estimation using Mahalanobis Distance in Deep Classifiers [ICCV Workshop2023] - [PyTorch]
• A Simple and Explainable Method for Uncertainty Estimation using Attribute Prototype Networks [ICCV Workshop2023]
• Training, Architecture, and Prior for Deterministic Uncertainty Methods [ICLR Workshop2023] - [PyTorch]
• Latent Discriminant deterministic Uncertainty [ECCV2022] - [PyTorch]
• On the Practicality of Deterministic Epistemic Uncertainty [ICML2022]
• Improving Deterministic Uncertainty Estimation in Deep Learning for Classification and Regression [CoRR2021]
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network [ICML2020] - [PyTorch]
• Training normalizing flows with the information bottleneck for competitive generative classification [NeurIPS2020]
• Simple and principled uncertainty estimation with deterministic deep learning via distance awareness [NeurIPS2020]
• Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks [ICML Workshop2020]
• Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance Propagation [ICCV2019] - [PyTorch]
• Single-Model Uncertainties for Deep Learning [NeurIPS2019] - [PyTorch]

Journal

• ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference [TMLR2024] - [Pytorch]
• Density estimation in representation space [EDSMLS2020]

Arxiv

• The Hidden Uncertainty in a Neural Network’s Activations [arXiv2020]
• A simple framework for uncertainty in contrastive learning [arXiv2020]
• Distance-based Confidence Score for Neural Network Classifiers [arXiv2017]

Quantile-Regression/Predicted-Intervals

Conference

• Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging [ICML2022] - [PyTorch]
• Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles [UAI2020] - [PyTorch]
• Classification with Valid and Adaptive Coverage [NeurIPS2020]
• Single-Model Uncertainties for Deep Learning [NeurIPS2019] - [PyTorch]
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach [ICML2018] - [TensorFlow]

Journal

• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors [CMAME2022]
• Exploring uncertainty in regression neural networks for construction of prediction intervals [Neurocomputing2022]

Arxiv

• Interval Neural Networks: Uncertainty Scores [arXiv2020]
• Tight Prediction Intervals Using Expanded Interval Minimization [arXiv2018]

Conformal Predictions

Awesome Conformal Prediction [GitHub]

Calibration/Evaluation-Metrics

Conference

• Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing [ICLR2024]
• Calibrating Transformers via Sparse Gaussian Processes [ICLR2023] - [PyTorch]
• Beyond calibration: estimating the grouping loss of modern neural networks [ICLR2023] - [Python]
• Dual Focal Loss for Calibration [ICML 2023]
• What Are Effective Labels for Augmented Data? Improving Calibration and Robustness with AutoLabel [SaTML2023]
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration [CVPR2022] - [PyTorch]
• AdaFocal: Calibration-aware Adaptive Focal Loss [NeurIPS2022]
• Calibrating Deep Neural Networks by Pairwise Constraints [CVPR2022]
• Top-label calibration and multiclass-to-binary reductions [ICLR2022]
• From label smoothing to label relaxation [AAAI2021]
• Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain [AIStats2021]
• Rethinking Calibration of Deep Neural Networks: Do Not Be Afraid of Overconfidence [NeurIPS2021]
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification [NeurIPS2021]
• Soft Calibration Objectives for Neural Networks [NeurIPS2021] - [TensorFlow]
• Confidence-Aware Learning for Deep Neural Networks [ICML2020] - [PyTorch]
• Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning [ICML2020]
• Regularization via structural label smoothing [ICML2020]
• Well-Calibrated Regression Uncertainty in Medical Imaging with Deep Learning [MIDL2020] - [PyTorch]
• Calibrating Deep Neural Networks using Focal Loss [NeurIPS2020] - [PyTorch]
• Stationary activations for uncertainty calibration in deep learning [NeurIPS2020]
• Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off [CVPR Workshop2020]
• Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision [CVPR Workshop2020] - [PyTorch]
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers [ICLR2019]
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration [NeurIPS2019] - [GitHub]
• When does label smoothing help? [NeurIPS2019]
• Verified Uncertainty Calibration [NeurIPS2019] - [GitHub]
• Measuring Calibration in Deep Learning [CVPR Workshop2019]
• Accurate Uncertainties for Deep Learning Using Calibrated Regression [ICML2018]
• Generalized zero-shot learning with deep calibration network [NeurIPS2018]
• On calibration of modern neural networks [ICML2017] - [TorchUncertainty]
• On Fairness and Calibration [NeurIPS2017]
• Obtaining Well Calibrated Probabilities Using Bayesian Binning [AAAI2015]

Journal

• Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error [TMLR2023] - [PyTorch]
• Evaluating and Calibrating Uncertainty Prediction in Regression Tasks [Sensors2022]
• Calibrated Prediction Intervals for Neural Network Regressors [IEEE Access 2018] - [Python]

Arxiv

• Towards Understanding Label Smoothing [arXiv2020]
• An Investigation of how Label Smoothing Affects Generalization [arXiv2020]

Misclassification Detection & Selective Classification

• A Data-Driven Measure of Relative Uncertainty for Misclassification Detection [ICLR2024]
• Plugin estimators for selective classification with out-of-distribution detection [ICLR2024]
• SURE: SUrvey REcipes for building reliable and robust deep networks [CVPR2024] - [PyTorch]
• Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data [ACCV2022]
• Anomaly Detection via Reverse Distillation from One-Class Embedding [CVPR2022]
• Rethinking Confidence Calibration for Failure Prediction [ECCV2022] - [PyTorch]

Uncertainty sources & Aleatoric and Epistemic Uncertainty Disentenglement

ArXiv

• Sources of Uncertainty in Machine Learning - A Statisticians’ View [ArXiv2024]
• How disentangled are your classification uncertainties? [ArXiv2024
• Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks [ArXiv2024 - [PyTorch]

Applications

Classification and Semantic-Segmentation

Conference

• Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts [ICLR2023] - [PyTorch]
• Anytime Dense Prediction with Confidence Adaptivity [ICLR2022] - [PyTorch]
• CRISP - Reliable Uncertainty Estimation for Medical Image Segmentation [MICCAI2022]
• TBraTS: Trusted Brain Tumor Segmentation [MICCAI2022] - [PyTorch]
• Robust Semantic Segmentation with Superpixel-Mix [BMVC2021] - [PyTorch]
• Deep Deterministic Uncertainty for Semantic Segmentation [ICMLW2021]
• DEAL: Difficulty-aware Active Learning for Semantic Segmentation [ACCV2020]
• Classification with Valid and Adaptive Coverage [NeurIPS2020]
• Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation [ICCV2019]
• Human Uncertainty Makes Classification More Robust [ICCV2019]
• Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation [MICCAI2019] - [PyTorch]
• Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
• A Probabilistic U-Net for Segmentation of Ambiguous Images [NeurIPS2018] - [PyTorch]
• Evidential Deep Learning to Quantify Classification Uncertainty [NeurIPS2018] - [PyTorch]
• To Trust Or Not To Trust A Classifier [NeurIPS2018]
• Classification uncertainty of deep neural networks based on gradient information [IAPR Workshop2018]
• Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding [BMVC2017]

Journal

• Explainable machine learning in image classification models: An uncertainty quantification perspective." [KnowledgeBased2022]
• Region-Based Evidential Deep Learning to Quantify Uncertainty and Improve Robustness of Brain Tumor Segmentation [NCA2022]

Arxiv

• Leveraging Uncertainty Estimates to Improve Classifier Performance [arXiv2023]
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation [arXiv2018]

Regression

Conference

• Learning the Distribution of Errors in Stereo Matching for Joint Disparity and Uncertainty Estimation [CVPR2023] - [PyTorch]
• Probabilistic MIMO U-Net: Efficient and Accurate Uncertainty Estimation for Pixel-wise Regression [ICCV Workshop2023] - [PyTorch]
• Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate [AAAI2022]
• Learning Structured Gaussians to Approximate Deep Ensembles [CVPR2022]
• Uncertainty Quantification in Depth Estimation via Constrained Ordinal Regression [ECCV2022]
• On Monocular Depth Estimation and Uncertainty Quantification using Classification Approaches for Regression [ICIP2022]
• Anytime Dense Prediction with Confidence Adaptivity [ICLR2022] - [PyTorch]
• Variational Depth Networks: Uncertainty-Aware Monocular Self-supervised Depth Estimation [ECCV Workshop2022]
• SLURP: Side Learning Uncertainty for Regression Problems [BMVC2021] - [PyTorch]
• Robustness via Cross-Domain Ensembles [ICCV2021] - [PyTorch]
• Learning to Predict Error for MRI Reconstruction [MICCAI2021]
• On the uncertainty of self-supervised monocular depth estimation [CVPR2020] - [PyTorch]
• Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel [ICLR2020] - [TensorFlow]
• Fast Uncertainty Estimation for Deep Learning Based Optical Flow [IROS2020]
• Well-Calibrated Regression Uncertainty in Medical Imaging with Deep Learning [MIDL2020] - [PyTorch]
• Deep Evidential Regression [NeurIPS2020] - [TensorFlow]
• Inferring Distributions Over Depth from a Single Image [IROS2019] - [TensorFlow]
• Multi-Task Learning based on Separable Formulation of Depth Estimation and its Uncertainty [CVPR Workshop2019]
• Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
• Structured Uncertainty Prediction Networks [CVPR2018] - [TensorFlow]
• Uncertainty estimates and multi-hypotheses networks for optical flow [ECCV2018] - [TensorFlow]
• Accurate Uncertainties for Deep Learning Using Calibrated Regression [ICML2018]

Journal

• How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts? [TMLR2023] - [PyTorch]
• Evaluating and Calibrating Uncertainty Prediction in Regression Tasks [Sensors2022]
• Exploring uncertainty in regression neural networks for construction of prediction intervals [Neurocomputing2022]
• Wasserstein Dropout [Machine Learning 2022] - [PyTorch]
• Deep Distribution Regression [Computational Statistics & Data Analysis2021]
• Calibrated Prediction Intervals for Neural Network Regressors [IEEE Access 2018] - [Python]
• Learning a Confidence Measure for Optical Flow [TPAMI2013]

Arxiv

• Understanding pathologies of deep heteroskedastic regression [arxiv2024]
• Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation [arXiv2023]
• UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographaphy [arXiv2022]
• Efficient Gaussian Neural Processes for Regression [arXiv2021]

Anomaly-detection and Out-of-Distribution-Detection

Conference

• Learning Transferable Negative Prompts for Out-of-Distribution Detection [CVPR2024] - [PyTorch]
• Epistemic Uncertainty Quantification For Pre-trained Neural Networks [CVPR2024]
• NECO: NEural Collapse Based Out-of-distribution Detection [ICLR2024]
• When and How Does In-Distribution Label Help Out-of-Distribution Detection? [ICML2024] - [PyTorch]
• Anomaly Detection under Distribution Shift [ICCV2023] - [PyTorch]
• Normalizing Flows for Human Pose Anomaly Detection [ICCV2023] - [PyTorch]
• RbA: Segmenting Unknown Regions Rejected by All [ICCV2023] - [PyTorch]
• Uncertainty-Aware Optimal Transport for Semantically Coherent Out-of-Distribution Detection [CVPR2023] - [PyTorch]
• Modeling the Distributional Uncertainty for Salient Object Detection Models [CVPR2023] - [PyTorch]
• SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection [CVPR2023] - [PyTorch]
• How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection? [ICLR2023] - [PyTorch]
• Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization [ICLR2023]
• Can CNNs Be More Robust Than Transformers? [ICLR2023]
• A framework for benchmarking class-out-of-distribution detection and its application to ImageNet [ICLR2023]
• Extremely Simple Activation Shaping for Out-of-Distribution Detection [ICLR2023] - [PyTorch]
• Quantification of Uncertainty with Adversarial Models [NeurIPS2023]
• The Robust Semantic Segmentation UNCV2023 Challenge Results [ICCV Workshop2023]
• Continual Evidential Deep Learning for Out-of-Distribution Detection [ICCV Workshop2023]
• Far Away in the Deep Space: Nearest-Neighbor-Based Dense Out-of-Distribution Detection [ICCV Workshop2023]
• Gaussian Latent Representations for Uncertainty Estimation using Mahalanobis Distance in Deep Classifiers [ICCV Workshop2023]
• Calibrated Out-of-Distribution Detection with a Generic Representation [ICCV Workshop2023] - [PyTorch]
• Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model [AAAI2022]
• Towards Total Recall in Industrial Anomaly Detection [CVPR2022] - [PyTorch]
• POEM: Out-of-Distribution Detection with Posterior Sampling [ICML2022] - [PyTorch]
• VOS: Learning What You Don't Know by Virtual Outlier Synthesis [ICLR2022] - [PyTorch]
• Fully Convolutional Cross-Scale-Flows for Image-based Defect Detection [WACV2022] - [PyTorch]
• Out-of-Distribution Detection Using Union of 1-Dimensional Subspaces [CVPR2021] - [PyTorch]
• NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization [ICCV2021]
• On the Importance of Gradients for Detecting Distributional Shifts in the Wild [NeurIPS2021]
• Exploring the Limits of Out-of-Distribution Detection [NeurIPS2021]
• Detecting out-of-distribution image without learning from out-of-distribution data. [CVPR2020]
• Learning Open Set Network with Discriminative Reciprocal Points [ECCV2020]
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation [ECCV2020] - [PyTorch]
• NADS: Neural Architecture Distribution Search for Uncertainty Awareness [ICML2020]
• PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization [ICPR2020] - [PyTorch]
• Energy-based Out-of-distribution Detection [NeurIPS2020]
• Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples [NeurIPS Workshop2020]
• Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection [ICCV2019] - [PyTorch]
• Detecting the Unexpected via Image Resynthesis [ICCV2019] - [PyTorch]
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks [ICLR2018]
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks [ICLR2017] - [TensorFlow]

Journal

• Revisiting Confidence Estimation: Towards Reliable Failure Prediction [TPAMI2024] - [PyTorch]
• One Versus all for deep Neural Network for uncertaInty (OVNNI) quantification [IEEE Access2021]

Arxiv

• Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization [arXiv2023] - [PyTorch]
• A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection [arXiv2021]
• Generalized out-of-distribution detection: A survey [arXiv2021]
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection? [arXiv2021]
• Frequentist uncertainty estimates for deep learning [arXiv2018]

Object detection

Conference

• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [CVPR2023] - [PyTorch]
• Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection [ECCV Workshop2022] - [PyTorch]
• Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors [ICLR2021] - [PyTorch]
• Multivariate Confidence Calibration for Object Detection [CVPR Workshop2020] - [PyTorch]
• Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving [ICCV2019] - [CUDA] - [PyTorch] - [Keras]

Domain adaptation

Conference

• Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation [CVPR2023] - [PyTorch]
• Uncertainty-guided Source-free Domain Adaptation [ECCV2022] - [PyTorch]

Semi-supervised

Conference

• Confidence Estimation Using Unlabeled Data [ICLR2023] - [PyTorch]

Natural Language Processing

Awesome LLM Uncertainty, Reliability, & Robustness [GitHub]

Conference

• R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents [ICML2023] - [GitHub]
• Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement [TrustNLP2023] - [GitHub]
• Disentangling Uncertainty in Machine Translation Evaluation [EMNLP2022] - [PyTorch]
• Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers [EMNLP2022 Findings]
• DATE: Detecting Anomalies in Text via Self-Supervision of Transformers [NAACL2021]
• Calibrating Structured Output Predictors for Natural Language Processing [ACL2020]
• Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data [EMNLP2020] - [PyTorch]

Journal

• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering [TACL2021] - [PyTorch]

Arxiv

• Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models [arXiv2024]
• To Believe or Not to Believe Your LLM [arXiv2024]
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [arXiv2023]

Others

Conference

• PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness [CVPR2024] - [Website]
• Uncertainty Quantification via Stable Distribution Propagation [ICLR2024]
• Assessing Uncertainty in Similarity Scoring: Performance & Fairness in Face Recognition [ICLR2024]

Arxiv

• Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood - [arxiv2024]
• Urban 3D Panoptic Scene Completion with Uncertainty Awareness [arXiv2023] - [PyTorch]

Datasets and Benchmarks

• SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation [CVPR2022]
• MUAD: Multiple Uncertainties for Autonomous Driving, a benchmark for multiple uncertainty types and tasks [BMVC2022] - [PyTorch]
• ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding [ICCV2021]
• The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection [IJCV2021]
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation [NeurIPS2021]
• Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning [arXiv2021] - [TensorFlow]
• Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding [IJCV2020]
• Benchmarking the Robustness of Semantic Segmentation Models [CVPR2020]
• Fishyscapes: A Benchmark for Safe Semantic Segmentation in Autonomous Driving [ICCV Workshop2019]
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming [NeurIPS Workshop2019] - [GitHub]
• Semantic Foggy Scene Understanding with Synthetic Data [IJCV2018]
• Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles [IROS2016]

Libraries

Python

• Uncertainty Calibration Library [GitHub]
• MAPIE: Model Agnostic Prediction Interval Estimator [Sklearn]
• Uncertainty Toolbox [GitHub]
• OpenOOD: Benchmarking Generalized OOD Detection [GitHub]
• Darts: Forecasting and anomaly detection on time series [GitHub]
• Mixture Density Networks (MDN) for distribution and uncertainty estimation [GitHub]

PyTorch

• TorchUncertainty [GitHub]
• Bayesian Torch [GitHub]
• Blitz: A Bayesian Neural Network library for PyTorch [GitHub]

JAX

• Fortuna [GitHub - JAX]

TensorFlow

• TensorFlow Probability [Website]

Lectures and tutorials

• Dan Hendrycks: Intro to ML Safety course [Website]
• Uncertainty and Robustness in Deep Learning Workshop in ICML (2020, 2021) [SlidesLive]
• Yarin Gal: Bayesian Deep Learning 101 [Website]
• MIT 6.S191: Evidential Deep Learning and Uncertainty (2021) [Youtube]
• Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users [IEEE Computational Intelligence Magazine]

Books

• The "Probabilistic Machine-Learning" book series by Kevin Murphy [Book]

Other Resources

Uncertainty Quantification in Deep Learning [GitHub]

Awesome Out-of-distribution Detection [GitHub]

Anomaly Detection Learning Resources [GitHub]

Awesome Conformal Prediction [GitHub]

Awesome LLM Uncertainty, Reliability, & Robustness [GitHub]

UQSay - Seminars on Uncertainty Quantification (UQ), Design and Analysis of Computer Experiments (DACE) and related topics @ Paris Saclay [Website]

ProbAI summer school [Website]

Gaussian process summer school [Website]