[CVPR 2023] Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection

Paper (arXiv) | Poster | Video | Paper (CVPR)

Authors: Muhammad Akhtar Munir, Muhammad Haris Khan, Salman Khan, Fahad Shahbaz Khan

This paper is accepted at CVPR 2023 and this repository contains the PyTorch implementation of our proposed Bridging Precision and Confidence (BPC) loss with D-DETR.

Abstract

Deep neural networks (DNNs) have enabled astounding progress in several vision-based problems. Despite showing high predictive accuracy, recently, several works have revealed that they tend to provide overconfident predictions and thus are poorly calibrated. The majority of the works addressing the miscalibration of DNNs fall under the scope of classification and consider only in-domain predictions. However, there is little to no progress in studying the calibration of DNN-based object detection models, which are central to many vision-based safety-critical applications. In this paper, inspired by the train-time calibration methods, we propose a novel auxiliary loss formulation that explicitly aims to align the class confidence of bounding boxes with the accurateness of predictions (i.e. precision). Since the original formulation of our loss depends on the counts of true positives and false positives in a minibatch, we develop a differentiable proxy of our loss that can be used during training with other application-specific loss functions.

Reliability Diagrams: Selected classes from MS-COCO (In-Domain) and CorCOCO (Out-Domain). Top: Baseline trained as D-DETR and Bottom: D-DETR trained with our proposed BPC loss.

Results

BPC calibration loss comparison with baseline, post-hoc and other train-time calibration losses

Results report Detection Expected Calibration Error (D-ECE) for In-Domain (MS-COCO) and Out-Domain (CorCOCO).

Methods	D-ECE (MS-COCO)	APbox (MS-COCO)	D-ECE (CorCOCO)	APbox (CorCOCO)	model
Baseline	12.8	44.0	10.8	23.9	link
Temp. Scaling	14.2	44.0	12.3	23.9	-
MDCA	12.2	44.0	11.1	23.5	link
MbLS	15.7	44.4	12.4	23.5	link
BPC (Ours)	10.3	43.7	9.4	23.2	link

Setup

Requirements: This implementation is on:

Linux, CUDA>=11.0

Python>=3.7

PyTorch>=1.7.0

Installation

For complete Installation, and usage instructions, follow guidelines here

Training and Evaluation

Training on single node

For example, the command for training BPC loss with Deformable DETR on <#GPUs> is as following:

GPUS_PER_NODE=<#GPUs> ./tools/run_dist_launch.sh <#GPUs> ./configs/r50_deformable_detr.sh

Training on slurm cluster

If you are using slurm cluster, you can run the following command to train on 1 node with <#GPUs>:

GPUS_PER_NODE=<#GPUs> ./tools/run_dist_slurm.sh <partition> deformable_detr <#GPUs> configs/r50_deformable_detr.sh

Evaluation

Config file and pretrained model available in the above section of Results, run following command to evaluate

<path to config file> --resume <path to pre-trained model> --eval

D-ECE Evaluation

For Detection Expected Calibration Error (D-ECE) evaluation, follow the guidelines here

Citation

Please cite the following, if you find this work useful in your research:

@article{munir2023bridging,
  title={Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection},
  author={ Munir, Muhammad Akhtar and Khan, Muhammad Haris and Khan, Salman and Khan, Fahad},
  journal={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023}
}

Contact

In case of any query, create issue or contact akhtar.munir@mbzuai.ac.ae

Acknowledgement

This codebase is built on Deformable-DETR, Object Detection Metrics and Detection Calibration