UGainS: Uncertainty Guided Anomaly Instance Segmentation (GCPR 2023)

Alexey Nekrasov¹, Alexander Hermans¹, Lars Kuhnert², Bastian Leibe¹

¹RWTH Aachen University, ²Ford Motor Company

UGainS is an approach that provides anomaly instance segmentation and per-pixel anomaly scores.

[Project Webpage] [arXiv]

News

• 2023-07-14: Paper Accepted at GCPR 2023
• 2023-08-03: Paper on arXiv
• 2024-04-18: Code release
• 2024-06-15: New benchmark! OoDIS: Anomaly Instance Segmentation Benchmark

Code

I took a long time to release the code, mainly because it was a lot of hacking together at the time and didn't really have a concise structure with multiple commented lines instead of having configs and hardwired paths all over the code. To make the code understandable, I reduced the context a bit. This version only contains experiments for a single dataset, but provides a clean overview of the project. Please feel free to open a github issue if you have any questions or need help with the code.

Installation

Prerequisites

• Python 3.11
• CUDA 11.8
• cuDNN 8.6.0

Setting Up Your Environment

It is recommended to use a virtual environment for the project to manage dependencies effectively.

1. Create and activate a virtual environment (optional but recommended):

   python -m venv venv
   source venv/bin/activate

2. Install dependencies: Ensure you have Poetry installed. If not, install Poetry using the following command:

   pip install poetry

   Install Mask2former ops:

   cd ugains/models/mask2former/pixel_decoder/ops/
   bash make.sh
   cd ../../../../..
   

   Install FPS:

   cd third_party/pointnet2
   python setup.py install
   

   Then, install the project dependencies with Poetry:

   poetry install

Example Usage

To run the model with the specified configuration, use the following command:

poetry run test datamodule=cityfishy model=sam_sampling logger=csv experiment=experiment description=description

Dependencies

The project relies on several key Python packages, including but not limited to:

• PyTorch (torch>=2.0.1)
• TorchVision (torchvision>=0.15.2)
• PyTorch Lightning (pytorch-lightning>=2.0.2)
• Hydra Core (hydra-core>=1.3.2)
• Weights & Biases (wandb>=0.15.2)
• Albumentations (albumentations>=1.3.0)
• Rich (rich>=13.3.5)
• Python-dotenv (python-dotenv>=1.0.0)
• Fire (fire>=0.5.0)
• Joblib (joblib>=1.2.0)
• Ninja (ninja>=1.11.1)
• GitPython (gitpython>=3.1.31)
• Pandas (pandas>=2.0.1)
• Seaborn (seaborn>=0.12.2)
• Matplotlib (matplotlib>=3.7.1)
• Numpy (numpy>=1.24.3)
• pycocotools
• einops
• git+https://github.com/mcordts/cityscapesScripts.git

Data Preparation

data
├── fs_lost_found
│   ├── leftImg8bit
│   └── gtCoarse
│       ├── train
│       └── test
│           ├── xx_xxx_000000_000000_gtCoarse_labelTrainIds.png
│           └── ...
├── fs_lost_found_instance
│   └── gtCoarse
│       ├── train
│       └── test
│           ├── xx_xxx_000000_000000_gtCoarse_instanceIds.png
│           └── ...
├── ignore_mask.pth
├── rude0fhk.ckpt
└── sam_vit_h_4b8939.pth

Instance labels, Mask2Former model, and ignore mask could be downloaded here. Sam checkpoint could be downloaded from here. Lost and Found images could be found here, and validation images from Fishyscapes Lost and Found here.

BibTeX

@inproceedings{nekrasov2023ugains,
  title     = {{UGainS: Uncertainty Guided Anomaly Instance Segmentation}},
  author    = {Nekrasov, Alexey and Hermans, Alexander and Kuhnert, Lars and Leibe, Bastian},
  booktitle = {GCPR},
  year      = {2023}
}

Thanks

A big thanks to the authors of Mask2Former, DenseHybrid, SegmentAnything, PEBAL and all the others.