A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation

This repository contains the official PyTorch implementation of the following paper.

A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation,
Georgy Ponimatkin, Nermin Samet, Yang Xiao, Yuming Du, Renaud Marlet, Vincent Lepetit,
WACV 2023. (arXiv pre-print, Project page)

Preparing the environment and data

To prepare the environment run the following commands:

conda env create --name ssl-vos python=3.8 pip
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
pip install scikit-learn opencv-python fast_slic imageio matplotlib scikit-image easydict path.py

python setup.py build
python setup.py install

To download the required assets run bash get_raw_data.sh and bash download_models.sh. Data can be prepared by running the following script

python prepare_data.py

Please refer to ARFlow and RAFT repositories in order to prepare rest of your environment.

Running the pipeline

Our approach requires three steps to run produce segmentations. At first, extract DINO features and optical flows by via

python extract_dino_features.py --dataset davis
python generate_flow_arflow.py --dataset davis --step 1

The second step consists of extracting the initial eigenvectors, which can be done by

python generate_pic_eigenvectors.py --use-gpu --dataset davis

Global optimization then can be run via

python global_optimization.py --dataset davis

The masks can be generated from the obtained solution by running

python extract_masks.py --dataset davis --method name_of_the_folder_in_data/davis

Citation

If you use this code in your research, please cite the following paper:

G. Ponimatkin, N. Samet, Y. Xiao, Y. Du, R. Marlet and V. Lepetit "A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation", In IEEE Winter Conference on Applications of Computer Vision (WACV), 2023.

BibTeX entry:

@inproceedings{ponimatkin2023sslvos, 
title= {A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation}, 
author={G. {Ponimatkin} and N. {Samet} and Y. {Xiao} and Y. {Du} and R. {Marlet} and V. {Lepetit}}, 
booktitle={Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)}, 
year={2023}} }