Initial commit

README.md

@@ -0,0 +1,112 @@
+Self-Supervised Multi-Object Tracking with Cross-Input Consistency
+------------------------------------------------------------------
+
+UNS20 is the code for "Self-Supervised Multi-Object Tracking with Cross-Input Consistency" (NeurIPS 2021).
+UNS20 is an approach for training a robust multi-object tracking model using
+only an object detector and a large corpus of unlabeled video.
+
+
+Installation
+------------
+
+Requires Tensorflow 1.15:
+
+	pip install 'tensorflow<2.0' scikit-image
+
+Download MOT17 dataset:
+
+	mkdir /home/ubuntu/data/
+	wget https://motchallenge.net/data/MOT17.zip
+	unzip MOT17.zip
+	mv MOT17 /home/ubuntu/data/mot17/
+
+Download UNS20 model:
+
+	wget https://favyen.com/files/uns20-model.zip
+	mv model/ /home/ubuntu/model/
+
+
+Inference
+---------
+
+For SDP detections:
+
+	cd /path/to/uns20/
+	python scripts/mot2json.py /home/ubuntu/data/ test
+	python infer.py /home/ubuntu/model/model /home/ubuntu/data/
+
+DPM and FRCNN detections have lower accuracy than SDP detections. Recent
+methods universally perform regression and classification pre-processing steps.
+Classification prunes incorrect input detections, while regression improves the
+bounding box coordinates. These steps don't really make sense, since they use a
+better detector to improve lower-quality detections. However, the steps are
+needed to achieve performance comparable with other methods, since all methods
+now use the same steps.
+
+To apply UNS20 on DPM and FRCNN detections, it should be executed after the
+regression and classification pre-processing steps from https://github.com/phil-bergmann/tracking_wo_bnw.
+
+For the most informative comparison, we highly recommend comparing performance
+only on the SDP detections, which have the highest accuracy. While evaluating
+on lower-quality detections sounds like it could be useful, one would really be
+evaluating the pre-processing steps more than the method itself.
+
+
+Evaluation
+----------
+
+Convert from JSON to the TXT format:
+
+	mkdir /home/ubuntu/outputs/
+	python scripts/json2mot.py /home/ubuntu/data/ train /home/ubuntu/outputs/
+
+Compare:
+
+	pip install motmetrics
+	python -m motmetrics.apps.eval_motchallenge /home/ubuntu/data/mot17/train/ /home/ubuntu/outputs/
+
+
+Training
+--------
+
+First, obtain PathTrack and YT-Walking datasets:
+
+	wget https://data.vision.ee.ethz.ch/daid/MOT/pathtrack_release_v1.0.zip
+	wget https://favyen.com/files/yt-walking.zip
+	mkdir /home/ubuntu/data/yt-walking/
+	unzip yt-walking.zip -d /home/ubuntu/data/yt-walking/
+	mkdir /home/ubuntu/data/pathtrack/
+	unzip pathtrack_release_v1.0.zip
+	mv pathtrack_release /home/ubuntu/data/pathtrack/
+
+Extract video frames from YT-Walking mp4 files:
+
+	python scripts/ytw-extract.py /home/ubuntu/data/
+
+Convert MOT17 object detections to uniform JSON format:
+
+	python scripts/mot2json.py /home/ubuntu/data/ train
+	python scripts/mot2json.py /home/ubuntu/data/ test
+
+Convert PathTrack object detections to uniform JSON format:
+
+	python scripts/pathtrack.py /home/ubuntu/data/
+
+Normalize MOT17 and PathTrack datasets:
+
+	python scripts/symlink.py mot17 /home/ubuntu/data/
+	python scripts/symlink.py pathtrack /home/ubuntu/data/
+
+Pre-process each of the three datasets using `scripts/preprocess-info.py` and `scripts/preprocess-matches.go`.
+
+	python scripts/preprocess-info.py mot17 /home/ubuntu/data/ 8
+	python scripts/preprocess-info.py pathtrack /home/ubuntu/data/ 8
+	python scripts/preprocess-info.py yt-walking /home/ubuntu/data/ 8
+	go run scripts/preprocess-matches.go mot17 /home/ubuntu/data/
+	go run scripts/preprocess-matches.go pathtrack /home/ubuntu/data/
+	go run scripts/preprocess-matches.go yt-walking /home/ubuntu/data/
+
+Train the model:
+
+	mkdir /home/ubuntu/model/
+	python train.py /home/ubuntu/data/ /home/ubuntu/model/model