Skip to content

anubhav-jangra/Two_branch_network

 
 

Repository files navigation

Two-Branch Neural Networks

Usage:

  • Train a model from scratch: Set the path to the training dataset and checkpoint save directory in run_two_branch.sh. Run sh run_two_branch.sh --train.
  • Evaluate a model on the validation set while training: Set the path to validation dataset and checkpoint save directory. Additionally, adjust the time interval between each evaluation in eval_embedding.py. By default, the script will evaluate once per minute on the newest checkpoint in the given checkpoint directory. Run sh run_two_branch.sh --val.
  • Evaluate a model on a specific checkpoint: Set the path to the test dataset and checkpoint MetaGraph (.meta file). Run sh run_two_branch.sh --test.
  • Use a pre-trained model: Download checkpoints from the URLs below. Follow the instruction for evaluating model on a specific checkpoint.

Any questions, feel free to contact me, lwang97@illinois.edu

Dataset:

Due to the size of the features (~17G for MSCOCO and 7G for Flickr30K), only the test split is available for download.

Data set splits:

  • Train, and test splits (the left is used for validation), please check readme.txt download

To extract images and sentences features,

  • For image features, we use the VGG 19 network to extract features.
  • For sentence features, we are using the code from "B. Klein, G. Lev, G. Sadeh, L. Wolf. Associating Neural Word Embeddings With Deep Image Representations Using Fisher Vectors (CVPR 2015)". Their code can be downloaded here.

Pre-trained models:

If you find our code helpful, please cite our Two-Branch Network Papers:

References

@inproceedings{wang2016learning,
title={Learning deep structure-preserving image-text embeddings},
author={Wang, Liwei and Li, Yin and Lazebnik, Svetlana},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={5005--5013},
year={2016}
}

@article{wang2017learning,
title={Learning Two-Branch Neural Networks for Image-Text Matching Tasks},
author={Wang, Liwei and Li, Yin and Lazebnik, Svetlana},
journal={arXiv preprint arXiv:1704.03470},
year={2017}
}

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 96.1%
  • Shell 3.9%