Visually Grounded Compound PCFGs

Code for the paper Visually Grounded Compound PCFGs by Yanpeng Zhao and Ivan Titov @ EMNLP 2020.

Update (06/02/2022): Please look in vpcfg_text.py, vpcfg.py, and vpcfg_image.py for parsing MSCOCO and Flickr30k captions, creating data splits, and encoding images, respectively, for VC-PCFG. NOTE that VG-NSL and VC-PCFG used a pre-trained ResNet-152 to encode images but mistakenly reported it as ResNet-101. We have corrected it in the image encoding script.

Update (12/12/2021): VC-PCFG has been integrated into a repo dedicated for CFG-focused models.

Data

The processed data can be downloaded here.

Pre-trained Models

The best checkpoints of VC-PCFG can be downloaded here (MD5 checksum: d7de50d06590004061d24a69db0ac64c).

Learning

C-PCFGs on Text

python train.py \
    --data_path $DATA_PATH \
    --logger_name $SAVE_PATH

C-PCFGs on Image & Text

python train.py \
    --visual_mode "O" \
    --data_path $DATA_PATH \
    --logger_name $SAVE_PATH

In both cases please specify DATA_PATH and SAVE_PATH before running (DATA_PATH is where the downloaded data resides; SAVE_PATH is where your model will be saved).

Parsing

Remember to specify MODEL_FILE and DATA_PATH first.

python eval.py \
    --model $MODEL_FILE \
    --data_path $DATA_PATH

Dependencies

It requires a tailored Torch-Struct.

git clone --branch beta https://github.com/zhaoyanpeng/vpcfg.git
cd vpcfg
virtualenv -p python3.7 ./pyenv/oops
source ./pyenv/oops/bin/activate
pip install --upgrade pip
pip install --upgrade setuptools
pip install -r requirements.txt
git clone --branch infer_pos_tag https://github.com/zhaoyanpeng/pytorch-struct.git
cd pytorch-struct
pip install -e .

Acknowledgements

This repo is developed based on VGNSL, C-PCFGs, and Torch-Struct.

License

MIT