RetinaQA: A Robust Knowledge Base Question Answering Model for both Answerable and Unanswerable Questions

This repository contains the open-sourced official implementation of the paper published at ACL'24.

[video] | [slides] | [poster]

• RetinaQA is a robust KBQA model for detecting answerability.
• It is a multi-staged Retrieve, Generate and Rank framework.
• It contains following components : Schema Retriver, Logical Form Retriver, Sketch Generation, Logical Form Integrator and a Discriminator.

Environment Setup

conda create -n retinaqa python=3.9
conda activate retinaqa
pip install -r requirements.txt

Note: Please install cuda dependencies with version which matches configurations of your machine.

KG Setup

• For GrailQA and WebQSP we use the original latest dump of FreeBase, please follow the steps here.
• For GrailQAbility we update the FreeBase KB as steps mentioned here.

Please note that around 55 GB disk space and 30 GB CPU RAM will be required to host the KB. If you have a machine with higher RAM configurations, then you can increase the ServerThreads paramter in (virtuoso.py](https://github.com/dki-lab/Freebase-Setup/blob/master/virtuoso.py) file.

Data Setup

To download and set up data for RetinaQA, run the following commands:

python setup.py --dataset grailqability # for GrailQAbility
python setup.py --dataset grailqa # for GrailQA
python setup.py --dataset webqsp # for WebQSP

The outputs from various retrievers are used as input to RetinaQA. The downloaded dataset includes outputs from multiple pre-trained retrievers. If you wish to train retrievers from scratch, please refer to the steps outlined in the next section.

Model Checkpoints

To run inference directly on GrailQAbility, use the links provided here to access pre-trained model checkpoints for each stage. Follow the instructions in the respective sections to execute the inference process.

Training Type	Module	Checkpoint link
A	Sketch Generation Logical Form Candidates Cache Discriminator	link link link
AU	Sketch Generation Logical Form Candidates Cache Discriminator	link link link

To run inference directly on GrailQA, use the links provided here to access pre-trained model checkpoints for each stage. Follow the instructions in the respective sections to execute the inference process.

Module	Checkpoint link
Sketch Generation Logical Form Candidates Cache Discriminator	link link link

Training RetinaQA from scratch

1. Entity Linker Please refer RnG-KBQA to train entity linker for GrailQAbility. Follow (i) Detecting Entities and (ii) Disambiguating Entities. For GrailQA and WebQSP we use TIARA's Entity Linker. Note: TIARA's entity linker is better than RnG-KBQA's entity linker but is not opne-sourced, so for training for GrailQAbility we use RnG-KBQA's entity linker.

2. Logical Form Retriever Please refer RnG-KBQA for logical form retriever. It includes two steps: Enumerating Logical Form Candidates(section (iii)) and Running Ranker (section (iv)). From the ranked logical form candidates we use top-10 logical forms as input to the final Discriminator stage of RetinaQA.

3. Schema Retriever Please refer TIARA for schema retriever. It includes training two models- one for entity types and one for relations. From the ranked schema items we use top-10 entity types(classes) and top-10 relations as input to the Logical Form Integrator stage of RetinaQA.

4. Sketch Generator Please follow steps mentioned in src/sketch_generation.

5. Logical form Integrator Please follow steps as mentioned in src/lf_integrator.

6. Discriminator (Final Stage) Please follow steps as mentioned in src/discriminator.

Acknowledgements

Parts of our code/data are taken from:

• https://github.com/dair-iitd/GrailQAbility
• https://github.com/microsoft/KC/tree/main/papers/TIARA
• https://github.com/salesforce/rng-kbqa/

Citation

@inproceedings{faldu-etal-2024-retinaqa,
    title = "{R}etina{QA}: A Robust Knowledge Base Question Answering Model for both Answerable and Unanswerable Questions",
    author = "Faldu, Prayushi  and
      Bhattacharya, Indrajit  and
      ., Mausam",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.359",
    doi = "10.18653/v1/2024.acl-long.359",
    pages = "6643--6656"
}