Skip to content

Latest commit

 

History

History
65 lines (48 loc) · 2.74 KB

README.md

File metadata and controls

65 lines (48 loc) · 2.74 KB
Raw

Text2Brain

brain interpreter review

Generating brain activation maps from free-form text query

Reference

Gia H. Ngo, Minh Nguyen, Nancy F. Chen, Mert R. Sabuncu. Text2Brain: Synthesis of Brain Activation Maps from Free-form Text Query. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021. arxiv

Overview

Text2Brain is a search engine for efficiently combing through rapidly growing wealth of neuroimaging literature brain activation patterns. It accepts not only keywords but also flexible free-form text queries. It encodes the text queries using a finetuned Transformer encoder (SciBERT) and generates whole-brain activation maps using a 3D convolutional neural network (CNN).

Demo is available at: braininterpreter.com

Outputs for Synonymous Queries

Activation maps predicted by Text2Brain for 3 synonymous queries: default network, self-generated thought, and task-unrelated thought. The ground-truth activation map is also included in the figure.

Setting Up Text2Brain Project

  1. Install Anaconda
  2. Clone this project from Github to some place on your computer (e.g. /home/gia/text2brain)
  3. Create a Conda environment using the env.yml file 
        conda env create -f env.yml -n text2brain
  4. Download a checkpoint of the Text2Brain model from Google Drive
  5. Extract the downloaded file. You should see a file named best_loss.pth. 
        tar -xzvf text2brain_checkpoint.tar.gz
  6. Move the file best_loss.pth into the project directory (e.g. /home/gia/text2brain)
  7. Download the pre-trained uncased SciBERT model using this link
  8. Extract the downloaded file. 
        tar -xvf scibert_scivocab_uncased.tar
  9. Move the scibert_scivocab_uncased folder into the project directory (e.g. /home/gia/text2brain)
  10. Activate the Conda environment 
        conda activate text2brain
  11. To generate brain activation maps from free-form text query, run python predict.py <input_query> <output_file>. For example, 
    python predict.py "self-generated thought" prediction.nii.gz

Bugs and Questions

Please contact Gia at ngohoanggia@gmail.com