Reconstructing visual illusory experiences from human brain activity

Fan Cheng, Tomoyasu Horikawa, Kei Majima, Misato Tanaka, Mohamed Abdelhack, Shuntaro C. Aoki, Jin Hirano, Yukiyasu Kamitani
Paper · CCN2022 · bioRxiv

Our paper has been published in Science Advances !

Getting Started

Installation

Clone the repo:

git clone https://github.com/KamitaniLab/IllusionReconstruction.git

Build Environment

Step1: Navigate to the base directory and create the environment by running the following command.

conda env create -f env.yaml

Step2: Activate the environment.

conda activate brain_decoding-to-generator

Download Dataset

To utilize this project, you'll need to download the required dataset Figshare and organize the dataset appropriately. You can download the required data with the following commands.

fMRI data and image feature:

# In "data" directory:

# Training and test fMRI data
$ python download.py fmri_training
$ python download.py fmri_test

# Stimulus image features
$ python download.py stimulus_feature

Pre-trained generator:

# In "generator" directory:

# GAN
$ python download.py GAN

Dataset links

• fMRI data (unpreprocessed, BIDS): OpenNeuro doi:10.18112/openneuro.ds004670.v1.0.1
• Preprocessed fMRI data, timulus image features, and pretrained image generators: figshare doi:10.6084/m9.figshare.23590302

Usage

(1) To quickly test the reconstruction code, run:

./feature_decoding-to-generator_quick_test.sh

(2) To replicate main results of the paper, run:

Reconstruct from single-trial fMRI samples using GAN generator

./feature_decoding-to-generator.sh

Evaluate single-trial reconstructions from individual brain regions

./evaluation_line_color.sh

(3) In addition, to test results using diffusion generator, run:

./feature_decoding-to-generator_diffusion.sh

Example output figure

Example 1 (GAN): You can find the following figure in results/plots/quick_test. From left to right columns: stimulus (1), reconstruction from stimulus features (2), reconstruction from brain-decoded features (3-9 correspond to Subject 1-7; using fMRI sample averaged across trials)

Example 2 (diffusion): You can find the following figure in results/plots/. Each row shows the reconstructions from the same fMRI sample (a single trial of the stimulus) with different random seeds.

Citation

Cheng, F. L., Horikawa, T., Majima, K., Tanaka, M., Abdelhack, M., Aoki, S. C., Hirano, J., & Kamitani, Y. (2023) Reconstructing visual illusory experiences from human brain activity. Sci. Adv. 9, eadj3906. DOI: 10.1126/sciadv.adj3906

Contact

Fan Cheng - @LibraCheng - chengfanbrain@gmail.com

(back to top)