Q-Bench Data Release

Dataset

This release is to publish the low-level-vision to language datasets for Q-Bench (A1/A2). The datasets include the LLVisionQA dataset (dev set and test set), and the LLDescribe dataset. All labels and images are in this release.

Submission

For the first two tasks, please prepare your outputs and corresponding image ids (image paths) in json format and email to haoning001@e.ntu.edu.sg. Start your email with [Q-Bench Submission].

Updated Leaderboard

The new leaderboard is cut-off on day 13rd October, and has 13 models evaluated for the three tasks now. See leaderboards.

Commit No.

The latest commit under this release is export COMMIT_NO=5ae6be7f7f1cf645b2630835f9902d0114d05ac2. You can always checkout Q-Bench at this version via git checkout $COMMIT_NO.

Citation

If you find our work interesting, please feel free to cite our paper:

@article{wu2023qbench,
    title={Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision},
    author={Wu, Haoning and Zhang, Zicheng and Zhang, Erli and Chen, Chaofeng and Liao, Liang and Wang, Annan and Li, Chunyi and Sun, Wenxiu and Yan, Qiong and Zhai, Guangtao and Lin, Weisi},
    year={2023},
}```