Awesome-ComposableAI

A curated list of Composable AI methods: Building AI system by composing modules.

It includes Composable-Model, Composable-Task, Composable-Gen, Composable-Agent, and Composable-X, etc.

Contributions are welcome!

Let's build AI as Lego!

Table of Content

• Composable-Task
  • Projects & Papers
  • Project Only
• Composable-Model
  • Projects & Papers
• Composable-Gen
  • Projects & Papers
• Composable-Agent
  • Paper Only
• Composable-Product
• Composable-X

Definition

A paramount issue in AI is the combinational challenge: It is infeasible to enumerate all possibilities for an intelligent system.

Composable AI offers a solution to this challenge by emphasizing the creation of modular, flexible, and reusable AI components. These components can be assembled and reconfigured in various ways, enabling the construction of customized AI systems that are specifically tailored to individual tasks, domains, or applications.

Composable-Task

Projects & Papers

Title & Authors	Intro	Useful Links
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang Preprint'23 [Jarvis (Project)]		[Github] [Demo]
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, Lijuan Wang Preprint'23 [MM-REACT (Project)]		[Github] [Page] [Demo]
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, Nan Duan Preprint'23		[Github]
OpenAGI: When LLM Meets Domain Experts Yingqiang Ge, Wenyue Hua, Jianchao Ji, Juntao Tan, Shuyuan Xu, Yongfeng Zhang [OpenAGI (Project)]		Github
ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, Mohamed Elhoseiny Preprint'23 [ChatCaptioner (Project)]		Github
Visual Programming: Compositional visual reasoning without training Tanmay Gupta, Aniruddha Kembhavi CVPR'23 Visprog (Project)		[Github] [Page]
ViperGPT: Visual Inference via Python Execution for Reasoning Dídac Surís, Sachit Menon, Carl Vondrick CVPR'23 Viper (Project)		[Github] [Page]
Composing Ensembles of Pre-trained Models via Iterative Consensus Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Igor Mordatch ICLR'23		[Page]
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence ICLR'23 socraticmodels (Project)		[Github] [Page]

Project Only

Title & Authors	Intro	Useful Links
Grounded-SAM (Project) [Grounding DINO + Segment-Anything + X] Shilong Liu and Zhaoyang Zeng and Tianhe Ren and Feng Li and Hao Zhang and Jie Yang and Chunyuan Li and Jianwei Yang and Hang Su and Jun Zhu and Lei Zhang		[Github] [Demo]
Semantic-Segment-Anything (Project) [Close-set Segmenters + Open-vocabulary Models] Jiaqi Chen, Zeyu Yang, Li Zhang		[Github]
Inpaint-Anything: Segment Anything Meets Image Inpainting (Project) [SAM + LaMa + Stable Diffusion] Tao Yu		[Github]
Segment Anything and Name It (Project) [Visual ChatGPT + GLIP + Segment-Anything] Peize Sun and Shoufa Chen		[Github]
EditAnything (Project) [Segment Anything + ControlNet + BLIP2 + Stable Diffusion] Shanghua Gao, Pan Zhou		[Github]

Composable-Model

Projects & Papers

Title & Authors	Intro	Useful Links
AdapterHub: A Framework for Adapting Transformers Álvaro Barbero Jiménez EMNLP'20 [adapter-transformers (Project)]		[Github] [Page]
Deep Model Reassembly Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, Xinchao Wang NeurIPS'22 [DeRy (Project)]		[Github] [Page]
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean ICLR'17 [mixture-of-experts (Project)]		[Github]

Composable-Gen

Projects & Papers

Title & Authors	Intro	Useful Links
Mixture of Diffusers for scene composition and high resolution image generation Álvaro Barbero Jiménez Preprint'23 [Mixture-of-Diffusers (Project)]		[Github] [Demo]
Compositional Visual Generation with Composable Diffusion Models Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, Joshua B. Tenenbaum ECCV'22 [Composable Diffusion (Project)]		[Github] [Demo] [Page]
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, William Yang Wang ICLR'23 [Structured-Diffusion-Guidance (Project)]		[Github] [Page]
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, Will Grathwohl Preprint'23 [reduce_reuse_recycle (Project)]		[Github] [Page]
Learning to Compose Visual Relations Nan Liu, Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba NeurIPS'21 [compose-visual-relations (Project)]		[Github] [Page]

Composable-Agent

Paper Only

• MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, Sergey Levine NeurIPS'19 [Paper]
• Composable Planning with Attributes Amy Zhang, Sainbayar Sukhbaatar, Adam Lerer, Arthur Szlam, Rob Fergus ICML'18 [Paper]

Composable-Product

Title & Authors	Intro	Useful Links
langchain (Project) [LLM + X]		Github

Composable-X

TO BE UPDATE