The Cradle framework is a first attempt at General Computer Control (GCC). Cradle supports agents to ace any computer task by enabling strong reasoning abilities, self-improvment, and skill curation, in a standardized general environment with minimal requirements.

[CVPR 2024 🔥] Grounding Large Multimodal Model (GLaMM), the first-of-its-kind model capable of generating natural language responses that are seamlessly integrated with object segmentation masks.

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

LLaVA-Interactive-Demo

[CVPR'24] HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5, and Other Multi-modality Models

PG-Video-LLaVA: Pixel Grounding in Large Multimodal Video Models

The code for "TokenPacker: Efficient Visual Projector for Multimodal LLM".

Official code for Paper "Mantis: Multi-Image Instruction Tuning" (TMLR2024)

😎 curated list of awesome LMM hallucinations papers, methods & resources.

[LMM + AIGC] What do we expect from LMMs as AIGI evaluators and how do they perform?

A RLHF Infrastructure for Vision-Language Models

MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning

Official Repo of Graphist

🌋👵🏻 Yo'LLaVA: Your Personalized Language and Vision Assistant

LLaVA inference with multiple images at once for cross-image analysis.

A Large Multimodal Model for Pixel-Level Visual Grounding in Videos

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI.

LMM which strictly superset LLM embedded

[COLING 2025] Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputs

Utilities for field trial analysis.