Official repo for MeshXL

MeshXL: Neural Coordinate Field for Generative 3D Foundation Models

Project Page • Arxiv Paper • HuggingFace Demo • Citation

🏃 Intro MeshXL

MeshXL is a family of generative pre-trained foundation models for 3D mesh generation. With the Neural Coordinate Field representation, the generation of unstructured 3D mesh data can be seaminglessly addressed by modern LLM methods.

The polygon mesh representation of 3D data exhibits great flexibility, fast rendering speed, and storage efficiency, which is widely preferred in various applications. However, given its unstructured graph representation, the direct generation of high-fidelity 3D meshes is challenging. Fortunately, with a pre-defined ordering strategy, 3D meshes can be represented as sequences, and the generation process can be seamlessly treated as an auto-regressive problem. In this paper, we validate the Neural Coordinate Field (NeurCF), an explicit coordinate representation with implicit neural embeddings, is a simple-yet-effective representation for large-scale sequential mesh modeling. After that, we present MeshXL, a family of generative pre-trained auto-regressive models, which addresses the process of 3D mesh generation with modern large language model approaches. Extensive experiments show that MeshXL is able to generate high-quality 3D meshes, and can also serve as foundation models for various down-stream applications.

🚩 News

• [2024/07/24] Upload the inference code and pre-trained weights.
• [2024/06/02] Upload paper and init project.

⚡ Quick Start

Environment Setting Up

You can build the environment using the provided script:

bash set_env.sh

Data

Work in progress...

💻 Training and Evaluation

Download Pre-Trained Weights

[Special Notes]: All the currently released weights are only used for unconditional 3D mesh generation.

We provide pre-trained weights for different sizes of models (i.e. 125m, 350m, and 1.3b) on huggingface. Download the pre-trained weights from the links below to replace the pytorch_model.bin files in the corresponding folders under the ./mesh-xl/ folder. The model details are shown below:

Model Size	#Layers	#Heads	$d_\text{model}$	$d_\text{FFN}$	GPU Hours	Download Link
125M	12	12	768	3072	1944	download link
350M	24	16	1024	4096	6000	download link
1.3B	24	32	2048	8192	23232	download link

MeshXL Generative Pre-Training

Work in progress...

Generating Samples

To generate 3D meshes with different sizes, feel free to use the following commands. By default, we generate samples with 8 GPUs and the top-k top-p sampling strategy for diverse samples.

bash scripts/sample-1.3b.sh
bash scripts/sample-350m.sh
bash scripts/sample-125m.sh

[Special Notes]: All the currently released weights are fine-tuned for unconditional 3D mesh generation on a specified category.

Want to generating shapes for a specified category? We have also uploaded the supervised fine-tuned checkpoints on chair, table, bench, lamp to huggingface too! Download the fine-tuned weights from the links below.

Model Size	Table	Chair	Lamp	Bench
125M	download link	download link	download link	download link
350M	download link	download link	download link	download link
1.3B	download link	download link	download link	download link

After you have downloaded the corresponding checkpoints, please use the following command to generate samples.

export LLM_CONFIG='mesh-xl/mesh-xl-125m'
# the checkpoint mush align with the $LLM_CONFIG
export TEST_CKPT='./ckpts-meshxl-125m-sft/meshxl-125m-bench.pth'

accelerate launch \
    --num_machines 1 \
    --num_processes 8 \
    --mixed_precision bf16 \
    main.py \
    --dataset dummy_dataset \
    --n_max_triangles 800 \
    --n_discrete_size 128 \
    --llm mesh-xl/mesh-xl-125m \
    --model mesh_xl \
    --checkpoint_dir ./outputs \
    --batchsize_per_gpu 2 \
    --test_ckpt $TEST_CKPT \
    --sample_rounds 100 \
    --dataset_num_workers 0 \
    --test_only

Want to see more results? Check our project page out here!

📖 Citation

If you find our code or paper helps, please consider citing:

@misc{chen2024meshxl,
      title={MeshXL: Neural Coordinate Field for Generative 3D Foundation Models}, 
      author={Sijin Chen and Xin Chen and Anqi Pang and Xianfang Zeng and Wei Cheng and Yijun Fu and Fukun Yin and Yanru Wang and Zhibin Wang and Chi Zhang and Jingyi Yu and Gang Yu and Bin Fu and Tao Chen},
      year={2024},
      eprint={2405.20853},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgments

Thanks to these amazing works: ShapeNet, 3D-FUTURE, Objaverse, and Objaverse-XL.