Official repo for MeshXL

MeshXL: Neural Coordinate Field for Generative 3D Foundation Models

Project Page • Arxiv Paper • HuggingFace Demo • Citation

🏃 Intro MeshXL

MeshXL is a family of generative pre-trained foundation models for 3D mesh generation. With the Neural Coordinate Field representation, the generation of unstructured 3D mesh data can be seaminglessly addressed by modern LLM methods. In this paper, we validate the Neural Coordinate Field (NeurCF), an explicit coordinate representation with implicit neural embeddings, is a simple-yet-effective representation for large-scale sequential mesh modeling.

🚩 News

• [2024/09/26] MeshXL is accepted to NeurIPS 2024🔥! See you in Vancouver!
• [2024/08/29] Upload code and 🤗weights for text-to-mesh generation, welcome to check it out!
• [2024/07/24] Upload the inference code and pre-trained weights.
• [2024/06/02] Upload paper and init project.

⚡ Quick Start

Environment Setting Up

You can build the environment using the provided script:

bash set_env.sh

Data

Work in progress...

💻 Training and Evaluation

Download Pre-Trained Weights

[Special Notes]: All the following models are generative pre-trained base models. They are capable of unconditional 3D mesh generation and partial mesh completion.

We provide pre-trained weights for different sizes of models (i.e. 125m, 350m, and 1.3b) on huggingface🤗. Download the pre-trained weights from the links below to replace the pytorch_model.bin files in the corresponding folders under the ./mesh-xl/ folder. The model details are shown below:

Model Size	#Layers	#Heads	$d_\text{model}$	$d_\text{FFN}$	GPU Hours	Download Link
125M	12	12	768	3072	1944	download link
350M	24	16	1024	4096	6000	download link
1.3B	24	32	2048	8192	23232	download link

MeshXL Generative Pre-Training

Work in progress...

Generating Samples

To generate 3D meshes with different sizes, feel free to use the following commands. By default, we generate samples with 8 GPUs and the top-k top-p sampling strategy for diverse samples.

bash scripts/sample-1.3b.sh
bash scripts/sample-350m.sh
bash scripts/sample-125m.sh

[Special Notes]: The following weights are fine-tuned for unconditional 3D mesh generation on a specified category.

Want to generating shapes for a specified category? We have also uploaded the supervised fine-tuned checkpoints on chair, table, bench, lamp to huggingface too! Download the fine-tuned weights from the links🤗 below.

Model Size	Table	Chair	Lamp	Bench
125M	download link	download link	download link	download link
350M	download link	download link	download link	download link
1.3B	download link	download link	download link	download link

After you have downloaded the corresponding checkpoints, please use the following command to generate samples.

export LLM_CONFIG='mesh-xl/mesh-xl-125m'
# the checkpoint mush align with the $LLM_CONFIG
export TEST_CKPT='./ckpts-meshxl-125m-sft/meshxl-125m-bench.pth'

accelerate launch \
    --num_machines 1 \
    --num_processes 8 \
    --mixed_precision bf16 \
    main.py \
    --dataset dummy_dataset \
    --n_max_triangles 800 \
    --n_discrete_size 128 \
    --llm mesh-xl/mesh-xl-125m \
    --model mesh_xl \
    --checkpoint_dir ./outputs \
    --batchsize_per_gpu 2 \
    --test_ckpt $TEST_CKPT \
    --sample_rounds 100 \
    --dataset_num_workers 0 \
    --test_only

Want to see more results? Check our project page out here!

Text-to-Mesh Generation

We thank the awesome language annotations from PointLLM for object captions. We fine-tune a 350m MeshXL model on Objaverse with 8 RTX-3090 GPUs.

Note: please download the pre-trained checkpoint from huggingface🤗 to replace the ./mesh-xl/x-mesh-xl-350m/pytorch_model.bin file.

We are actively working on Gradio demos. Currently, we encourage you to generate samples locally with at least 1 GPU with the following code:

bash scripts/sample-t2mesh.sh

You are also welcome to explore other text conditions and hyper-parameters for better controls:

accelerate launch \
  --num_machines 1 \
  --num_processes 1 \
  --mixed_precision bf16 \
  sample_t2m.py \
  --test_ckpt mesh-xl/x-mesh-xl-350m/pytorch_model.bin \
  --text '3d model of a table' \  # change to the text prompt you need
  --top_k 50 \                    # larger k -> larger randomness
  --top_p 0.95 \                  # larger p -> larger randomness
  --temperature 0.1               # larger temperature -> larger randomness

📖 Citation

If you find our code or paper helps, please consider citing:

@misc{chen2024meshxl,
      title={MeshXL: Neural Coordinate Field for Generative 3D Foundation Models}, 
      author={Sijin Chen and Xin Chen and Anqi Pang and Xianfang Zeng and Wei Cheng and Yijun Fu and Fukun Yin and Yanru Wang and Zhibin Wang and Chi Zhang and Jingyi Yu and Gang Yu and Bin Fu and Tao Chen},
      year={2024},
      eprint={2405.20853},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgments

We use Paint3D for texturing generated 3D meshes.

We express our genuine thanks to the amazing work: ShapeNet, 3D-FUTURE, Objaverse, Objaverse-XL, PolyGen, Get3D and MeshGPT, and the amazing MeshGPT-pytorch codebase.