Merge branch 'main' of github.com:Ir1d/Comp4D

README.md

@@ -1,20 +1,19 @@
-# Comp4D: LLM-Guided Compositional 4D Scene Generation
+# Comp4D: Compositional 4D Scene Generation
 
-Authors: Dejia Xu, Hanwen Liang, Neel P. Bhatt, Hezhen Hu, Hanxue Liang,
-Konstantinos N. Plataniotis, and Zhangyang Wang
+The official implementation of paper "Comp4D: Compositional 4D Scene Generation".
 
 [[Project Page]](https://vita-group.github.io/Comp4D/) | [[Video (narrated)]](https://www.youtube.com/watch?v=9q8SV1Xf_Xw) | [[Video (results)]](https://www.youtube.com/watch?v=gXVoPTGb734) | [[Paper]](https://github.com/VITA-Group/Comp4D/blob/main/assets/Comp4D.pdf) | [[Arxiv]](https://arxiv.org/abs/2403.16993)
 
 ## News
-
+- 2024.8.19:  Revised to support more objects.
 - 2024.4.1:  Released code!
 - 2024.3.25:  Released on arxiv!
 
 ## Overview
 
 ![overview](docs/static/media/task.29476c66b38120ba3c46.jpg)
 
-As show in figure above, we introduce **Comp**ositional **4D** Scene Generation. Previous works concentrate on object-centric 4D objects with limited movement. In comparison, our work extends the boundaries to the demanding task of compositional 4D scene generation. We integrate GPT-4 to decompose the scene and design proper trajectories, resulting in larger-scale movements and more realistic object interactions.
+As shown in the figure above, we introduce **Comp**ositional **4D** Scene Generation. Previous works concentrate on object-centric 4D objects with limited movement. In comparison, our work extends the boundaries to the demanding task of compositional 4D scene generation. We integrate GPT-4 to decompose the scene and design proper trajectories, resulting in larger-scale movements and more realistic object interactions.
 
 <!-- ## Representative Results
 
@@ -49,8 +48,16 @@ pip install ./simple-knn
 
 #### Compositional Scene training
 ```
-python train_comp.py --configs arguments/comp_butterfly_flower_zs.py -e butterflyflower_exp --image_weight_override 0.02 --nn_weight 1000 --with_reg --cfg_override 100.0 --loss_dx_weight_override 0.005
+python train_comp.py --configs arguments/comp_butterfly_flower_zs.py --expname butterflyflower_exp --cfg_override 100.0 --image_weight_override 0.02 --nn_weight 1000 --with_reg  --loss_dx_weight_override 0.005
 ```
+--- 
+We provide a quick overview of some important arguments:  
+- `--expname`:  Experimental path.
+- `--configs`: Configuration of scene training including prompt, object identity, object scales, and trajectory. You can also use [VideoCrafter](https://github.com/AILab-CVC/VideoCrafter) in replace of Zeroscope for video-based diffusion model.
+- `--image_weight`: Weight of sds loss from image-based diffusion model.
+- `--nn_weight`: Weight of knn based rigidity loss.
+- `--loss_dx_weight`: Weight of regularization acceleration loss.
+
 
 #### Rendering
 ```
@@ -60,32 +67,29 @@ python render_comp_video.py --skip_train --configs arguments/comp_butterfly_flow
 
 ## Static Assets Preparation
 
-We release a set of pre-generated static assets in `data/` directory. During training we keep the static 3D Gaussians fixed and only optimize the deformation modules. We refered to the first two stages of [4D-fy](https://github.com/sherwinbahmani/4dfy) to generate the static 3D objects. Then we convert them to point clouds (in `data/`) which are used to initialize 3D Gaussians. Thanks the authors for sharing their awesome work!
+We release a set of pre-generated static assets in `data/` directory. During training, we keep the static 3D Gaussians fixed and only optimize the deformation modules. We referred to the first two stages of [4D-fy](https://github.com/sherwinbahmani/4dfy) to generate the static 3D objects. Then we convert them to point clouds (in `data/`) which are used to initialize 3D Gaussians. Thanks to the authors for sharing their awesome work!
 
 #### Example case
 ```
 
 # cd /path_to_4dfy/
-# seed=0
-# gpu=0
 
 ## Stage 1
-# python launch.py --config configs/fourdfy_stage_1_low_vram.yaml --train --gpu $gpu exp_root_dir=output/ seed=$seed system.prompt_processor.prompt="a flower"
+# python launch.py --config configs/fourdfy_stage_1_low_vram.yaml --train --gpu 0 exp_root_dir=output/ seed=0 system.prompt_processor.prompt="a flower"
 
 ## Stage 2
 # ckpt=output/fourdfy_stage_1_low_vram/a_flower@timestamp/ckpts/last.ckpt
-# python launch.py --config configs/fourdfy_stage_2_low_vram.yaml --train --gpu $gpu exp_root_dir=output/ seed=$seed system.prompt_processor.prompt="a flower" system.weights=$ckpt
+# python launch.py --config configs/fourdfy_stage_2_low_vram.yaml --train --gpu 0 exp_root_dir=output/ seed=0 system.prompt_processor.prompt="a flower" system.weights=$ckpt
 
 ## Post-Process. Convert to mesh file.
-# python launch.py --config output/fourdfy_stage_2_low_vram/a_flower@timestamp/configs/parsed.yaml --export --gpu $gpu \
+# python launch.py --config output/fourdfy_stage_2_low_vram/a_flower@timestamp/configs/parsed.yaml --export --gpu 0 \
 #   resume=output/fourdfy_stage_2_low_vram/a_flower@timestamp/ckpts/last.ckpt system.exporter_type=mesh-exporter \
 #   system.exporter.context_type=cuda system.exporter.fmt=obj
 ## saved to output/fourdfy_stage_2_low_vram/a_flower@timestamp/save/iterations-export/
 
 ## Convert to point cloud.
 # cd /path_to_Comp4D/
 # python mesh2ply_8w.py /path_to_4dfy/output/fourdfy_stage_2_low_vram/a_flower@timestamp/save/iterations-export/model.obj data/a_flower.ply
-
 ```