novel-video-synthesis

Base mode

Open Source Video Generation Models to Checkout

• https://github.com/lichao-sun/Mora
• https://github.com/G-U-N/AnimateLCM
• https://github.com/ali-vilab/VGen
• https://github.com/TIGER-AI-Lab/ConsistI2V
• https://github.com/showlab/MotionDirector
• https://github.com/pixeli99/SVD_Xtend
• https://github.com/Picsart-AI-Research/StreamingT2V
• https://ecnu-cilab.github.io/ExVideoProjectPage/ (ExVideo is a post-tuning technique aimed at enhancing the capability of video generation models. We have extended Stable Video Diffusion to achieve the generation of long videos up to 128 frames. The)

Miscellaneous Video Generation Models

• https://github.com/mbzuai-oryx/Video-LLaVA

References & citations

Spatial & Temporal Transformer

• https://research.nvidia.com/labs/toronto-ai/VideoLDM/

https://huggingface.co/hotshotco/Hotshot-XL https://github.com/PKU-YuanGroup/Open-Sora-Plan

ControlNet

• https://github.com/lllyasviel/ControlNet

Distributive training

• https://huggingface.co/docs/accelerate/

https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/comfyanonymous/ComfyUI https://github.com/facebookresearch/xformers https://github.com/Dao-AILab/flash-attention

FiT: Flexible Vision Transformer for Diffusion Model

SOTA Caption Generation for video : https://github.com/snap-research/Panda-70M https://github.com/willisma/SiT

Unet

• https://huggingface.co/docs/diffusers/en/api/loaders/unet
• https://paperswithcode.com/method/u-net

https://github.com/Vchitect/Latte

VAE

• Consistency Distilled Diff VAE : https://github.com/openai/consistencydecoder

AnimateDiff + freeinit

• https://github.com/guoyww/animatediff/
• https://dagshub.com/ByteDance/AnimateDiff-Lightning
• https://github.com/TianxingWu/FreeInit