Compatibility

[inferno46n2]

Hello,

So this require a uniquely trained video model to run with? Did your team also finetune a SVD model to use with this?

Confused what type of video diffusion models accepts these compressed latents.

[yzhang2016]

The tuned SVD model will be released soon. The original SVD can also be used, but its performance is worse than the tuned one.

[frankchieng]

how to generate more frames with SVD and CV-VAE?

[sijeh]

Hello,

So this require a uniquely trained video model to run with? Did your team also finetune a SVD model to use with this?

Confused what type of video diffusion models accepts these compressed latents.

Any downstream models obtained from SD1.5 and SD2.1, such as various image diffusion community models, as well as video models, such as SVD, Videocrafter, and Animatediff, can be used with CV-VAE. In addition, compatibility can be further enhanced by performing a small amount of fine-tuning on the diffusion model.

[sijeh]

The tuned SVD model will be released soon. The original SVD can also be used, but its performance is worse than the tuned one.

The inference code and model weights of SVD will be coming soon.

[radna0]

Is it possible to fine-tune Open-Sora or Open-Sora-PLan with CV-VAE, Have your team tried comparing this with SVD? @sijeh

[sijeh]

Is it possible to fine-tune Open-Sora or Open-Sora-PLan with CV-VAE, Have your team tried comparing this with SVD? @sijeh

Open-Sora and Open-Sora-Plan are initialized from pixelart-alpha, which also uses the SD2.1 VAE. However, they later trained their own video VAE, and the latent space is no longer compatible with SD2.1.