[Precomputed ControlNet] Speed up ControlNet by 45% - but is it necessary?

[lllyasviel]

Hi everyone, we plan to experiment with a feature called "Precomputed ControlNet". This can be achieved by modifying 2 or 3 lines of the training code to progressively disconnect the input concat here:

By doing this, we will be able to execute controlnet only one time before diffusion, rather than in every diffusion iteration. The controlnet should be equally powerful and robust as before. This will lead to a speed up by about 40% to 45%. And this will further decrease the required GPU memory.

Nevertheless, if we can observe any performance decrease (even minimal decrease) in any experimental setting (including no prompt setting, short prompt setting, and long prompt setting), we will make this experimental feature on-hold to avoid confusing and to prevent giving new users a mis-estimate of the models' capabilities.

Let us know what you think about it! Thank you for your support as always.

Update (after all experiments of cn1.1): this experiment fails to train ControlNets as good as the proposed implementation in our paper. We observe that models trained with this method tends to produce more artifacts and less robust results. We have given up this feature. The input from each diffusion steps is necessary for robustness, and necessary for special models like Shuffle and IP2P (in controlnet 1.1).

[yakuzadave]

Great work so far. This has been a lot of fun to play with.

[axsddlr]

I am down to try it out

[OedoSoldier]

Good idea, afaik many ppl w/ low vram GPUs are struggling w/ using CN w/ highres fix so if it works it'll be great.

[TheLukaDragar]

Great which 3 lines did you change? I would like to try it.

[drbobo0]

looks awesome , keep it up

[Njasa2k]

Is this speedup competitive with T2I adapter?

[Aridea2021]

that's good news for me, 期待早点体验，目前gpu vram总是不够，计算也很慢

[neverix]

Wouldn't this let you use a much larger model

[Aridea2021]

autoreply：Your mail has been received, and I will reply as soon as I canBest Regards；） 

[dan4ik94]

Yes, it's really necessary. I have a 4GB GPU and I would benefit of this. Thank you for you hard work.

[xueqing0622]

Good idea, I need it very much,

[Georgefwt]

Could we use the released model to achieve this or should we retrain a new model to achieve this?
I tried to tweak the sampling steps in the way you described and used the released model, and it didn't work.😢

[ffdown]

we need it =))

[lioo717]

how to deal with the timesteps? Does it use T at the first step, and the T result for all following steps?

[Aridea2021]

autoreply：Your mail has been received, and I will reply as soon as I canBest Regards；） 

[lllyasviel]

Update: this experiment fails to train ControlNets as good as the standard implementation. We have given up this feature. The input from each diffusion steps is necessary for robustness, and necessary for special models like Shuffle and IP2P.

[Njasa2k]

can we see the images?

[Aridea2021]

autoreply：Your mail has been received, and I will reply as soon as I canBest Regards；） 

[karkra911]

can we use NAS (neural architecture search) for even more GPU efficiency ?

finding new neural architecture which is even much faster ?

does anyone have Nvidia dgx workstation to run NAS , and find a better and even more gpu efficient and fast algorithm ?

[Aridea2021]

autoreply：Your mail has been received, and I will reply as soon as I canBest Regards；）