ckpt in HuggingFace

[mo230761]

I really like your work！ But I found that the results using the HuggingFace version are not as good as when using the source code. Upon further exploration, I discovered that the HuggingFace version might not be using EMA. Is that the case? If so, how can I use the EMA version of the checkpoint with HuggingFace?

[drogozhang]

We were using EMA. Please see fine-tune.yaml

[mo230761]

Thank you for answering my questions！！ I would like to know how you convert a ckpt file into a format that can be used by Hugging Face.

[drogozhang]

That's actually a very good question haha. Amazing @vinesmsuic helped deploy it so I don't really know. Maybe @vinesmsuic can point us to somewhere?

[mo230761]

Thank you again for your answer！！！ And I would like to ask if @vinesmsuic could provide some guidance? If possible, I would be extremely grateful.

[vinesmsuic]

Hi @mo230761 , I don't exactly remembered how did I convert the weights back then since it has been a while. It could be possible that I missed the EMA option. But IIRC diffusers port are performing a bit worse, even for InstructPix2Pix.

Would you mind helping me to test the performance of this?
https://huggingface.co/vinesmsuic/magicbrush-ema

[mo230761]

Of course! I’ve also run some data using the EMA version you provided. Its results are indeed slightly worse than the original, but it’s better than the previous ported version that didn’t use EMA. It’s a pity that you don’t exactly remember how you converted the weights back then.

[vinesmsuic]

@mo230761
it would be great if you can share some visualizations.

[mo230761]

This might not be possible, as the data I’m testing is under a confidentiality agreement. My apologies.