PEFT Lora transfer adapters between pipelines

[alejobrainz]

Hi. We are creating a system that makes inferences on the fly using different models. At present we are caching the pipeline.components which includes text encoders, unets and vae from different models and loading them on the fly by instancing a new pipeline based on the component cache stored in memory. I wanted to know if there is a similar way to load and cache a Lora adapter and apply it on the fly across different models without needing to load the weights on each cached model.

[BenjaminBossan]

Hi, I honestly don't fully understand what you're doing. Could you maybe show some (simplified) code that illustrates what you're doing?

In general, it is possible to merge the LoRA weights into the base model (merge_and_unload), so if it's always the same LoRA adapter and you don't need the base model for predictions, that could be an option for you.