Load and unload different LORAs during training

[antoni4040]

Hello. I have a complex experiment I want to run. Let's say I have 20 different LORAs I want to train. I want to dynamically load and unload a few of them (as well as save each one to a file) during training. For example, I use 4 of them for the first 100 steps, then I want to save them and switch to a different set of 4 etc. The choice of which LORA is done dynamically from the input. The concept is not very complex, but I want to use HuggingFaces trainers (like SFTTrainer) and I find the documentation a bit difficult. Thank you!

[BenjaminBossan]

I'm not sure how well this could be implemented with SFTTrainer out of the box, I think you will have to subclass it and add your custom logic to the right method. Since it is a subclass of the Trainer class, you can read this doc to find the right place to hook into.

A tip: When creating the optimizer, ensure that the weights of all LoRA adapters are included. When you have disabled adapters, their requires_grad is False and the Trainer will not add their parameters to the optimizer. Then, those adapters will never be updated, even if you activate them. Therefore, it might be a good idea to create an optimizer manually with all parameters and pass that to the Trainer class.

[antoni4040]

Thank you. Is there an easy way to load and save the weights for the LORA adapters only?

[BenjaminBossan]

In general, when you call save_pretrained on a PEFT model, it should only save the adapter weights.