How to enable multiple LoRA adapters?

[Jaja612]

Hi, thanks for this great project! I am wondering what should I revise to support one forward pass with multiple LoRA adapters. It seems to be a straightforward extension, but the current version doesn't support stack (activate) multiple LoRA adapters.

Any guidance and help would be appreciated, thanks! @calpt

[StephennFernandes]

hey would this collab notebook help: https://colab.research.google.com/github/Adapter-Hub/adapter-transformers/blob/master/notebooks/Parallel_Adapter_Inference.ipynb

[Jaja612]

Hi @StephennFernandes , thanks for the reply. However, what I need is more like "Stack" rather than this "Parallel" composition for LoRA, see this doc.
Let me clarify my question. Suppose we have a model M, and a LoRA adapter A, I want to first merge their weights, and then train another LoRA adapter B based on M+A. I think this merge property is one advantage for LoRA, but it seems this repo only support merge for inference, rather than training.

When I tried to activate two LoRA adapters A and B, and set only B as trainable, it can not perform the forward pass because LoRA doesn't support stack operation.

Any guidance about how to revise this package to support the above feature would be greatly appreciated, thanks!

[lenglaender]

Hi @Jaja612,

The LoRA paper suggests merging LoRA adapters to not change the inference time: "Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to a fully fine-tuned model, by construction." (Lora: Low-Rank Adaptation of Large Language Models by Hu et al.)

Other modular combinations, such as stacking, are thus not in line with the paper's original idea and thus we did not yet implement them.
To support stacking and similar approaches, you must change: https://github.com/adapter-hub/adapter-transformers/blob/master/src/transformers/adapters/lora.py
For the implementation, you can orientate yourself on the prefix tuning stacking:
https://github.com/adapter-hub/adapter-transformers/blob/master/src/transformers/adapters/prefix_tuning.py#L399-L464