Can we reduce the rank of a high-rank LoRA?

[sayakpaul]

Can we lower the rank of high-rank LoRA to reduce mem requirements, have LoRAs in the same rank to enable torch.compile()?

Turns out that we can! That too w/ good'ol random projections & SVD 📷

Original vs. Reduced Rank 🐶

Code, more results, detailed write up are in: https://huggingface.co/sayakpaul/lower-rank-flux-lora