How to use Nemo To train/replicating WeightTying LoRA?

[yhyu13]

Hi

A recent paper https://arxiv.org/abs//2311.09578 released by NVIDIA, has used Nemo for implementing a novel LoRA method called WeightTying-Lora which significantly reduce LoRA params w/o sacrificing performance

The link mentioned in the paper is here https://github.com/NVIDIA/NeMo/commits/adithyare/vera, but there is no guideline for how to setup training script, and args, etc

Plus, I had not successfully found resources on using NeMO for LoRA training, so there is much confusion here.

Could anyone with LoRA experience pointing out how to use NeMO for LoRA or even weighttying LoRA?

Thanks!

[yhyu13]

Ok, found one here

https://github.com/NVIDIA/NeMo/blob/9c7926db4ae375b77dae7eb57656213de1dd76a5/tutorials/nlp/lora.ipynb

[yhyu13]

But this one is that helpful due to its being tied to Megatron gpts, not open source models like LLaMA. However, the original paper uses LLaMA in NeMO. I couldn't find one that introduce how to use LLaMA in NeMO.

[SKDDJ]

Have you managed to use NeMO to replicate the WeightTying LoRA? If so, please inform me.