GPU Memory-Usage, full finetuning vs LoRA vs LoRA + CPU Offloading

[EthanChen1234]

It's a great work!

hi, how to calculate the GPU Memory in different methods?

In addition, using LoRA method, the base parameter will go through forward pass, will the base parameter go through backward pass?

[younesbelkada]

Hi !
Thanks !
One way to calculate effectively memory stats is to call: torch.cuda.reset_peak_memory_stats(device) at the beginning of your script then call: max_memory = torch.cuda.max_memory_allocated(device)

In addition, using LoRA method, the base parameter will go through forward pass, will the base parameter go through backward pass?

Since backward pass is based on the chain rule, I would say so, however we only calculate gradients on the LoRA parameters

[EthanChen1234]

ok, thanks