Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Understanding the forward operation #8

Open
evertonaleixo opened this issue Oct 3, 2023 · 1 comment
Open

Understanding the forward operation #8

evertonaleixo opened this issue Oct 3, 2023 · 1 comment

Comments

@evertonaleixo
Copy link

I noticed that you apply the mul operation in LoraA and LoraB, then, you sum the result with the input.

image

I think the result of multiplying LoraA and LoraB has to be summed to the original weights, or I am wrong?

Could you also explain the scaling factor?

Thanks.

@cccntu
Copy link
Owner

cccntu commented Oct 4, 2023

I noticed that you apply the mul operation in LoraA and LoraB, then, you sum the result with the input.
I think the result of multiplying LoraA and LoraB has to be summed to the original weights, or I am wrong?

This is the mechanism of torch.parametrizations
https://pytorch.org/tutorials/intermediate/parametrizations.html

Could you also explain the scaling factor?

scaling follows the original implementation https://github.com/microsoft/LoRA
It's mentioned in the paper. From my understanding it's not important, it's only there to control for the change of rank.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants