-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Request to add 4-bit AdamW #208
Comments
thanks for the request! it requires cuda files to compile like I'll figure it out how to smoothly support this in pytorch-optimizer :) thank you! |
Liked the idea of supporting it like the example. Maybe even easier, make the FR to bnb and include it here? Made the FR to bitsandbytes-foundation/bitsandbytes#775 |
that's a great idea! it'd be much better that bnb supports 4-bit optimizers. then, for now, wait for the response and decide whether including both or only bnb. |
Paper and Code
Paper: Memory Efficient Optimizers with 4-bit States
Code : https://github.com/thu-ml/low-bit-optimizers/blob/main/lpmm/optim/optimizer.py
The text was updated successfully, but these errors were encountered: