Releases: kozistr/pytorch_optimizer
pytorch-optimizer v3.1.2
pytorch-optimizer v3.1.1
pytorch-optimizer v3.1.0
Change Log
Feature
- Implement
AdaLomo
optimizer. (#258) - Support
Q-GaLore
optimizer. (#258)- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.
- you can use by
optimizer = load_optimizer('q_galore_adamw8bit')
- Support more bnb optimizers. (#258)
bnb_paged_adam8bit
,bnb_paged_adamw8bit
,bnb_*_*32bit
.
- Improve
power_iteration()
speed up to 40%. (#259) - Improve
reg_noise()
(E-MCMC) speed up to 120%. (#260) - Support
disable_lr_scheduler
parameter forRanger21
optimizer to disable built-in learning rate scheduler. (#261)
Refactor
- Refactor
AdamMini
optimizer. (#258) - Deprecate optional dependency,
bitsandbytes
. (#258) - Move
get_rms
,approximate_sq_grad
functions toBaseOptimizer
for reusability. (#258) - Refactor
shampoo_utils.py
. (#259) - Add
debias
,debias_adam
methods inBaseOptimizer
. (#261) - Refactor to use
BaseOptimizer
only, not inherit multiple classes. (#261)
Bug
- Fix several bugs in
AdamMini
optimizer. (#257)
Contributions
thanks to @sdbds
pytorch-optimizer v3.0.2
pytorch-optimizer v3.0.1
Change Log
Feature
- Implement
FAdam
optimizer. (#241, #242) - Tweak
AdaFactor
optimizer. (#236, #243)- support not-using-first-momentum when beta1 is not given
- default dtype for first momentum to
bfloat16
- clip second momentum to 0.999
- Implement
GrokFast
optimizer. (#244, #245)
Bug
- Wrong typing of reg_noise. (#239, #240)
- Lookahead`s param_groups attribute is not loaded from checkpoint. (#237, #238)
Contributions
thanks to @michaldyczko
pytorch-optimizer v3.0.0
Change Log
The major version is updated! (v2.12.0
-> v3.0.0
) (#164)
Many optimizers, learning rate schedulers, and objective functions are in pytorch-optimizer
.
Currently, pytorch-optimizer
supports 67 optimizers (+ bitsandbytes
), 11 lr schedulers, and 13 loss functions, and reached about 4 ~ 50K downloads / month (peak is 75K downloads / month)!
The reason for updating the major version from v2
to v3
is that I think it's a good time to ship the recent implementations (the last update was about 7 months ago) and plan to pivot to new concepts like training utilities while maintaining the original features (e.g. optimizers).
Also, rich test cases, benchmarks, and examples are on the list!
Finally, thanks for using the pytorch-optimizer
, and feel free to make any requests :)
Feature
- Implement
REX
lr scheduler. (#217, #222) - Implement
Aida
optimizer. (#220, #221) - Implement
WSAM
optimizer. (#213, #216) - Implement
GaLore
optimizer. (#224, #228) - Implement
Adalite
optimizer. (#225, #229) - Implement
bSAM
optimizer. (#212, #233) - Implement
Schedule-Free
optimizer. (#230, #233) - Implement
EMCMC
. (#231, #233)
Fix
- Fix SRMM to allow operation beyond memory_length. (#227)
Dependency
- Drop
Python 3.7
support officially. (#221)- Please check the README.
- Update
bitsandbytes
to0.43.0
. (#228)
Docs
- Add missing parameters in
Ranger21 optimizer
document. (#214, #215) - Fix
WSAM
optimizer paper link. (#219)
Contributions
Diff
- from the previous major version : 2.0.0...3.0.0
- from the previous version: 2.12.0...3.0.0
pytorch-optimizer v2.12.0
Change Log
Feature
- Support
bitsandbytes
optimizer. (#211)- now, you can install with
pip3 install pytorch-optimizer[bitsandbytes]
- supports 8 bnb optimizers.
bnb_adagrad8bit
,bnb_adam8bit
,bnb_adamw8bit
,bnb_lion8bit
,bnb_lamb8bit
,bnb_lars8bit
,bnb_rmsprop8bit
,bnb_sgd8bit
.
- now, you can install with
Docs
- Introduce
mkdocs
withmaterial
theme. (#204, #206)- documentation : https://pytorch-optimizers.readthedocs.io/en/latest/
Diff
pytorch-optimizer v2.11.2
Change Log
Feature
- Implement DAdaptLion optimizer (#203)
Fix
- Fix Lookahead optimizer (#200, #201, #202)
- When using PyTorch Lightning which expects your optimiser to be a subclass of
Optimizer
.
- When using PyTorch Lightning which expects your optimiser to be a subclass of
- Fix default
rectify
toFalse
inAdaBelief
optimizer (#203)
Test
- Add
DynamicLossScaler
test case
Docs
- Highlight the code blocks
- Fix pepy badges
Contributions
thanks to @georg-wolflein
Diff
pytorch-optimizer v2.11.1
Change Log
Feature
- Implement Tiger optimizer (#192)
- Implement CAME optimizer (#196)
- Implement loss functions (#198)
- Tversky Loss : Tversky loss function for image segmentation using 3D fully convolutional deep networks
- Focal Tversky Loss
- Lovasz Hinge Loss : The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks
Diff
pytorch-optimizer v2.11.0
Change Log
Feature
- Implement PAdam optimizer (#186)
- Implement LOMO optimizer (#188)
- Implement loss functions (#189)
- BCELoss
- BCEFocalLoss
- FocalLoss : Focal Loss for Dense Object Detection
- FocalCosineLoss : Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble
- DiceLoss : Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
- LDAMLoss : Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
- JaccardLoss
- BiTemperedLogisticLoss : Robust Bi-Tempered Logistic Loss Based on Bregman Divergences