-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ranger sign inversion #232
Labels
bug
Something isn't working
Comments
hi! sorry for the late reply the current Ranger implementation in this repository is based on here. (I might be wrong) I guess there's no big difference between the original implementation and mine. (let me know if I'm wrong) could you please test with the original implementation and reproduce the same result? thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
From my experiments it seems like the sign for the Ranger is inverted. All other optimizers (including Ranger21) has steps in the opposite direction of Ranger.
Note that I'm testing context-free step directions/magnitudes using a 'perfect' gradient (scaled by 4), so if Ranger somehow reverts course when gradients from different directions are accumulated that would be missed from my test.
Hyperparameters:
{'betas': (0.003344506587403595, 0.9685357345548955), 'lr': 0.4616639698903086}
(found through hyperparameter search, also done for the other optimizers) and evaluated on the Ackley (dim=2) function.(I didn't want to create a PR before discussing if this might be intended)
To Reproduce
Log
Ranger:
![image](https://private-user-images.githubusercontent.com/50617709/325239891-e607fff8-2264-4dfc-bab2-1293d0776837.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxMDkxMTAsIm5iZiI6MTcyNjEwODgxMCwicGF0aCI6Ii81MDYxNzcwOS8zMjUyMzk4OTEtZTYwN2ZmZjgtMjI2NC00ZGZjLWJhYjItMTI5M2QwNzc2ODM3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDAyNDAxMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTEzYzg5YmVhYzA2MzZkMjY4OTc4MDRkNzhmYjBhMzI0ZjhkOWUyODdkNmZhOGRiMGJmNzRhYjQ4YTg4YjFhNzcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.K9XPhA0WuWFoUz2lNjI17Sw3PtU3j8FBotK-5bP_agc)
For comparison SGD:
![image](https://private-user-images.githubusercontent.com/50617709/325240126-69e25fc7-3456-489e-bec2-7c7c2fdee7a1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxMDkxMTAsIm5iZiI6MTcyNjEwODgxMCwicGF0aCI6Ii81MDYxNzcwOS8zMjUyNDAxMjYtNjllMjVmYzctMzQ1Ni00ODllLWJlYzItN2M3YzJmZGVlN2ExLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDAyNDAxMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg3YzliNmZmMjZiYjcxNmY3ZDgwMWU0YmQwOTdlMGM3Y2Y5ODg4NDkyMjc1YzQ2ZWJiMGY5MjFkNmU0ZGE1ZTMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.UbJpvFtsotNle7ChLRTYprWpFe1j_QcV82IMPbSEJbQ)
The text was updated successfully, but these errors were encountered: