-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question about Lion #1198
Comments
Thanks for the data point. Can you please tell us your baseline settings? Like which optimizer, which learning rate and weight decay? I think for instability problem, you can either reduce the learning rate a little bit or try to lower the beta2, for example, we used (0.95, 0.98) in our llm training. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thank your for your paper and work, but I meet some wired problem in training. Can you help me?
![image](https://private-user-images.githubusercontent.com/19380638/256708420-1ed88f72-0f24-49ef-a6f1-748cf6f7645a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAxMTY5NzYsIm5iZiI6MTcyMDExNjY3NiwicGF0aCI6Ii8xOTM4MDYzOC8yNTY3MDg0MjAtMWVkODhmNzItMGYyNC00OWVmLWE2ZjEtNzQ4Y2Y2Zjc2NDVhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzA0VDE4MTExNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNkMGFjYmQ0OTNhNTFmMGQzY2Y4ZDMyNzczZTU4ZTQ1YzRkN2E3YjZmNDk0ZTVlZmQzMDExNDQ3MjY1ZTk1ZTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.liLa-9IPykFGnGXLvKG7TRaQQHqQTCFLoeYocyGL81E)
![image](https://private-user-images.githubusercontent.com/19380638/256708691-6f580d1a-82a5-4fd8-83df-ad9b0807ff96.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAxMTY5NzYsIm5iZiI6MTcyMDExNjY3NiwicGF0aCI6Ii8xOTM4MDYzOC8yNTY3MDg2OTEtNmY1ODBkMWEtODJhNS00ZmQ4LTgzZGYtYWQ5YjA4MDdmZjk2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzA0VDE4MTExNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTk3ZWIzZGZjMTZhNjU1ODEwY2Y4MTJjYjQyZjBiNWQ0NDZjZWJkYmE4MzY2NzIyMDI5MzcyY2NjMzNkYTA0MjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Ix4Vr_LdX75hcGN9s095zcKl5Jm3VARm7i6wmNywZdc)
It's my training process curve. Notice Lion will increase loss sometimes but decease later. It seems not stability? Can you give me some advises how to tune Lion's parameters?
Below is my setting:
Batchsize = 8, GPU = 64 A100, with syncBN
Network: PointPillars
Optimizer setting:
The text was updated successfully, but these errors were encountered: