Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

预训练阶段段loss下降后升高了，模型训崩了 #505

Open
liuheng0111 opened this issue Jul 8, 2024 · 2 comments
Open

预训练阶段段loss下降后升高了，模型训崩了 #505

liuheng0111 opened this issue Jul 8, 2024 · 2 comments

Comments

@liuheng0111
Copy link

在第一个阶段使用laion-2B的caption训练数据，放开VIT，mlp projector，vision export训练，freeze大语言模型进行训练，训练过程中loss先慢慢下降，但后面升高了，升高之后发现模型训崩了，已排除了训练数据问题，learning rate也调小了都不行，请问是哪里的问题？
image
@mactavish91
Copy link
Member

lr设置的多少呢

@liuheng0111
Copy link
Author

lr设置的多少呢

lr设置的1e-5, 1e-6都是过，loss都是先下降后上升

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@liuheng0111 @mactavish91