Lbfgs

[L-Xiafeng]

oneflow,strong_wolfe训练结果

num_epoch: 100, loss: 0.09013008

real    1m17.526s
user    2m24.450s
sys     0m50.515s

torch,strong_wolfe训练结果

num_epoch: 100, loss: 0.09780428

real    1m10.364s
user    1m9.138s
sys     0m5.576s

[levi131]

需要在oneflow代码仓中补充测试验证，分别验证使用strong wolfe和不使用strong wolfe的情况

[L-Xiafeng]

float32,1000epoch稳定性测试

框架	allow_tf32	allow_fp16_reduced	线搜索方法	RTX3090结果	RTX2080Ti结果
oneflow	False	False	None	OK	OK
oneflow	False	True	None	OK	OK
oneflow	True	False	None	会训飞	OK
oneflow	True	True	Nonel	会训飞	OK
oneflow	False	False	Strong_wolfe	OK	OK
oneflow	False	True	Strong_wolfe	OK	OK
oneflow	True	False	Strong_wolfe	loss卡在0.27	OK
oneflow	True	True	Strong_wolfe	loss卡在0.27	OK
torch	False	False	None	OK	OK
torch	False	True	None	OK	OK
torch	True	False	None	loss为nan	OK
torch	True	True	None	loss为nan	OK
torch	False	False	Strong_wolfe	loss卡死在0.094	loss卡死在0.102
torch	False	True	Strong_wolfe	loss卡死在0.094	loss卡死在0.102
torch	True	False	Strong_wolfe	loss卡死在1.70	loss卡死在0.102
torch	True	True	Strong_wolfe	loss卡死在1.70	loss卡死在0.102

[levi131]

LGTM