CUDA_VISIBLE_DEVICES=0,1 python test.py F0223 08:15:30.722712 2135 syncedmem.cpp:19] Check failed: error == cudaSuccess (29 vs. 0) driver shutting down *** Check failure stack trace: *** Aborted (core dumped)

[li0128]

CUDA_VISIBLE_DEVICES=0,1 python test.py

F0223 08:15:30.722712 2135 syncedmem.cpp:19] Check failed: error == cudaSuccess (29 vs. 0) driver shutting down
*** Check failure stack trace: ***
Aborted (core dumped)

[fyangneil]

@li0128 might be the problem of cuda version, please check the open or closed issue, there are someone have the same problem

[bolongpa]

I meet the same problem. I essentially just increased the base learning rate in solver_fphb_crack.prototxt and the system throws out the error above. In my experiments, the error appears only when base learning rate is higher than 1e-6, while it works well when I try to use 1e-7 to 1e-9. I searched for solutions but I didn't find a feasible one.

[bolongpa]

could you please clarify what this mean in solver_fphb_crack.prototxt?

lr for fine-tuning should be lower than when starting from scratch