New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: copy_if failed to synchronize: the launch timed out and was terminated #17

Open

jaeyun95 opened this issue Mar 20, 2019 · 1 comment

jaeyun95 commented Mar 20, 2019 •

edited

Loading

i have a problem like this:
RuntimeError: copy_if failed to synchronize: the launch timed out and was terminated

Traceback (most recent call last):
File "train.py", line 125, in
loss = output_dict['loss'].mean() + output_dict['cnn_regularization_loss'].mean()
File "/home/ailab/anaconda2/envs/r2c/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/ailab/r2c/models/multiatt/model.py", line 156, in forward
obj_reps = self.detector(images=images, boxes=boxes, box_mask=box_mask, classes=objects, segms=segms)
File "/home/ailab/anaconda2/envs/r2c/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call**
result = self.forward(*input, **kwargs)
File "/home/ailab/r2c/utils/detector.py", line 112, in forward
box_inds = box_mask.nonzero()
RuntimeError: copy_if failed to synchronize: the launch timed out and was terminated

When I was training this model, it is stop !
what can i do?

env
titan X
cuda 9.0

Owner

rowanz commented Mar 25, 2019

Sorry, i've never encountered this error while training the model a variety of setups.

There could be a few things going on. one thing to double check - are you using my exact environment with pytorch 1.0?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment