cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[Tclz]

Hi, i meet a problem like this:
File "train.py", line 131, in
output_dict = model(**batch)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "../models/multiatt/model.py", line 157, in forward
obj_reps = self.detector(images=images, boxes=boxes, box_mask=box_mask, classes=objects, segms=segms)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "../utils/detector.py", line 111, in forward
img_feats = self.backbone(images)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torchvision/models/resnet.py", line 98, in forward
out = self.conv2(out)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/envs/r2c_1/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

The environment i use:
python3.6.6
cuda9.0.176
cudnn7.5.1
torch1.1.0
torchvision0.3.0
and i have tried several environment configs(like cudnn7.4, torch1.0, etc) but none of them works.
what should i do?
thank you :)

[harukaza]

i also meet the problem, so i add two sentences in the train file.
import torch
torch.backends.cudnn.enabled = False

[Tclz]

@harukaza i fixed that after adjusting the environment :
cuda10.1
cudnn7.6.4
python3.7
pytorch1.3.1
torchvision0.4.1