You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, I am wondering if anyone else has gotten this THCudaCheck Fail error while running this. I am running cuda 12.2v with an NVIDIA RTX A3000 Laptop GPU. Please let me know if you figure anything out.
Here is my full output after running:
07-19 23:21:43.075 train fine cities: ['train/aachen', 'train/bochum', 'train/bremen', 'train/cologne', 'train/darmstadt', 'train/dusseldorf', 'train/erfurt', 'train/hamburg', 'train/hanover', 'train/jena', 'train/krefeld', 'train/monchengladbach', 'train/strasbourg', 'train/stuttgart', 'train/tubingen', 'train/ulm', 'train/weimar', 'train/zurich']
07-19 23:21:43.083 Cityscapes-train: 2975 images
07-19 23:21:43.083 val fine cities: ['val/frankfurt', 'val/lindau', 'val/munster']
07-19 23:21:43.085 Cityscapes-val: 500 images
07-19 23:21:43.085 Using Per Image based weighted loss
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/loss.py:216: UserWarning: NLLLoss2d has been deprecated. Please use NLLLoss instead as a drop-in replacement and see https://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss for more details.
warnings.warn("NLLLoss2d has been deprecated. "
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.
warnings.warn(warning.format(ret))
07-19 23:21:43.085 Using Cross Entropy Loss
Creating wide resnet
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/encoding/nn/syncbn.py:149: EncodingDeprecationWarning: encoding.nn.BatchNorm2d is now deprecated in favor of encoding.nn.SyncBatchNorm.
.format('BatchNorm2d', SyncBatchNorm.name), EncodingDeprecationWarning)
/home/nrose171/Documents/GitHub/GSCNN/network/mynn.py:29: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal.
nn.init.kaiming_normal(module.weight)
07-19 23:21:44.878 Model params = 32.2M
07-19 23:21:44.895 Loading weights from model checkpoints/best_cityscapes_checkpoint.pth
07-19 23:21:45.211 Load Compelete
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=8 : invalid device function
Traceback (most recent call last):
File "train.py", line 383, in
main()
File "train.py", line 142, in main
optim, default_eval_epoch, writer)
File "train.py", line 302, in validate
seg_out, edge_out = net(input) # output = (1, 19, 713, 713)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/Documents/GitHub/GSCNN/network/gscnn.py", line 272, in forward
m1 = self.mod1(inp)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (8) : invalid device function at /pytorch/aten/src/THC/THCGeneral.cpp:383
The text was updated successfully, but these errors were encountered:
Hello, I am wondering if anyone else has gotten this THCudaCheck Fail error while running this. I am running cuda 12.2v with an NVIDIA RTX A3000 Laptop GPU. Please let me know if you figure anything out.
Here is my full output after running:
07-19 23:21:43.075 train fine cities: ['train/aachen', 'train/bochum', 'train/bremen', 'train/cologne', 'train/darmstadt', 'train/dusseldorf', 'train/erfurt', 'train/hamburg', 'train/hanover', 'train/jena', 'train/krefeld', 'train/monchengladbach', 'train/strasbourg', 'train/stuttgart', 'train/tubingen', 'train/ulm', 'train/weimar', 'train/zurich']
07-19 23:21:43.083 Cityscapes-train: 2975 images
07-19 23:21:43.083 val fine cities: ['val/frankfurt', 'val/lindau', 'val/munster']
07-19 23:21:43.085 Cityscapes-val: 500 images
07-19 23:21:43.085 Using Per Image based weighted loss
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/loss.py:216: UserWarning: NLLLoss2d has been deprecated. Please use NLLLoss instead as a drop-in replacement and see https://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss for more details.
warnings.warn("NLLLoss2d has been deprecated. "
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.
warnings.warn(warning.format(ret))
07-19 23:21:43.085 Using Cross Entropy Loss
Creating wide resnet
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/encoding/nn/syncbn.py:149: EncodingDeprecationWarning: encoding.nn.BatchNorm2d is now deprecated in favor of encoding.nn.SyncBatchNorm.
.format('BatchNorm2d', SyncBatchNorm.name), EncodingDeprecationWarning)
/home/nrose171/Documents/GitHub/GSCNN/network/mynn.py:29: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal.
nn.init.kaiming_normal(module.weight)
07-19 23:21:44.878 Model params = 32.2M
07-19 23:21:44.895 Loading weights from model checkpoints/best_cityscapes_checkpoint.pth
07-19 23:21:45.211 Load Compelete
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=8 : invalid device function
Traceback (most recent call last):
File "train.py", line 383, in
main()
File "train.py", line 142, in main
optim, default_eval_epoch, writer)
File "train.py", line 302, in validate
seg_out, edge_out = net(input) # output = (1, 19, 713, 713)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/Documents/GitHub/GSCNN/network/gscnn.py", line 272, in forward
m1 = self.mod1(inp)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (8) : invalid device function at /pytorch/aten/src/THC/THCGeneral.cpp:383
The text was updated successfully, but these errors were encountered: