-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
训练时报错 OSError: (External) CUDA error(719), unspecified launch failure. #3137
Comments
class_gt.txt 生成的数据有写问题,但我已经修复了 |
(PaddleClas) F:\code\PaddleClas>pip list anyio 4.3.0 ··························· |
@Sunting78 能帮助下吗? |
修复完class_gt.txt之后报什么错呢 |
和原来一样,找个问题跟class_gt报错没关系 |
现在的你的机器的共享内存有多大呢 |
你把numworker改为0看看还有问题吗? |
你好, 问题解决了么, 我也出现同样的问题了,数据集用的是百度推荐的数据集, 但是也会报错 |
欢迎您使用PaddleClas并反馈相关问题,非常感谢您对PaddleClas的贡献!
提出issue时,辛苦您提供以下信息,方便我们快速定位问题并及时有效地解决您的问题:
a. 具体操作系统,Windows10
b. Python版本号,Python38
c. CUDA/cuDNN版本, CUDA11.7
Error: C:\home\workspace\Paddle\paddle\phi\kernels\gpu\cross_entropy_kernel.cu:1010 Assertion
false
failed. The value of label expected >= 0 and < 7, or == -100, but got 29. Please check label value.Error: C:\home\workspace\Paddle\paddle\phi\kernels\gpu\cross_entropy_kernel.cu:1010 Assertion
false
failed. The value of label expected >= 0 and < 7, or == -100, but got 29. Please check label value.Error: C:\home\workspace\Paddle\paddle\phi\kernels\gpu\cross_entropy_kernel.cu:1010 Assertion
false
failed. The value of label expected >= 0 and < 7, or == -100, but got 29. Please check label value.Traceback (most recent call last):
File "tools/train.py", line 32, in
engine.train()
File "F:\code\PaddleClas\ppcls\engine\engine.py", line 339, in train
self.train_epoch_func(self, epoch_id, print_batch_step)
File "F:\code\PaddleClas\ppcls\engine\train\train.py", line 54, in train_epoch
loss_dict = engine.train_loss_func(out, batch[1])
File "F:\code\PaddleClas\ppcls\loss_init_.py", line 58, in call
loss = self.loss_func[0](input, batch)
File "D:\anaconda\envs\PaddleClas\lib\site-packages\paddle\nn\layer\layers.py", line 1254, in call
return self.forward(*inputs, **kwargs)
File "F:\code\PaddleClas\ppcls\loss\celoss.py", line 57, in forward
loss = F.cross_entropy(x, label=label, soft_label=soft_label)
File "D:\anaconda\envs\PaddleClas\lib\site-packages\paddle\nn\functional\loss.py", line 2790, in cross_entropy
if paddle.count_nonzero(is_ignore) > 0: # ignore label
File "D:\anaconda\envs\PaddleClas\lib\site-packages\paddle\fluid\dygraph\tensor_patch_methods.py", line 673, in bool
return self.nonzero()
File "D:\anaconda\envs\PaddleClas\lib\site-packages\paddle\fluid\dygraph\tensor_patch_methods.py", line 670, in nonzero
return bool(np.array(self) > 0)
File "D:\anaconda\envs\PaddleClas\lib\site-packages\paddle\fluid\dygraph\tensor_patch_methods.py", line 696, in array
array = self.numpy(False)
OSError: (External) CUDA error(719), unspecified launch failure.
[Hint: 'cudaErrorLaunchFailure'. An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointerand accessing out of bounds shared memory. L
ess common cases can be system specific - more information about these cases canbe found in the system specific user guide. This leaves the process in an inconsistent state and any further CUDA work willreturn the same error. To continue using CUDA, the process must be terminated and relaunched.] (at ..\paddle\phi\backends\gpu\cuda\cuda_info.cc:267)
········
上面是报错信息
我得cuda环境是没问题的,训练OCR都可以。下面是配置文件与分类文件
······
class_gt.txt
PPLCNet_x1_0_search.txt
The text was updated successfully, but these errors were encountered: