training on maskrcnn

[chaibosong]

we training on maskrcnn, have this
question

2022-05-08 17:13:41,136 maskrcnn_benchmark.trainer INFO: Start training
/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [25,0,0] Assertion t >= 0 && t < n_classes failed.
Traceback (most recent call last):
File "./tools/train_net.py", line 209, in
main()
File "./tools/train_net.py", line 201, in main
use_tensorboard=args.use_tensorboard
File "./tools/train_net.py", line 95, in train
tb_logger=tb_logger
File "/media/wly/0000678400004823/syj毕业设计/0maskrcnnnew/maskrcnn_benchmark/engine/trainer.py", line 67, in do_train
loss_dict = model(images, targets)
File "/home/wly/anaconda3/envs/maskrcnn/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/media/wly/0000678400004823/syj毕业设计/0maskrcnnnew/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 71, in forward
x, result, detector_losses = self.roi_heads(features, proposals, targets)
File "/home/wly/anaconda3/envs/maskrcnn/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/media/wly/0000678400004823/syj毕业设计/0maskrcnnnew/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py", line 23, in forward
x, detections, loss_box = self.box(features, proposals, targets)
File "/home/wly/anaconda3/envs/maskrcnn/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/media/wly/0000678400004823/syj毕业设计/0maskrcnnnew/maskrcnn_benchmark/modeling/roi_heads/box_head/box_head.py", line 55, in forward
[class_logits], [box_regression]
File "/media/wly/0000678400004823/syj毕业设计/0maskrcnnnew/maskrcnn_benchmark/modeling/roi_heads/box_head/loss.py", line 144, in call
sampled_pos_inds_subset = torch.nonzero(labels > 0).squeeze(1)
RuntimeError: copy_if failed to synchronize: device-side assert triggered