Could not find any implementation for node {ForeignNode[971_57...Concat_510]}

[hch-baobei]

command：
python tools/deploy.py configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py mmcode/config/yolov3_d53_mstrain-608_273e_coco.py mmcode/checkpoints/yolov3_d53_fp16_mstrain-608_273e_coco.pth mmcode/demo/demo.jpg --device cuda:0

Environmental information
2022-08-05 16:26:17,800 - mmdeploy - INFO - sys.platform: linux
2022-08-05 16:26:17,801 - mmdeploy - INFO - Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
2022-08-05 16:26:17,801 - mmdeploy - INFO - CUDA available: True
2022-08-05 16:26:17,801 - mmdeploy - INFO - GPU 0,1: NVIDIA GeForce RTX 2080 Ti
2022-08-05 16:26:17,801 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2022-08-05 16:26:17,801 - mmdeploy - INFO - NVCC: Build cuda_11.4.r11.4/compiler.30521435_0
2022-08-05 16:26:17,801 - mmdeploy - INFO - GCC: gcc (Ubuntu 7.5.0-3ubuntu1~16.04) 7.5.0
2022-08-05 16:26:17,801 - mmdeploy - INFO - PyTorch: 1.8.2
2022-08-05 16:26:17,802 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

2022-08-05 16:26:17,802 - mmdeploy - INFO - TorchVision: 0.9.2
2022-08-05 16:26:17,802 - mmdeploy - INFO - OpenCV: 4.5.5
2022-08-05 16:26:17,802 - mmdeploy - INFO - MMCV: 1.4.0
2022-08-05 16:26:17,802 - mmdeploy - INFO - MMCV Compiler: GCC 7.3
2022-08-05 16:26:17,802 - mmdeploy - INFO - MMCV CUDA Compiler: 11.1
2022-08-05 16:26:17,802 - mmdeploy - INFO - MMDeploy: 0.6.0+ef56036
2022-08-05 16:26:17,802 - mmdeploy - INFO -

2022-08-05 16:26:17,802 - mmdeploy - INFO - Backend information
2022-08-05 16:26:18,244 - mmdeploy - INFO - onnxruntime: 1.8.1 ops_is_avaliable : True
2022-08-05 16:26:18,277 - mmdeploy - INFO - tensorrt: 8.2.3.0 ops_is_avaliable : True
2022-08-05 16:26:18,289 - mmdeploy - INFO - ncnn: None ops_is_avaliable : False
2022-08-05 16:26:18,291 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-08-05 16:26:18,304 - mmdeploy - INFO - openvino_is_avaliable: True
2022-08-05 16:26:18,316 - mmdeploy - INFO - snpe_is_available: False
2022-08-05 16:26:18,316 - mmdeploy - INFO -

2022-08-05 16:26:18,316 - mmdeploy - INFO - Codebase information
2022-08-05 16:26:18,316 - mmdeploy - INFO - mmdet: 2.25.1
2022-08-05 16:26:18,316 - mmdeploy - INFO - mmseg: None
2022-08-05 16:26:18,316 - mmdeploy - INFO - mmcls: None
2022-08-05 16:26:18,316 - mmdeploy - INFO - mmocr: None
2022-08-05 16:26:18,316 - mmdeploy - INFO - mmedit: None
2022-08-05 16:26:18,316 - mmdeploy - INFO - mmdet3d: None
2022-08-05 16:26:18,316 - mmdeploy - INFO - mmpose: None
2022-08-05 16:26:18,316 - mmdeploy - INFO - mmrotate: None

When I run the sample according to the document, convert Onnx success but Onnx appeared when converting tensorrt E 10: [optimizer. CPP: : computeCosts: : 2011] the Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[971_57...Concat_510]}.)

error log
2022-08-05 16:15:09,905 - mmdeploy - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
load checkpoint from local path: mmcode/checkpoints/yolov3_d53_fp16_mstrain-608_273e_coco.pth
/home/hzzt/.conda/envs/mmdeploy/lib/python3.7/site-packages/mmdet/datasets/utils.py:70: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
'data pipeline in your config file.', UserWarning)
2022-08-05 16:15:29,700 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
2022-08-05 16:15:30,547 - mmdeploy - INFO - Export PyTorch model to ONNX: /home/hzzt/project/mmdeploy/mmdeploy/end2end.onnx.
2022-08-05 16:15:31,402 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
/home/hzzt/project/mmdeploy/mmdeploy/mmdeploy/core/optimizers/function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
ys_shape = tuple(int(s) for s in ys.shape)
/home/hzzt/.conda/envs/mmdeploy/lib/python3.7/site-packages/torch/nn/functional.py:3458: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode)
/home/hzzt/.conda/envs/mmdeploy/lib/python3.7/site-packages/mmdet/models/dense_heads/yolo_head.py:127: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead
warnings.warn('DeprecationWarning: anchor_generator is deprecated, '
/home/hzzt/.conda/envs/mmdeploy/lib/python3.7/site-packages/mmdet/core/anchor/anchor_generator.py:333: UserWarning: grid_anchors would be deprecated soon. Please use grid_priors
warnings.warn('grid_anchors would be deprecated soon. '
/home/hzzt/.conda/envs/mmdeploy/lib/python3.7/site-packages/mmdet/core/anchor/anchor_generator.py:370: UserWarning: single_level_grid_anchors would be deprecated soon. Please use single_level_grid_priors
'single_level_grid_anchors would be deprecated soon. '
/home/hzzt/.conda/envs/mmdeploy/lib/python3.7/site-packages/mmdet/core/bbox/coder/yolo_bbox_coder.py:73: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert pred_bboxes.size(-1) == bboxes.size(-1) == 4
/home/hzzt/project/mmdeploy/mmdeploy/mmdeploy/pytorch/functions/topk.py:56: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if k > size:
/home/hzzt/project/mmdeploy/mmdeploy/mmdeploy/codebase/mmdet/core/post_processing/bbox_nms.py:259: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
dets, labels = TRTBatchedNMSop.apply(boxes, scores, int(scores.shape[-1]),
/home/hzzt/project/mmdeploy/mmdeploy/mmdeploy/mmcv/ops/nms.py:178: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
out_boxes = min(num_boxes, after_topk)
/home/hzzt/project/mmdeploy/mmdeploy/mmdeploy/mmcv/ops/nms.py:181: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
(batch_size, out_boxes)).to(scores.device))
2022-08-05 16:15:40,750 - mmdeploy - INFO - Execute onnx optimize passes.
2022-08-05 16:15:42,033 - mmdeploy - INFO - Finish pipeline mmdeploy.apis.pytorch2onnx.torch2onnx
2022-08-05 16:15:46,648 - mmdeploy - INFO - Start pipeline mmdeploy.backend.tensorrt.onnx2tensorrt.onnx2tensorrt in subprocess
2022-08-05 16:15:46,880 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /home/hzzt/project/mmdeploy/mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[08/05/2022-16:15:47] [TRT] [I] [MemUsageChange] Init CUDA: CPU +313, GPU +0, now: CPU 390, GPU 312 (MiB)
[08/05/2022-16:15:47] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 391 MiB, GPU 312 MiB
[08/05/2022-16:15:47] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 525 MiB, GPU 346 MiB
[08/05/2022-16:15:48] [TRT] [W] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/05/2022-16:15:48] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[08/05/2022-16:15:49] [TRT] [I] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
[08/05/2022-16:15:49] [TRT] [I] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace:
[08/05/2022-16:15:49] [TRT] [I] Successfully created plugin: TRTBatchedNMS
[08/05/2022-16:15:53] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.5 but loaded cuBLAS/cuBLAS LT 11.2.1
[08/05/2022-16:15:53] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +383, GPU +160, now: CPU 1420, GPU 506 (MiB)
[08/05/2022-16:15:54] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +432, GPU +172, now: CPU 1852, GPU 678 (MiB)
[08/05/2022-16:15:54] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[08/05/2022-16:16:32] [TRT] [I] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[08/05/2022-16:16:44] [TRT] [W] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[08/05/2022-16:16:44] [TRT] [W] (# 3 (SHAPE input))
[08/05/2022-16:16:44] [TRT] [W] (# 2 (SHAPE input))
[08/05/2022-16:16:44] [TRT] [W] (ONNX_RESIZE (+ (CEIL_DIV (+ (# 2 (SHAPE input)) -32) 32) 1) 2.000000e+00)
[08/05/2022-16:16:44] [TRT] [W] (ONNX_RESIZE (+ (CEIL_DIV (+ (# 3 (SHAPE input)) -32) 32) 1) 2.000000e+00)
[08/05/2022-16:16:44] [TRT] [W] (ONNX_RESIZE (ONNX_RESIZE (+ (CEIL_DIV (+ (# 3 (SHAPE input)) -32) 32) 1) 2.000000e+00) 2.000000e+00)
[08/05/2022-16:16:44] [TRT] [W] (ONNX_RESIZE (ONNX_RESIZE (+ (CEIL_DIV (+ (# 2 (SHAPE input)) -32) 32) 1) 2.000000e+00) 2.000000e+00)
[08/05/2022-16:16:45] [TRT] [W] Skipping tactic 0 due to insuficient memory on requested size of 2442276864 detected for tactic 0.
[08/05/2022-16:16:45] [TRT] [E] 10: [optimizer.cpp::computeCosts::2011] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[971_57...Concat_510]}.)

[hanrui1sensetime]

Hi, @hch-baobei, please try using TensorRT 8.4 GA version according to NVIDIA/TensorRT#1917.

[hch-baobei]

好的，这个我也看到了

[hch-baobei]

Hi, @hch-baobei, please try using TensorRT 8.4 GA version according to NVIDIA/TensorRT#1917.

I tried using TensorRT 8.4.2.1 and it worked.

[hch-baobei]

this is my environment information，just for your reference.
2022-08-09 15:52:52,159 - mmdeploy - INFO - Environmental information
2022-08-09 15:52:52,384 - mmdeploy - INFO - sys.platform: linux
2022-08-09 15:52:52,385 - mmdeploy - INFO - Python: 3.8.12 (default, Oct 12 2021, 13:49:34) [GCC 7.5.0]
2022-08-09 15:52:52,385 - mmdeploy - INFO - CUDA available: True
2022-08-09 15:52:52,385 - mmdeploy - INFO - GPU 0,1: NVIDIA GeForce RTX 2080 Ti
2022-08-09 15:52:52,385 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda-11.4
2022-08-09 15:52:52,385 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.4, V11.4.152
2022-08-09 15:52:52,385 - mmdeploy - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2022-08-09 15:52:52,386 - mmdeploy - INFO - PyTorch: 1.8.2
2022-08-09 15:52:52,386 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

2022-08-09 15:52:52,386 - mmdeploy - INFO - TorchVision: 0.9.2
2022-08-09 15:52:52,386 - mmdeploy - INFO - OpenCV: 4.5.5
2022-08-09 15:52:52,386 - mmdeploy - INFO - MMCV: 1.5.3
2022-08-09 15:52:52,386 - mmdeploy - INFO - MMCV Compiler: GCC 7.3
2022-08-09 15:52:52,386 - mmdeploy - INFO - MMCV CUDA Compiler: 11.1
2022-08-09 15:52:52,386 - mmdeploy - INFO - MMDeploy: 0.7.0+f957284
2022-08-09 15:52:52,386 - mmdeploy - INFO -

2022-08-09 15:52:52,386 - mmdeploy - INFO - Backend information
2022-08-09 15:52:52,660 - mmdeploy - INFO - onnxruntime: 1.8.1 ops_is_avaliable : True
2022-08-09 15:52:52,678 - mmdeploy - INFO - tensorrt: 8.4.2.4 ops_is_avaliable : True
2022-08-09 15:52:52,688 - mmdeploy - INFO - ncnn: None ops_is_avaliable : False
2022-08-09 15:52:52,688 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-08-09 15:52:52,699 - mmdeploy - INFO - openvino_is_avaliable: True
2022-08-09 15:52:52,708 - mmdeploy - INFO - snpe_is_available: False
2022-08-09 15:52:52,708 - mmdeploy - INFO -

2022-08-09 15:52:52,708 - mmdeploy - INFO - Codebase information
2022-08-09 15:52:52,709 - mmdeploy - INFO - mmdet: 2.25.1
2022-08-09 15:52:52,709 - mmdeploy - INFO - mmseg: None
2022-08-09 15:52:52,709 - mmdeploy - INFO - mmcls: 0.23.2
2022-08-09 15:52:52,709 - mmdeploy - INFO - mmocr: None
2022-08-09 15:52:52,709 - mmdeploy - INFO - mmedit: None
2022-08-09 15:52:52,709 - mmdeploy - INFO - mmdet3d: None
2022-08-09 15:52:52,709 - mmdeploy - INFO - mmpose: None
2022-08-09 15:52:52,709 - mmdeploy - INFO - mmrotate: None

[hanrui1sensetime]

Hi, @hch-baobei, please try using TensorRT 8.4 GA version according to NVIDIA/TensorRT#1917.

I tried using TensorRT 8.4.2.1 and it worked.

Ok, seems like a TensorRT bug.

[hanrui1sensetime]

I will close this issue. If you have any questions about it, please reopen this issue.