Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert trt fail #916

Closed
rylynchen opened this issue Aug 18, 2022 · 3 comments
Closed

Convert trt fail #916

rylynchen opened this issue Aug 18, 2022 · 3 comments
Assignees
@hanrui1sensetime

Comments

@rylynchen
Copy link

Run demo follow https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md

python tools/deploy.py \
    configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    /root/mmdetection/configs/yolo/yolov3_d53_mstrain-608_273e_coco.py \
    /root/yolov3_d53_mstrain-608_273e_coco.pth \
    /root/mmdetection/demo/demo.jpg \
    --work-dir work_dir \
    --show \
    --device cuda:0

get error :

root@4e713889a71f:~/mmdeploy# python tools/deploy.py \
>     configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
>     /root/mmdetection/configs/yolo/yolov3_d53_mstrain-608_273e_coco.py \
>     /root/yolov3_d53_mstrain-608_273e_coco.pth \
>     /root/mmdetection/demo/demo.jpg \
>     --work-dir work_dir \
>     --show \
>     --device cuda:0
[2022-08-18 03:48:24.469] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
[2022-08-18 03:48:25.593] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
[2022-08-18 03:48:26.758] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
2022-08-18 03:48:26,762 - mmdeploy - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
load checkpoint from local path: /root/yolov3_d53_mstrain-608_273e_coco.pth
/root/mmdetection/mmdet/datasets/utils.py:66: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
  warnings.warn(
2022-08-18 03:48:29,718 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
2022-08-18 03:48:29,718 - mmdeploy - INFO - Export PyTorch model to ONNX: work_dir/end2end.onnx.
2022-08-18 03:48:29,762 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_deduplicate_initializers, function rewrite will not be applied
/root/mmdeploy/mmdeploy/core/optimizers/function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
/root/mmdetection/mmdet/models/dense_heads/yolo_head.py:127: UserWarning: DeprecationWarning: `anchor_generator` is deprecated, please use "prior_generator" instead
  warnings.warn('DeprecationWarning: `anchor_generator` is deprecated, '
/root/mmdetection/mmdet/core/anchor/anchor_generator.py:333: UserWarning: ``grid_anchors`` would be deprecated soon. Please use ``grid_priors``
  warnings.warn('``grid_anchors`` would be deprecated soon. '
/root/mmdetection/mmdet/core/anchor/anchor_generator.py:369: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors``
  warnings.warn(
/root/mmdetection/mmdet/core/bbox/coder/yolo_bbox_coder.py:73: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(-1) == bboxes.size(-1) == 4
/root/mmdeploy/mmdeploy/pytorch/functions/topk.py:56: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if k > size:
/root/mmdeploy/mmdeploy/codebase/mmdet/core/post_processing/bbox_nms.py:259: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  dets, labels = TRTBatchedNMSop.apply(boxes, scores, int(scores.shape[-1]),
/root/mmdeploy/mmdeploy/mmcv/ops/nms.py:178: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  out_boxes = min(num_boxes, after_topk)
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
2022-08-18 03:48:36,849 - mmdeploy - INFO - Execute onnx optimize passes.
2022-08-18 03:48:37,779 - mmdeploy - INFO - Finish pipeline mmdeploy.apis.pytorch2onnx.torch2onnx
[2022-08-18 03:48:39.302] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
2022-08-18 03:48:39,306 - mmdeploy - INFO - Start pipeline mmdeploy.backend.tensorrt.onnx2tensorrt.onnx2tensorrt in subprocess
2022-08-18 03:48:39,442 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /root/mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[08/18/2022-03:48:39] [TRT] [I] [MemUsageChange] Init CUDA: CPU +312, GPU +0, now: CPU 393, GPU 261 (MiB)
[08/18/2022-03:48:40] [TRT] [W] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/18/2022-03:48:40] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[08/18/2022-03:48:41] [TRT] [I] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
[08/18/2022-03:48:41] [TRT] [I] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace:
[08/18/2022-03:48:41] [TRT] [I] Successfully created plugin: TRTBatchedNMS
[08/18/2022-03:48:41] [TRT] [I] [MemUsageSnapshot] Builder begin: CPU 985 MiB, GPU 289 MiB
[08/18/2022-03:48:42] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.5.1
[08/18/2022-03:48:42] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +484, GPU +206, now: CPU 1470, GPU 495 (MiB)
[08/18/2022-03:48:42] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +468, GPU +204, now: CPU 1938, GPU 699 (MiB)
[08/18/2022-03:48:42] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[08/18/2022-03:49:23] [TRT] [I] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[08/18/2022-03:49:33] [TRT] [W] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[08/18/2022-03:49:33] [TRT] [W]  (# 3 (SHAPE input))
[08/18/2022-03:49:33] [TRT] [W]  (ONNX_RESIZE (ONNX_RESIZE (+ (CEIL_DIV (+ (# 2 (SHAPE input)) -32) 32) 1) 2.000000e+00) 2.000000e+00)
[08/18/2022-03:49:33] [TRT] [W]  (ONNX_RESIZE (+ (CEIL_DIV (+ (# 2 (SHAPE input)) -32) 32) 1) 2.000000e+00)
[08/18/2022-03:49:33] [TRT] [W]  (ONNX_RESIZE (ONNX_RESIZE (+ (CEIL_DIV (+ (# 3 (SHAPE input)) -32) 32) 1) 2.000000e+00) 2.000000e+00)
[08/18/2022-03:49:33] [TRT] [W]  (ONNX_RESIZE (+ (CEIL_DIV (+ (# 3 (SHAPE input)) -32) 32) 1) 2.000000e+00)
[08/18/2022-03:49:33] [TRT] [W]  (# 2 (SHAPE input))
[08/18/2022-03:49:33] [TRT] [W] Skipping tactic 0 due to insuficient memory on requested size of 2442276864 detected for tactic 0.
[08/18/2022-03:49:33] [TRT] [E] 10: [optimizer.cpp::computeCosts::1981] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[onnx::Gather_971...Concat_506]}.)
Process Process-3:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/root/mmdeploy/mmdeploy/backend/tensorrt/onnx2tensorrt.py", line 79, in onnx2tensorrt
    from_onnx(
  File "/root/mmdeploy/mmdeploy/backend/tensorrt/utils.py", line 153, in from_onnx
    assert engine is not None, 'Failed to create TensorRT engine'
AssertionError: Failed to create TensorRT engine
2022-08-18 03:49:34,436 - mmdeploy - ERROR - `mmdeploy.backend.tensorrt.onnx2tensorrt.onnx2tensorrt` with Call id: 1 failed. exit.

Could you please tell me how to fix this problem? Thank you!
Env

ubuntu 18.04
cuda 11.4
TensorRT-8.2.0.6

And python tools/check_env.py

root@4e713889a71f:~/mmdeploy# python tools/check_env.py
2022-08-18 03:52:12,632 - mmdeploy - INFO -

2022-08-18 03:52:12,632 - mmdeploy - INFO - **********Environmental information**********
2022-08-18 03:52:12,787 - mmdeploy - INFO - sys.platform: linux
2022-08-18 03:52:12,787 - mmdeploy - INFO - Python: 3.8.12 (default, Oct 12 2021, 13:49:34) [GCC 7.5.0]
2022-08-18 03:52:12,787 - mmdeploy - INFO - CUDA available: True
2022-08-18 03:52:12,787 - mmdeploy - INFO - GPU 0: NVIDIA GeForce RTX 2070 SUPER
2022-08-18 03:52:12,787 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2022-08-18 03:52:12,787 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.3, V11.3.109
2022-08-18 03:52:12,787 - mmdeploy - INFO - GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
2022-08-18 03:52:12,787 - mmdeploy - INFO - PyTorch: 1.11.0
2022-08-18 03:52:12,787 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

2022-08-18 03:52:12,787 - mmdeploy - INFO - TorchVision: 0.12.0
2022-08-18 03:52:12,787 - mmdeploy - INFO - OpenCV: 4.6.0
2022-08-18 03:52:12,787 - mmdeploy - INFO - MMCV: 1.5.2
2022-08-18 03:52:12,787 - mmdeploy - INFO - MMCV Compiler: GCC 7.5
2022-08-18 03:52:12,787 - mmdeploy - INFO - MMCV CUDA Compiler: 11.3
2022-08-18 03:52:12,787 - mmdeploy - INFO - MMDeploy: 0.7.0+cad0092
2022-08-18 03:52:12,787 - mmdeploy - INFO -

2022-08-18 03:52:12,787 - mmdeploy - INFO - **********Backend information**********
2022-08-18 03:52:13,195 - mmdeploy - INFO - onnxruntime: 1.8.1	ops_is_avaliable : True
2022-08-18 03:52:13,218 - mmdeploy - INFO - tensorrt: 8.2.0.6	ops_is_avaliable : True
2022-08-18 03:52:13,231 - mmdeploy - INFO - ncnn: None	ops_is_avaliable : False
2022-08-18 03:52:13,231 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-08-18 03:52:13,232 - mmdeploy - INFO - openvino_is_avaliable: False
2022-08-18 03:52:13,244 - mmdeploy - INFO - snpe_is_available: False
2022-08-18 03:52:13,244 - mmdeploy - INFO -

2022-08-18 03:52:13,244 - mmdeploy - INFO - **********Codebase information**********
2022-08-18 03:52:13,245 - mmdeploy - INFO - mmdet:	2.25.1
2022-08-18 03:52:13,245 - mmdeploy - INFO - mmseg:	None
2022-08-18 03:52:13,245 - mmdeploy - INFO - mmcls:	None
2022-08-18 03:52:13,245 - mmdeploy - INFO - mmocr:	None
2022-08-18 03:52:13,245 - mmdeploy - INFO - mmedit:	None
2022-08-18 03:52:13,245 - mmdeploy - INFO - mmdet3d:	None
2022-08-18 03:52:13,245 - mmdeploy - INFO - mmpose:	None
2022-08-18 03:52:13,245 - mmdeploy - INFO - mmrotate:	None
@hanrui1sensetime
Copy link
Collaborator

The same as #869.
Please upgrade your TensorRT to the 8.4.x.

@hanrui1sensetime hanrui1sensetime self-assigned this Aug 18, 2022
@rylynchen
Copy link
Author

The same as #869. Please upgrade your TensorRT to the 8.4.x.

ok，thanks your reply. I will try soon.

@rylynchen
Copy link
Author

Upgrade TensorRT to 8.4, it work !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hanrui1sensetime hanrui1sensetime

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rylynchen @hanrui1sensetime