Assertion failure in TensorRT inference

[oranfire]

Environment

TensorRT Version: 8.5.1.7

NVIDIA GPU: GeForce RTX 3060

NVIDIA Driver Version: 530.41.03

CUDA Version: 12.1

CUDNN Version: 8.9.5

Operating System: Ubuntu 18.04.6 LTS

Python Version (if applicable): 3.9.19

PyTorch Version (if applicable): 2.2.2

Baremetal or Container (if so, version): Baremetal

Description

I convert a pytorch model to onnx model by using the onnx package, and then convert the onnx model to a tensort model (.engine). The tensorRT model works well when I use the trtexec command to run it, but it fails and reports the error message as follows when I try to run it in my C++ code:

[04/23/2024-10:44:44] [V] [TRT] Total per-runner device persistent memory is 12564992
[04/23/2024-10:44:44] [V] [TRT] Total per-runner host persistent memory is 58320
[04/23/2024-10:44:44] [V] [TRT] Allocated activation device memory of size 698434560
[04/23/2024-10:44:45] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +678, now: CPU 0, GPU 694 (MiB)
[04/23/2024-10:44:45] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
480, 752 116
[04/23/2024-10:44:45] [E] [TRT] 2: [runner.cpp::getGraphIndex::354] Error Code 2: Internal Error (Assertion static_cast<size_t>(index) < mBinaries.size() failed. )

The model, trtexec test report and the C++ code to run the model are uploaded to the Google Drive as follows:

Model link: https://drive.google.com/file/d/1WPAcFxdTycIFlfIHFam2ihoCcvvCi38m/view?usp=sharing
Trtexec test report link: https://drive.google.com/file/d/1rNLlWlymsSCw1xxoU4GyqrChRaXD1dL3/view?usp=sharing
C++ code link: https://drive.google.com/file/d/1FcU29OsZl2iR0KTJwwtqpdSi_ML2WiOd/view?usp=sharing. I first call the function SuperPointV2::build to deserialize the model, and then call the function SuperPointV2::infer() to run the model. The two inputs have dims of [1, 1, 480, 752] and [1, 130, 2, 2] respectively in my C++ test.

[lix19937]

The tensorRT model works well when I use the trtexec command to run it, but it fails and reports the error message as follows when I try to run it in my C++ code.

Your code should be an error. trtexec/samples is OSS, you can compare.

[zerollzeng]

+1

[oranfire]

The problem is dynamic output shape depended on the input value. Issue close.