Error[1]: [executionContext.cpp::handleTrainStationRunnerPhase1::146] Error Code 1: Cuda Runtime (operation not permitted when stream is capturing)

[lix19937]

Description

When I build trt plan by trtexec --useCudaGraph --onnx=test.onnx --verbose --fp16 , it raise follow error:

Error[1]: [executionContext.cpp::handleTrainStationRunnerPhase1::146] Error Code 1: Cuda Runtime (operation not permitted when stream is capturing)


[05/25/2023-20:44:13] [V] Latency: the summation of H2D Latency, GPU Compute Time, and D2H Latency. This is the latency to infer a single query.
[05/25/2023-20:44:13] [I] 
[05/25/2023-20:44:13] [E] Error[1]: [executionContext.cpp::handleTrainStationRunnerPhase1::146] Error Code 1: Cuda Runtime (operation not permitted when stream is capturing)
[05/25/2023-20:44:13] [W] The CUDA graph capture on the stream has failed.
[05/25/2023-20:44:13] [W] The built TensorRT engine contains operations that are not permitted under CUDA graph capture mode.
[05/25/2023-20:44:13] [W] The specified --useCudaGraph flag has been ignored. The inference will be launched without using CUDA graph launch.

If I shut off Cuda graphs in trtexec cmd, it no any errors. From cuda doc, I find all CUDA objects need to be initialized and on the GPU prior to recording a graph, it is not allowed to create or transfer a cuda object inside a recorded graph. For onnx graph to plan,
What specific situations( op) can cause this error ? Thanks.

Environment

TensorRT Version:v8510

NVIDIA GPU:Orin

[zerollzeng]

dup of #2997

[lix19937]

@zerollzeng Thanks !