onnxruntime-linux-aarch64 with gpu support

[quangsonle]

I am trying to run a yolo-based model converted to Onnx format on Nvidia jetson nano. My code works well on a ubuntu 20 PC

the only thing i changed is, instead of onnxruntime-linux-x64-gpu-1.11.0 for the PC, i am using onnxruntime-linux-aarch64 for jetson.

It works but I got "GPU is not supported by your ONNXRuntime build. Fallback to CPU" and of course, very slow.

my question is, is there any version of gpu support for onnxruntime-linux-aarch64?

[andys-playground]

Seconded! Looking for onnxruntime-linux-arm64-gpu to take advantage of GPUs on aarch devices, notably Jetson devices from nvidia.

Their forums (https://forums.developer.nvidia.com/t/how-can-i-run-onnxruntime-c-api-in-jetson-os/201722/4) repeatedly refer to https://elinux.org/Jetson_Zoo#ONNX_Runtime, which is very python-oriented, not c/c++.

Is there a workaround? Do I need to build onnx from source on my target system?

[andys-playground]

I ended up solving this for my situation, at least for now.

References:

• Build on Jetson Xavier NX does not produce c++ API include and libs #9371
• https://onnxruntime.ai/docs/build/eps.html#nvidia-jetson-tx1tx2nanoxavier
• https://forums.developer.nvidia.com/t/how-can-i-run-onnxruntime-c-api-in-jetson-os/201722/5

I did the build on the Jetson Target (AGX Orin in my situation).

Assuming you have a default filesystem layout, steps to recreate:

1. clone onnxruntime repo
2. checkout the tagged version you want
3. init and update submodules recursively
4. Run their build.sh script with the following
   ./build.sh --config Release --update --build --parallel --build_wheel --build_shared_lib --use_tensorrt --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu/ --tensorrt_home /usr/lib/aarch64-linux-gnu/
5. Note, the build process will download more source, so internet access is needed.
6. Once the build process is complete, the Linux/Release/ folder will contain the .so files needed to link a c++ application, and can be copied out to an install destination folder.
7. The header files are in the repo, and can be copied out to an install destination folder

[csukuangfj]

I ended up solving this for my situation, at least for now.

References:

• Build on Jetson Xavier NX does not produce c++ API include and libs #9371
• https://onnxruntime.ai/docs/build/eps.html#nvidia-jetson-tx1tx2nanoxavier
• https://forums.developer.nvidia.com/t/how-can-i-run-onnxruntime-c-api-in-jetson-os/201722/5

I did the build on the Jetson Target (AGX Orin in my situation).

Assuming you have a default filesystem layout, steps to recreate:

1. clone onnxruntime repo
2. checkout the tagged version you want
3. init and update submodules recursively
4. Run their build.sh script with the following
   ./build.sh --config Release --update --build --parallel --build_wheel --build_shared_lib --use_tensorrt --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu/ --tensorrt_home /usr/lib/aarch64-linux-gnu/
5. Note, the build process will download more source, so internet access is needed.
6. Once the build process is complete, the Linux/Release/ folder will contain the .so files needed to link a c++ application, and can be copied out to an install destination folder.
7. The header files are in the repo, and can be copied out to an install destination folder

Thanks! It works!

[snnn]

Our team doesn't maintain or publish packages for Nvidia Jetson devices. However, you get one from https://elinux.org/Jetson_Zoo#ONNX_Runtime .

[csukuangfj]

https://elinux.org/Jetson_Zoo#ONNX_Runtime

The above link contains only packages for Python and it lacks of libonnxruntime.so so it cannot be used for linking in C/C++ projects.
@snnn