Support for yolov4 darknet

[neeraj-satyaki]

Hi, Great SDk for inferencing on Video streams parllely, do you guys support for Yolov4 Darknet weights too?

[miguelaeh]

Hi @neeraj-satyaki,

You can load any model and use any framework. You can either use the ONNX Runtime (so you will provide pre and post process functions only in your stage) or even use Pytorch, Tensorflow, or anything else by creating a process function (and also pre and post process).

This is an example of the first, directly loading a model in onnx format: https://github.com/pipeless-ai/pipeless/tree/main/examples/onnx-candy
This is an example of the second, which uses the Ultralytics SDK for YOLOv8: https://github.com/pipeless-ai/pipeless/tree/main/examples/yolo

[neeraj-satyaki]

Do we have to add anything specific in preprocess.py

[miguelaeh]

No, you don't even have to create that file if not required. You tipically provide it when your model requires a specific image format, size, etc so you perform the modifications there and assign it to inference input. If your model can receive the raw RGB image with the same size as it comes from the source you do not need to even create a pre-process file

[neeraj-satyaki]

My Process.json is as follows
{
"runtime": "onnx",
"model_uri": "file:///models/yolov4.onnx",
"inference_params": {
"execution_provider": "TensorRT"
}
}

As you can see in the image, I am unable to run this on Tensorrt or its not picking up GPU
[2024-01-21T16:27:14Z INFO ort::memory] new_cpu; allocator=Arena memory_type=Default
[2024-01-21T16:27:14Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c928b60, should_release: true }
[2024-01-21T16:27:14Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:14Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c642be0, should_release: true }
[2024-01-21T16:27:14Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:14Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c932020, should_release: true }
[2024-01-21T16:27:14Z INFO ort::memory] new_cpu; allocator=Arena memory_type=Default
[2024-01-21T16:27:14Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c929d20, should_release: true }
[2024-01-21T16:27:14Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:14Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c644620, should_release: true }
[2024-01-21T16:27:14Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:14Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c641fa0, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new_cpu; allocator=Arena memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c63c1a0, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c63cfe0, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c6408e0, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new_cpu; allocator=Arena memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c63eee0, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c92a8a0, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c92cb60, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new_cpu; allocator=Arena memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c537ba0, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c92d0a0, should_release: true }
[2024-01-21T16:27:15Z INFO ort::memory] new; allocation_device=CPU device_id=0 allocator_type=Device memory_type=Default
[2024-01-21T16:27:15Z INFO ort::memory] drop; self=MemoryInfo { ptr: 0x7f0a5c92dba0, should_release: true }

These are the logs above,
Below is the command,
docker run --gpus all -v ${PWD}/:/app -v /home/ubuntu/Models:/models -e "PIPELESS_USER_PYTHON_PACKAGES=opencv-python;ultralytics;onnxruntime;tensorrt;" -p 3030:3030 miguelaeh/pipeless start --stages-dir /app

Could please help me out on how to use

[miguelaeh]

Hi @neeraj-satyaki !

(This is just a productivity advice): It will be more comfortable for you to work if you also mount a volume into /.venv because in that way you will avoid installing the Python packages on every docker run, so you will work faster. See this section: https://www.pipeless.ai/docs/docs/v1/container#install-custom-python-packages

Regarding your issue, I just checked it and it seems you are right, there is an issue in the TensorRT container image due to some packages not being found bylibonnxruntime_providers_tensorrt.so. If you install Pipeless in your host directly it will work fine.
I am going to create a new PR fixing the image now and will link it here. Thanks for the report!

[miguelaeh]

@neeraj-satyaki I forgot to tell you your command is also using an image only valid for CPU. You have to use miguelaeh/pipeless:latest-tensorrt if you use latest it will be CPU only. At the beggining of this page you can see the complete list of images: https://www.pipeless.ai/docs/docs/v1/container

And, this PR should fix the problem: #125

[neeraj-satyaki]

I was uptill here, for miguelaeh/pipeless:latest-tensorrt. Could Tell me where to move through.
Sorry For asking you too much. Just Felt Stuck so

[miguelaeh]

Ey @neeraj-satyaki no problem!

The first issue you see about TensorRT failing to load is the same error we commented yesterday. I merged the PR, however, I am having trouble to release the new image in this job, for some reason it gets stuck (seems to be a known error of the docker buildx github action).

Then you have an error that is probably related to a wrong file path to yolov8n-seg-psda.pt. It seems you are using a file with that name in a init.py file, right? Ensure the path to the file is absolute and remember it must be a valid path inside the container not in your host.

The third error (the one stated by the parser when loading stages) seems to me like you mounted the /.venv folder inside /app. It must be mounted like -v "/tmp/venv:/.venv" (i.e. at /.venv). The Pipeless project (/app directory) cannot contain files that Pipeless do not expect (yet) so it is failing when parsing that unexpected venv folder at /app/.venv. Ensure you mount it at /.venv instead.

[neeraj-satyaki]

Thank you so much, for your insights. For now can i use
Tensorrt version of the image