How should be the stages connected?

[MikiGrit]

Forgive me if I've just missed it somewhere in the documentation. But how do I connect different stages?

I have Intel Realsense camera (which in my case basically outputs color and depth frame) and I want to process it and display the output (or save it, etc ...). I imagine that I should split the app to the following stages.

1. reading the video stream with two frames as outputs (color and depth)
2. first object detection (let's say I want to detect animals)
3. second object detection (humans)
4. analysing the detections
5. drawing results

Is it possible to do it with pipeless? Note that results from the first stage should go to second and third. And then from both join again in the fourth stage.

Thanks for your effort! Pipeless seems like really nice project.

[miguelaeh]

Hi @MikiGrit ,

Stages are connected by default. When you provide a stream with the stream add command, you need to provide a frame-path parameter, where you will list the stages in order (--frame-path stage-1,stage-2,etc).

The frame data that one stage produces is received by the next stage. For example, imagine a stage called modify, there you draw a bounding box and set the modified field of the received frame data. Now, in a stage whatever you can access the modified field of the frame data and it will contain the changes you did. You can find here the available frame data fields: https://www.pipeless.ai/docs/docs/v1/getting-started#frame-data-fields

Note the modify field can only contain data of the same shape as the input frame, since it is what will end up going to the output. In the case you expose if you perform object detection with some library instead of one of the supported inference runtimes you would add your data to the inference_output field of the frame data, which can contain data of any shape.

There is also a KV store you can use in case you need to maintain some kind of internal state. The KV store helps you communicate data between the hooks o a stage. For example, you can insert some data in the pre-process hook and obtain it in the post-process hook. See https://www.pipeless.ai/docs/docs/v1/kvstore

For your use case, I think you can use the KV store to save the analysis: You could use as key the frame id (which is also available in the frame data that the hook receives) plus maybe a suffix like -humans or -animals. The KV uses no-copy mechanism on reads so it should not introduce too much overhead.
Your case seems like a bit complex so you probably need those stages, however, if with "analysis" you mean to just parse the detection outputs and/or animals and humans can be processed as independent, you could save on the number stages. For example:

1. animals detection (process.py or process.json), then analysis + draw in post-process.py
2. humans detection (same approach). Just use original to get the original frame. Then when you edit modified you are editing over the already edited frame from the previous stage.

Finally, about producing 2 outputs from a stage, that's not supported right now. A hook receives the frame data structure and produces the frame data structure. They key here is to isolate processing steps in stages. In your case, I understand the colors processing is isolated from depth, then you can create a stage for each one.

I am sorry for this long response, let me know if I was not clear on something. You can definitely implement the use case with Pipeless. If you can share some extra context on the dependencies between each of the steps you need to implement I can help you designing which stages to create. It is all about isolating stages as much as possible.

[marklysze]

Thanks for asking @miguelaeh, unfortunately not. I've been working directly with realsense using their C++ library.

For Oak-D I have used both the C++ and Python libraries (https://github.com/luxonis/depthai-python).

Without trying to derail the thread, and I'm no expert with Pipeless, are there thoughts on how to manage two streams of image frames, synchronised, to provide a depth frame?

[miguelaeh]

Well, right now I would say it is not possible, I have not tried it though, as I do not have similar cameras. The current input pipelines cover common cases from different protocols, however I am not familiar with how those streams are read. Are they just like two different streams? If you can read those from the camera as two different inputs you could process them separatedly. I guess then the point would be to allow sharing information between those.

[marklysze]

Understand, yep and I think it's entirely up to you whether this is worth pursuing er whether it complicates the project.

My understanding is that there are two streams of frames and each image has a synchronised number, when you have received two frames with the same number you can determine the depth map from combining them. This values for each pixel in this depth map is a distance value. Hopefully I'm representing that correctly. Obviously there are lens parameters and the distance between sensors are required when working out the depth map.

No pressure from my side to support.

[miguelaeh]

Thanks for the explanation. So I think the best option would still be to find some kind of Gstreamer plugin for that. If more people need this feature it could be interesting to support it, but by the moment I am afraid I have no bandwidth to investigate deeper on that. If you feel confortable and would like to send a PR I will be more than happy to review it.

[marklysze]

Absolutely understand! If I do get the bandwidth I'll investigate options but it's not my area of expertise so I'm unlikely able to get to the stage of a PR. I'll keep my eye out.