Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

video pipeline on rt10xx: Fix chicken-egg issue in init priority #80052

Conversation

ngphibang
Copy link
Contributor

@ngphibang ngphibang commented Oct 18, 2024

Similar to the mcxn947 (ov7670/smartDMA) issue, the CSI needs to be initialized BEFORE the camera sensor to provide clock to the camera sensor. However, the CSI node refers to the camera sensor node in the DT so it has to be initialized AFTER the sensor to be able to compile.

This is a deadlock/chicken-egg issue as Zephyr currently does not allow circular dependency. Disable the init priority check as a work-around for this.

@ngphibang
Copy link
Contributor Author

We knew and did take the work-around internally but I think this should go into upstream so that people have an "out-of-the-box" working sample on i.MX RT10xx.

josuah
josuah previously approved these changes Oct 18, 2024
Copy link
Collaborator

@josuah josuah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the init dependency is in the opposite direction than the source = <&ov7670>; suggests: phandles are considered as dependencies by the python scripts, whereas it is the opposite here.

If source = <&ov7670>; is replaced by remote-endpoint-label = "ov7670_out"; (for instance), would that also be a workaround? In this case, there would not need to disable CONFIG_CHECK_INIT_PRIORITIES as there would be no more phandle at all?

Besides this, the workaround looks logical and avoids a compilation error, so seems necessary.

@danieldegrasse
Copy link
Collaborator

If I understand correctly, the init dependency is in the opposite direction than the source = <&ov7670>; suggests: phandles are considered as dependencies by the python scripts, whereas it is the opposite here.

If source = <&ov7670>; is replaced by remote-endpoint-label = "ov7670_out"; (for instance), would that also be a workaround? In this case, there would not need to disable CONFIG_CHECK_INIT_PRIORITIES as there would be no more phandle at all?

Besides this, the workaround looks logical and avoids a compilation error, so seems necessary.

I think this would work, but I'm not sure Zephyr's DT tooling is capable of getting a device object via another method than a phandle. We could probably use something like DT_NODELABEL(DT_STRING_TOKEN(remote_endpoint_label)) to get the node ID in this case? @decsny do you have any thoughts here?

@josuah
Copy link
Collaborator

josuah commented Oct 18, 2024

We could probably use something like DT_NODELABEL(DT_STRING_TOKEN(remote_endpoint_label)) to get the node ID in this case? @decsny do you have any thoughts here?

Absolutely! This is this PR and while I did not add ztests yet, this works as a workaround:
https://github.com/zephyrproject-rtos/zephyr/pull/73023/files#diff-d95b579605c12a2d00175b8b293dd3b30688ceed11edc95f473d4e3dc40b5babR3851

@ngphibang
Copy link
Contributor Author

ngphibang commented Oct 18, 2024

Yes, the intentional purpose of presenting remote-endpoint-label is to be able to find a way to work around the "circular dependency" issue but I didn't know that we can retrieve the object from the string (rather easily) with DT_STRING_TOKEN. Glad to know this through this PR.

I think both are work-around but this one (remote-endpoint-label with DT_STRING_TOKEN) is much better. Then we will need to pay attention when retrieving the object with DT_STRING_TOKEN to make sure that the peer device is already intialized.

I initially made this PR just in waiting for #73023 to get in first but yes, we can actually use DT_STRING_TOKEN individually in each driver while waiting for a common macro in device.h or video.h. Will change to this approach after checking that DT_STRING_TOKEN actually works

@ngphibang
Copy link
Contributor Author

ngphibang commented Oct 18, 2024

To broaden the discussion scope, Linux has all the mechanism to parse and retrieve the peer (sub-)device for example with some helper functions v4l2_async_notifier_parse_fwnode_endpoints_by_port(), etc.

And I think this circular dependency issue here is a sub-problem that we don't have "async init" in Zephyr where each device wait for each other to complete the initialization phase as each of them depends on others in different steps of initialization.

@josuah
Copy link
Collaborator

josuah commented Oct 18, 2024

So if I this gets merged first, it gives more time to design the DT endpoint API, and in the meantime the samples are working.

ngphibang and others added 5 commits October 21, 2024 22:08
As getting bytes per pixel of a pixel format is a very common operation,
add an utility function for it instead of repeating the same codes in
different drivers.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Remove the obsolete comment about the init order of the CSI and the
camera sensors (e.g. mt9m114) which is not true anymore.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
When stopping, video_flush() moves all the remaining buffers
from the fifo_in to the fifo_out queue.

When restarting, we have two issues:
- There might be insufficient or no buffers in the fifo_in
queue to get the pipeline started.

- When dequeuing, users might immediatly get the incorrect
buffers in the fifo_out which do not come from interruption.

Fix it by moving all the remaining buffers from the fifo_out
to the fifo_in before starting the video capture.

Signed-off-by: Trung Hieu Le <trunghieu.le@nxp.com>
Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Add support for changing frame rate

Signed-off-by: Trung Hieu Le <trunghieu.le@nxp.com>
Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
The mcux csi is an NXP IP and the driver has been recently changed much
by NXP, so add NXP copyright to it.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
ngphibang and others added 14 commits October 21, 2024 22:08
Switch to use the new video interfaces bindings

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Switch to use the new video interfaces bindings

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Switch to use the new video interfaces binding

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Switch to use the new video interfaces binding

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
The csi, mipicsi2rx, ov5640 devices now use the new video interface
bindings. Change the test accordingly to take into account this.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
As CSI now uses the new video-interfaces binding, this shield needs to
be changed too.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
The ov5640 camera driver now supports both MIPI CSI2 (DPHY) and DVP
modes. It is in MIPI CSI2 mode in this overlay. Add bus-type property
for this. In this mode, data-lanes property is required as well.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Get number of data lanes from device tree instead of hard-coding it.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Add set_ctrl callback to propagate controls to the sensor.

Signed-off-by: Farah Fliss <farah.fliss@nxp.com>
Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Instead of fixing csi2rx clock frequencies, set them according to the
pixel rate got from the camera sensor.

Signed-off-by: Trung Hieu Le <trunghieu.le@nxp.com>
Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
Add support for changing frame rate.

Signed-off-by: Trung Hieu Le <trunghieu.le@nxp.com>
Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
The peer remote device "sensor_dev" can be retrieved from the
remote-endpoint-label. Direct reference via phandle is not needed.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
The peer remote device "source_dev" can be retrieved from the
remote-endpoint-label. Direct reference via phandle is not needed.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
The CSI needs to be initialized BEFORE the camera sensor to provide
clock to the camera sensor. It is now possible to do so as direct
reference to the sensor via phandle is now not needed, so there
is no failed check on the init order anymore when compiling.

Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>
@ngphibang
Copy link
Contributor Author

I think I will close this PR and move the 3 commits in this PR to #76475 as they strongly depend on that PR and also fall in the same theme "Improve NXP csi, mipi csi2rx drivers".

@ngphibang ngphibang changed the title samples: video: capture: rt10xx: Add work-around for init priority issue samples: video: capture: rt10xx: Fix chicken-egg issue in init priority Oct 22, 2024
@ngphibang ngphibang changed the title samples: video: capture: rt10xx: Fix chicken-egg issue in init priority video pipeline on rt10xx: Fix chicken-egg issue in init priority Oct 22, 2024
@ngphibang ngphibang closed this Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Samples Samples area: Shields Shields (add-on boards) area: Video Video subsystem platform: NXP Drivers NXP Semiconductors, drivers platform: NXP NXP
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants