Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Camera support (Jetson-Utils/NanoLLM/Jetson-Containers) - need extra option "--input-decoder=cpu" #721

Open
TangmereCottage opened this issue Nov 12, 2024 · 0 comments

Comments

@TangmereCottage
Copy link
Contributor

TangmereCottage commented Nov 12, 2024

Problem Of about 20 different webcams tried, about 10 work out of the box, and the other 10 fail due to various errors. This makes the user experience working with NVIDIA Jetson poor, because even supposedly simple things - attach webcam and run a basic VLM, is a hit or miss experience.

Debugging Different cameras, especially newer ones, can generally be made to work with via gst-launch-1.0 by adjusting parameters, but these parameters cannot be passed to Jetson-Utils. For example, some cameras require an extra h264parse before nvv4l2decoder, but the logic that currently handles generation of the gst pipeline string is complex and even has camera-specific workarounds.

Solution 1 Rather than trying to accommodate hundreds of cameras, allow the user to pass a full gst pipeline string into the nano_llm.plugins VideoSource. The current pipeline string generation logic could remain untouched, but something like VideoSourceLowLevel(pipelinestring) could be used by more advanced users to handle essentially all cameras (at least those handled by gst streamer).

Solution2 Add a new flag --input-decoder=cpu or equivalent so that users can bypass the brittle vl2 logic, which tends to work only for very basic webcams like the Logitech 270.

Happy to help with this, please let me know if this could be useful

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant