This is the sample code for Leopard USB3.0 camera linux camera tool. It is a simple interface for capturing, viewing, controlling video stream from Leopard's uvc compatible devices, with a special emphasis for the linux v4l2 driver.
For Leopard USB3.0 Firmware Update Tool, please refer to firmware update tool README
- Leopard USB3.0 Camera Linux Camera Tool
This application uses v4l2(video for linux) API for linux. The related system calls are also UNIX based, e.g. open(), close(), fork() etc. It doesn't support running on Mac OS and Windows. For Windows Camera Tool for your Leopard USB3 camera, please contact leopard support for application/source code.
Docker hasn't supported USB video device pass through on Windows. So even running docker, it is suggested you have a linux operating system workstation there.
5.0.0-29-generic #31~18.04.1-Ubuntu SMP4.18.0-17-generic #18~18.04.1-Ubuntu SMP4.15.0-20-generic #21~Ubuntu 18.04.1 LTS4.15.0-32-generic #35~16.04.1-Ubuntu
Due to OpenCV CUDA support hasn't been updated to 4.0.X, it is suggested to install 3.4.3 for OpenCV CUDA support.
The non-OpenCV CUDA support part has been tested and verified on the following OpenCV versions
3.4.23.4.34.0.14.1.1
-
Camera: any Leopard USB3.0/2.0 cameras
-
Host PC: with USB3.0 port connectivity
Some boards like
OV580-STEREOhas more strict voltage requirements toward USB3.0 port. It is highly suggested to use a USB3.0 powered hub to regulate the voltage out from PC's USB3.0 port if aOV580-STEREOUSB3 device is used.The underlying issue is that different host PCs/USB3 PCIe manufacturers have different USB3 controllers. Some of the USB3 cameras are not able to handle transient voltage droop from the USB3 port from PC. So a USB3.0 powered hub is highly recommended if this issue is repeatable with direct USB3 connection on your host PC. (Any serdes USB3 camera which have an external 12V power supply will be immune to this as you assumed :D)
- Do a
lsusband found your Leopard USB3.0 camera device after connection: VID of our USB device is usually2a0b(for more hardware information like OV580-STEREO, please refer to hardware.h). - Do a
ls /dev/vi*and make sure there is video device listed. - Once you know which
/dev/videoXthe camera is at, use./leopard_cam -d Xwhen you run your camera
- Do a
-
USB3.0, USB2.0 Connectivity: If you plug in your USB camera device slow enough to a USB3.0 port, your host PC might still recognize the USB3.0 device as USB2.0. To be able to tell if your host PC has been recognized this USB3.0 camera truly as a USB3.0 device, do a
v4l2-ctl --device=/dev/videoX --all, whereXis the device number that being recognized by your PC. And check if its resolution, frame rate listed is the right one, (USB2.0 camera will have much lower bandwidth, so usually small resolution and frame rate).
foo@bar:~$ lsusb ## list all your USB device on your host PC
Bus 002 Device 034: ID 05e3:0612 Genesys Logic, Inc.
Bus 002 Device 035: ID 2a0b:00cd
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 035: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 034: ID 046d:c53d Logitech, Inc.
Bus 001 Device 031: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
foo@bar:~$ lsusb -v -d 2a0b:00cd ## list verbose information on connected Leopard USB3 camera
Bus 002 Device 035: ID 2a0b:00cd
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bDeviceClass 239 Miscellaneous Device
bDeviceSubClass 2 ?
bDeviceProtocol 1 Interface Association
bMaxPacketSize0 9
idVendor 0x2a0b
idProduct 0x00cd
bcdDevice 0.00
iManufacturer 1
iProduct 2
iSerial 3
bNumConfigurations 1
...
foo@bar:~$ ls /dev//vi* ## list all the video device in your host PC
/dev/video0 /dev/video1Make sure the libraries have installed. Run configure.sh in current directory for completing installing all the required dependencies.
$ chmod +X configure.sh
$ ./configure.shMake sure you have GTK 3 and OpenCV (3 at least) installed. The way you do to install this package varies by operational system.
Gtk3 and Gtk2 don't live together peaceful. If you try to run camera tool and got this error message:
Gtk-ERROR **: GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in the same proc
It is mostly you have OpenCV build with GTk2 support. The only way to fix it is rebuild OpenCV without GTk2:
opencv_dir/release$ cmake [other options] -D WITH_GTK_2_X=OFF ..
in order to disable Gtk2 from OpenCV.
# rebuild OpenCV, this might take a while
# this is just a sample of rebuilding OpenCV, modify them to fit your need
$ rm -rf build/
$ mkdir build && cd build/
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_NEW_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_GTK=ON -D WITH_GTK3=ON -D WITH_GTK_2_X=OFF ..
$ make -j6 # do a $lscpu to see how many cores for CPU
$ sudo make install -j6
# link OpenCV
$ sudo sh -c 'echo "/usr/local/lib" >> /etc/ld.so.conf.d/opencv.conf'
$ sudo ldconfigTo build OpenCV CUDA support, please refer to INSTALL_OPENCV_CUDA.md
- Make
- Use make when you don't need OpenCV CUDA support
$ make- CMake
- Use cmake when you need OpenCV CUDA support
$ mkdir build
$ cd build
########################################################################
$ cmake .. # USE_OPENCV_CUDA is turned off by default
$ cmake -DUSE_OPENCV_CUDA=ON .. # turn on USE_OPENCV_CUDA for OpenCV CUDA support
########################################################################
$ make
# Add to your project
add_subdirectory(linux_camera_tool)
include_directories(
linux_camera_tool/src/
)
...
target_link_libraries(<Your Target>
leopard_tools
)$ sudo cp ./leopard_cam /usr/bin$ sudo rm -f /usr/bin/leopard_cam - Run your Linux Camera Tool after installation
$ leopard_cam- Run your Linux Camera Tool when you want to assign a specific video device
/dev/videoX(usually when you have more than one camera in your host PC, and the Linux Camera Tool detected/opened a different device than you want)
# either the following four ways
$ leopard_cam -d X
$ leopard_cam -d /dev/videoX
$ leopard_cam --device X
$ leopard_cam --device /dev/videoX- Run your Linux Camera Tool with a different resolution and frame rate than the default one
$ leopard_cam -s 1280x720 -t 15- For detailed usage of commands in Linux Camera Tool, please run
$ leopard_cam -h Please refer to Docker README for more information
In the following five ways:
- Press ESC, q, Q in your keyboard to exit the whole program
- Use EXIT button on the grid1 control panel to exit the whole program
- Use Help -> Exit in the menu bar to exit the whole program
- User Ctrl + C to exit the control panel after exit from camera streaming GUI
- Unexpected quit results in defunct process, use
to see if the leopard_cam still alive, if it is, use
$ ps
to kill the zombie process$ killall -9 leopard_cam
If you forget to exit both windows and tried to run the camera tool again, it will give you the error of
VIDIOC_DQBUF: Invalid argument, or when something goes wrong unexpected when running Linux Camera Tool, please check your available processes and kill leopard_cam, then restart the camera tool application
$ ps
$ killall -9 leopard_camOriginal streaming after debayer for IMX477:
image is dark and blue
Modified streaming for IMX477:
enabled AWB, gamma correction, black level correction with non OpenCV CUDA support
enable awb, auto brightness & contrast, sharpness control with OpenCV CUDA support
Modified streaming for OS05A20 full resolution
change bayer pattern to GRBG enable software AWB, gamma correction increase exposure, gain
Modified OS05A20 resolution to an available one
binning mode capture raw and bmp, save them in the camera tool directory
Original streaming:
Default YUV output
CLAHE:
Run OpenCV CLAHE(Contrast Limited Adaptive Histogram Equalization)
Edge detection(Canny):
Run OpenCV Canny edge detection
$ linux_camera_tool .
├── Makefile # for make
├── configure.sh
├── CMakeLists.txt # for CMake
├── gitlog-to-changelog.md # automate generating CHANGELOG.md, version number in software
│
├── README.md # THIS FILE
├── INSTALL_OPENCV_CUDA.md
├── CHANGELOG.md
├── LICENSE.md
├── AUTHOR.md
│
├── config.json # configuration files for automating
├── BatchCmd.txt # camera controls
|
├── includes
│ ├── shortcuts.h # common used macro functions
│ ├── hardware.h # specific hardware feature support
│ ├── gitversion.h # auto-generated header file for revision
│ ├── utility.h # timer for benchmark
│ ├── cam_property.h
│ ├── extend_cam_ctrl.h
| ├── isp_lib.h
│ ├── core_io.h
│ ├── fd_socket.h
| ├── json_parser.h
│ ├── batch_cmd_parser.h
│ ├── ui_control.h
│ ├── uvc_extension_unit_ctrl.h
│ └── v4l2_devices.h
│
├── src
│ ├── cam_property.cpp # gain, exposure, ptz ctrl etc
│ ├── extend_cam_ctrl.cpp # camera stream, capture ctrl etc
| ├── isp_lib.cpp # ISP using OpenCV
| ├── core_io.cpp # string manipulation for parser
| ├── fd_socket.cpp # file descriptor socket for IPC
| ├── json_parser.cpp # json parser
| ├── batch_cmd_parser.cpp # BatchCmd.txt parser
| ├── ui_control.cpp # all GUI related
│ ├── uvc_extension_unit_ctrl.cpp # all uvc extension unit ctrl
│ └── v4l2_device.cpp # udev ctrl
│
└── test
└── main.c
Most of the image processing jobs in Linux Camera Tool(code for ISP) are done by using OpenCV. Since there are histogram matrix calculations and many other heavy image processing involved, enabling these features will slow down the streaming a lot. If you build Linux Camera Tool with OpenCV CUDA support, these feature speed won't be compromised a lot. Detailed benchmark result is listed below.
Benchmark test is performed on a 12M pixel RAW12 bayer camera (IMX477) with host PC OpenCV build TBB on.
| Functions | Non-CUDA OpenCV Support | Latency | CUDA OpenCV Support | Latency |
|---|---|---|---|---|
Auto White Balance |
Yes | 55ms | Yes | 3700us |
CLAHE |
Yes | 40ms | Yes | 7700us |
Gamma Correction |
Yes | 2.7ms | Yes | 1100us |
Sharpness |
Yes | 200ms | Yes | 20ms |
Show Edge |
Yes | 20ms | Yes | 16ms |
Color Correction Matrix |
Yes | 18ms | No | N/A |
Perform Shift |
No | 13ms | No | N/A |
Separate Display |
Yes | 9000us | No | N/A |
Brightness |
Yes | 7ms | Yes | 400us |
Contrast |
Yes | 7ms | Yes | 400us |
Auto Brightness & Contrast |
Yes | 14ms | Yes | 1180us |
Auto exposure is usually implemented on sensor/ISP, which when enabled, won't further slow down the streaming. You need to check with camera driver for auto exposure support. If some sensor don't have AE support built-in, the check button in the first tab of Camera Control (Enable auto exposure)won't work.
There is a software generated auto exposure support in the second tab of Camera Control (Software AE), you can enable that if you really need AE support for the camera. Since adjusting gain and exposure under linux will experience split screen issue, same thing will happen to this software AE.
Since V4L2 doesn't provide a easier way to tell you what exact exposure time in ms like what Windows does, here is the explanation for helping you figuring out your camera's current exposure time in ms:
Exposure time = exposure_time_line_number / (frame_rate * total_line_number)
, where
- exposure_time_line_number is the value that display is linux camera tool exposure control.
- you can get frame_rate from the log message
Get Current Frame Rate =, don't refer toCurrent Fpsthat displays in theCamera View, that's the frame rate your PC can process, not the physically USB received frame rate from camera. - total_line_number is usually around the height of the camera's current resolution, usually slightly larger than that since VD needs blanking. To get the exact total_line_number, you might want to read VTS register value for your sensor, which might be a hassle for you so I will only tell you this roughly calculation method...
- Noticing for exposure time change in Linux Camera Tool Control GUI, the exposure time range will usually be over the camera's resolution height. If you have the exposure value set to be over camera's resolution height, it might affect your frame rate. Since when camera exposure time line number is greater than the total line number, that will slow down your camera's frame rate. This varies from different sensors. Some sensors don't allow that happen, so the exposure time will stick the same when it is over total line number.
When changing exposure and gain under linux, camera tool will experience split screen issue at the moment change is happened. This issue happens for the USB3 camera that use manual DMA in the FX3 driver. For the camera that utilizes auto DMA, the image will be ok when exposure and gain change happens.
For updating driver from manual DMA to auto DMA, you need to ensure:
PTSembedded information is not needed in each frame- The camera board has a
CrosslinkFPGA on the tester board that is able to add uvc header to qualify forauto DMA FX3 driverneed to be updated
When use triggering mode instead of master free running mode for USB3 SerDes camera, the very first frame received will be bad and should be tossed away. It is recommended to use an external function generator or a dedicated triggering signal for triggering the cameras for the purpose of syncing different SerDes cameras.
The included shot 1 trigger function is only a demonstration on generating one pulse and let camera output one frame after shot 1 trigger is clicked once. User should not fully rely on this software generated trigger but use a hardware trigger for sync the camera streaming.
Report bugs by creating a new issue on this Github page: https://github.com/danyu9394/linux_camera_tool/issues
If it is clearly a firmware issue: likely some features hasn't been implemented in the driver, please contact Leopard support: support@leopardimaging.com
Fork the project and send pull requests, or send patches by creating a new issue on this Github page: https://github.com/danyu9394/linux_camera_tool/issues