Camera Calibration
The camera calibration module is responsible for calibrating the cameras and generating the matrices required to generate a top-down view of the road. The module uses OpenCV to find the corners of the ChArUco board in the images and calculate the homography between the cameras. The data generated by the calibration process is used to stitch the images together and transform them into a top-down view. Since the real-world size of a square on the ChArUco board is known, we can calculate the distance to a point in the image.
The cameras on the go-kart tend to change their positions slightly over time, which is why it is important to recalibrate them regularly. Doing this manually is time-consuming and error-prone, which is why this process is completely automated. It allows you to simply place the ChArUco board in front of the cameras and let the system do the rest.
- Usage
- CameraCalibrator Class
-
CalibrationData Class
- Example Usage
-
Methods
transform(images)transform_point(x, y, shape)get_distance_to_point(x, y, shape)
- How does it work?
- References
Before running the calibration process, ensure that the camera IDs are correctly configured in the configuration file.
To calibrate the cameras, execute the calibrate_cameras.py script:
python -m scripts.python.calibrate_camerasThe CameraCalibrator class includes the calibration process for the cameras. It provides matrices used to stitch the images together and turn them into one top-down image, and data required to calculate the distance to a certain point in the image.
from src.calibration.calibrate import CameraCalibrator
# Calibrate the Cameras
calibrator = CameraCalibrator([left_image, center_image, right_image], input_shape=(1280, 720))
calibrator.calibrate()
# Save the Results
calibrator.save("./data/calibration")The CalibrationData class serves as a wrapper for the calibrated camera data. It generates matrices necessary for stitching images together into a single top-down view and provides the required data for distance calculations.
from src.calibration.data import CalibrationData
# Load the Calibration Data
calibration = CalibrationData.load("./data/calibration/latest.npz")-
transform(images): Transforms images into a top-down view using the calibrated parameters.topdown = calibration.transform([left_image, center_image, right_image])
-
transform_point(x, y, shape): Transforms a point in the center camera to the corresponding coordinates in the top-down view.x, y = calibration.transform_point(780, 310, shape=(1280, 720))
-
get_distance_to_point(x, y, shape): Calculates the distance in meters to a point in the center camera.distance = calibration.get_distance_to_point(780, 310, shape=(1280, 720))
To calibrate the cameras, we need to capture images of the ChArUco board from all cameras. It is extremely important that the ChArUco board is clearly visible in all images. A ChArUco board is a chessboard with ArUco markers on it. The ArUco markers allow us to tell which corner is which, making it extremely useful for stitching images together.
# Import OpenCV
import cv2
# Initialize the cameras.
cam_left = cv2.VideoCapture(0)
cam_center = cv2.VideoCapture(1)
cam_right = cv2.VideoCapture(2)
# Get the frames of each camera.
ret, frame_left = cam_left.read()
ret, frame_center = cam_center.read()
ret, frame_right = cam_right.read()
Next up is locating the corners of the ChArUco board in all images. OpenCV has a built-in module for dealing with ArUco markers, called cv2.aruco. We can use this module to find the corners of the ChArUco board in each image.
The next step is to find the homography between the center camera and the other cameras. A homography is a transformation matrix that maps points from one image to another. We can use the homography to make the other cameras have the same perspective as the center camera. This is used to make one big image from all the cameras, giving us more details of the road.
To stitch the images together, we need to find the offsets between the leftmost camera and the other cameras. This is required to ensure that when we combine them, they blend seamlessly and look like one big image.
The bottom part of the stitched image contains the kart and parts of the road that are not relevant. To declutter the image and speed up processing, we can safely crop the bottom part of the image.
In order to avoid the Pac-Man Effect, we need to find the vanishing point of the road and crop away everything above it. We can do this by finding the intersection of the ChArUco board's lines.
Note
Pac-Man Effect: The belief that someone attempting to go over the edge of the flat Earth would teleport to the other side.
Table 1: On the left is an image illustrating the calculation of the vanishing point, while on the right is the outcome after cropping.
| Before Cropping | After Cropping |
|---|---|
 |
 |
To get a top-down view of the road, we need to find the homography between the real-world grid of the ChArUco board and the ChArUco board that we see in the image. A top-down view is useful for path planning, calculating distances, and the curvature of the road.
Table 2: On the left is an image showing the homography between ChArUco board points in the image and the real world. On the right is the resulting top-down view.
| Before Transform | After Transform |
|---|---|
 |
 |
If the ChArUco board is not perfectly parallel to the kart, the top-down image will be slightly rotated (or heavily, depending on how good you are at placing the ChArUco board). We can correct this by finding the angle of the center camera and rotating the top-down image. We can simply modify the homography matrix to rotate the image.
The final step is to calculate the region of interest. The top-down image is very large and contains a lot of unnecessary information. We don't need the entire image; instead, we only want to look ahead a certain amount of meters and a certain width.
We know the real-world size of a square on the ChArUco board, which gives us the pixel-to-meter ratio. We can modify the homography matrix to only show a certain region, e.g., 10 meters ahead and 5 meters on each side.
- Medium: Multi-View Image Stitching based on the pre-calibrated camera homographies
- OpenCV: ArUco Marker Detection
- OpenCV: Camera Calibration and 3D Reconstruction
- OpenCV: Camera Calibration using ChArUco Boards
- OpenCV: Documentation
- StackOverflow: Bird's eye view perspective transformation from camera calibration opencv python
- Wikipedia: Rotation Matrix
- Wikipedia: Vanishing Point