- Team Lead
- Feng Xiao (xiaofeng.ustc@gmail.com)
- Members
- David G (djg9610@gmail.com)
- David Simon (simonarama@yahoo.com)
- Guillermo Gómez (ggomezbella@gmail.com)
- Liheng Chen (liheng@freenet.de)
The result of this project can be found from the following videos:
-
Running everything on simulator: https://youtu.be/YFt2R3aFveA
-
Trafflic light detection and classification on simulator: https://youtu.be/4QHtUHBuSWI
-
Traffic light detection and classification on ros bag file of testing lot: https://youtu.be/2jP-MySAgEo
As introduced in the project walkthrough, The vehicle subsystem contains mainly three different parts, perception, planning and control. And each part contains several different ros nodes for different functions. Different ros nodes communicate with each other by subscribing or publishing to interesting ros topics. The project software architecture can be shown as following:
An important part of perception in this project is to detect the traffic light state from an image. Though in the simulator the ground truth traffic light state is provided, in a real car, we need to get the traffic light state based on the image only. And before doing that, we need to detect the traffic light state from the image in the simulator, too.
In this project, we make use of transfer learning and Tensorflow Object Detection API. The main steps can be described as following:
-
Train a model for general traffic light detection and classification
-
Select a pretrained general object detection model from Tensorflow Object Detection Model Zoo.
-
Collect general traffic light detection and classification dataset.
-
Perform transfer learning on the pretrained general object detection model using the traffic light dataset, and get a general traffic light detection and classification model.
-
-
Train a model based on general traffic light detection and classification for simulator images
-
Collect traffic light detection and classification dataset from simulator.
-
Perform transfer learning on the general traffic light detection and classification model using the simulator traffic light dataset.
-
Use this model in simulator for traffic light detection and classification.
-
-
Train a model based on general traffic light detection and classification for real world testing environment images
-
Collect traffic light detection and classification dataset from ros bag file provided for real world testing environment.
-
Perform transfer learning on the general traffic light detection and classification model using the real world testing environment traffic light dataset.
-
Use this model in testing car for traffic light detection and classification.
-
Tensorflow provides a lot of pre-trained object detection models in the Tensorflow Object Detection Model Zoo. Since this part will be running in the real car in real time, speed is quite important, and, of course, accuracy is also critical. After trying out several different pretrained models, we finally selected ssd_mobilenet_v2_coco. This model is very fast to run, and it is also accurate enough.
In this project, we use the Bosch Small Traffic Light Dataset to train the general purpose traffic light detection and classification model. The training set of this dataset contains 15 different labels; however, in our project, we only need 4 different labels, red, yellow, green and off. We preprocessed the test.yaml file and replaced those extra labels with corresponding red, yellow, green or off labels. We also split the dataset with 80% of the samples used for training and 20% of the samples used for validation.
By applying transfer learning on the general ssd_mobilenet_v2_coco model and training it with the Bosch small traffic light dataset, we got the following training result.
The config file for using the Tensorflow Object Detection pipeline can be found here. After the model is trained, it works quite well on general traffic light detection and classification. You can see the results from the following images:
After getting the general traffic light detection and classification, we tried to use it directly in our project for both the simulator and the ros bag file. However, the results are not so good. In the simulator, it can detect the traffic light states. However, the result is not so stable, and sometimes it fails. For the ros bag file, due to the special lighting conditions and the reflection of the windshield, it fails to detect the traffic lights most of the time.
To make the model perform better, we needed to feed it with special data for the simulator and the ros bag file. Since the simulator and the ros bag file are so different, we decided to train two different models, one for each.
The straight forward way to collect training data from the simulator and the training ros bag file is to annotate the images ourselves. Thanks to coldKnight, he shared this annotated simulator and ros bag file dataset. So we could save a lot of time annotating the images. After some double checking and minor fixing of the dataset, we used them for training our specific traffic light detection and classification models.
By applying transfer learning on the general traffic light detection and classification model and training it with the annotated simulator traffic light examples, we get the following training result:
The config file for using the Tensorflow Object Detection pipeline can be found here. The result of running this mode on the simulator images can be found from https://youtu.be/4QHtUHBuSWI. And the following are some examples of applying this model on some simulator images:
By applying transfer learning on the general traffic light detection and classification model and training it with the annotated ros bag file traffic light examples, we get the following training result:
The config file for using the Tensorflow Object Detection pipeline can be found here. The result of running this model on the provided ros bag file can be found https://youtu.be/2jP-MySAgEo. And the following are some examples of applying this model on some images from the testing ros bag file:
The vehicle controller handles driving behavior for the car. This is done by calculating appropriate values for actuation commands. "Appropriate" values can only be determined once we have a baseline of how the car should behave.
The waypoint loader node for the system (as can be seen in the system architecture above) loads the base waypoint for our "map", or driving area. These base waypoints are then fed into the waypoint updater node along with traffic waypoints and the current position of the car in the map. This is an important node for vehicle control, as it is used to determine the desired location and velocity of the car. Once the waypoint updater node makes these determinations, car actuations can be determined.
The actuation values for the vehicle are primarily handled or assisted by PID controllers. A PID controller is an algorithm that calculates a value that serves to reduce error in a system. The error in a system is the difference between the desired outcome, and the current state. In the case of this system, it is the difference between the waypoint position, and the car's current positioning.
A more detailed explanation of PID controllers in general can be found here
PID, and generally how it's used in the system, is defined as:
-
P: Proportional - Actuate proportional to the error of the system multiplied by a factor of Tau.
-
I: Integral - Actuate proportional to the sum of all errors. Used to correct for system bias that would normally prevent the car from reaching the desired state.
-
D: Derivative - Gradually introduces counter-actuation to avoid overshooting the target state.
kp = 0.3, ki = 0.1 and kd = 0.0 (no derivative term) as set by experiment testing and comes with the repo for twist_controller.py
The PID calculation can be seen in pid.py
Throttle and steering behavior are determined as such:
-
Throttle: The difference between our desired waypoint velocity (set in waypoint_updater.py) and the current car velocity is measured and fed into a PID controller to apply appropriate throttle values. The car's velocity is first filtered by a lowpass filter to prevent noisy measurements from causing high-variance actuation.
-
Steering: Steering is a special case, as it's comprised of a yaw controller (yaw_controller.py) for general control, and assisted by a PID controller for smaller actuation adjustments. By using the combination of a PID controller and a yaw controller, we can achieve much smoother steering than either method by itself. kp = 0.15, ki = 0.001, kd = 0.1 was used in twist_controller.py
The drive by wire system was disengaged to see how the car would operate. This is critical as a real vehicle like Carla may require a driver to take over as needed. The PID controllers can accumulate error which can result in a sudden jerking of the vehicle and loss of control as drive-by-wire mode is reengaged. As a result, in twist_controller.py under the control function, the throttle and steering controllers are reset with kp, ki, and kd set to 0. When this was tested in the simulator, the PID controllers stop and restart based on the status of dbw_enabled as expected. No accumulated error when DBW is turned off. The steering adapts back to the waypoints quickly if DBW is stopped and restarted.
Default velocity of 40 kilometers per hour is set in the waypoint_loader.launch under Velocity was tested at 40, 50, and 60 km/h. The car never went more than 0.05 mph above the speed limit. Some steering issues were seen at the higher speeds which led to implementation of the PID steering component.
Waypoint velocities are first set in the waypoint_updater node, as mentioned above. If a red traffic light is present within the calculated future waypoints, the system needs to set decelerated velocities for the waypoints leading up to that traffic light. The decelerated velocities are set as a function of distance. This can be seen in the decelerate_waypoints function of waypoint_updater.py. The calculation given in the repo is vel = math.sqrt (2 * MAX_DECEL * dist). This is derived from the classic kinematics equation v squared = v initial squared + (2 * constant acceleration * distance). Here, due to stopping, v squared is 0 and maximum deceleration as in previous projects was set at 5 m/s per second. The distance from the stopline decreases, so velocity vel goes down as the car gets closer to the stopline of the traffic light. The walkthrough indicated the square root can make this stopping less smooth than desired. That was confirmed in testing. Using https://www.desmos.com/calculator to experiment with different functions, vel = 0.006 * (dist * dist) + 1.5, was found to be smoother, more gradual. This is implemented in the decelerate_waypoints function of waypoint_updater.py, with the square root function commented out.
Braking is under the twist_controller.py file within the control function.
Once the waypoint velocities are set, braking behavior can then be determined if the velocity error (same as the throttle calculation) is a negative value. This value being negative means that the car is moving at a higher velocity than the waypoint's desired velocity, so the controller will apply braking force. Throttle is set to 0. Braking force is determined by the torque calculated from the vehicle mass * wheel radius * deceleration. The deceleration is the velocity error up to the maximum deceleration set in dbw.launch at -1 m/s per second. Vehicle_mass of 1736.35 kg and wheel_radius of 0.2413 meters are also in the dbw.launch file.
Once the car is stopped at the red light, throttle is set to 0. The brakes are set at 400 Nm braking torque as specified in the walkthrough to keep the car stopped. This braking torque comes from the vehicle_mass * wheel_radius * accel_limit (419 Nm actual)
When the light turns green, the car accelerates at 1 m/s each second as set in dbw.launch for accel_limit.