Skip to content

RecyclingRush ♻️: Towards Continuous Floating Invasive Plant Removal Using Unmanned Surface Vehicles and Computer Vision, IEEE Access 2024.

License

Notifications You must be signed in to change notification settings

EdwinTSalcedo/RecyclingRush

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

95 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

waving

Overview

Deficient domestic wastewater management, industrial waste, and floating debris are some leading factors that contribute to inland water pollution. The surplus of minerals and nutrients in overly contaminated zones can lead to the invasion of floating weeds. Lemnoideae, commonly known as duckweed, is a family of floating plants that has no leaves or stems and forms dense colonies with a fast growth rate. If not controlled, duckweed establishes a green layer on the surface and depletes fish and other organisms of oxygen and sunlight.

Consequently, we propose an open-source unmanned surface vehicle (USV) for automatic duckweed removal that costs less than $500. The USV uses 3D-printed parts and common components that can be easily purchased. It is worth mentioning that the proposed approach won the First Global Prize in the OpenCV AI Competition 2022, organized by the OpenCV Foundation. Moreover, further prototyping details and testing results are available in the paper:

Edwin Salcedo, Yamil Uchani, Misael Mamani, and Mariel Fernandez, Towards Continuous Floating Invasive Plant Removal Using Unmanned Surface Vehicles and Computer Vision, IEEE Access 2024.

[Paper] [Video Abstract]

USV

CAD Design

This section contains the CAD designs for the Unmanned Surface Vehicle (USV) proposed for the collection of duckweed. It includes all the parts that were used to 3D print the vehicle, such as the "Control Box", "Main Body", "Floaters", and "Main Grid".

Part Files

Assembly File

  • Assembly: The complete assembly of the USV.

Usage Instructions

The design was created in SolidWorks 2020, and it is recommended to use the same or a newer version for compatibility.

  1. Download the desired part or assembly files from the repository.
  2. Open the files using SolidWorks 2020 or a newer version.
  3. To view or modify individual parts, open the part (.SLDPRT) files.
  4. To view the complete assembly, open the Assembly.SLDASM file.

SAE

Framework

Architectural layout encapsulating the three most essential components:

  • A duckweed detection model embedded inside an OAK-D camera.
  • A steering angle classification model and a temporal weight module, namely momentum. Both were embedded inside a jetson nano development card.

Duckweed detection

The database used for this model was built by scrapping images from Google Images and the Global Biodiversity Information Facility. The resulting 4,000 images were grouped into five categories according to the view-point of the duckweed colonies inside the samples: close, near-close, near-wide, wide, and empty. The final dataset is available here (you'll need to create a IEEE Dataport account to access the dataset).

What follows is the list of experiments and trained models to replicate the results obtained using Yolov5 and the augmented versions of the dataset:

Yolo v5

Dataset Weights IoU mAP Accuracy
Only real Gdrive 0.7431 0.6426 0.7276
Only virtual Gdrive 0.8191 0.6487 0.8092
Real + Virtual Gdrive 0.8502 0.878 0.8327
Real + Virtual + Data Augmentation Gdrive 0.8625 0.866 0.8472

Yolo v8

Dataset Weights IoU mAP Accuracy
Only real Gdrive 0.8756 0.7779 0.8588
Only virtual Gdrive 0.874 0.6016 0.8889
Real + Virtual Gdrive 0.8085 0.6903 0.7878
Real + Virtual + Data Augmentation Gdrive 0.8942 0.7992 0.8796

Our data preprocessing approach, as well as our Yolov5 and Yolov8 implementations can be found here.

Steering angle classification

We tested two approaches for SAC: first, by collecting a syntethic dataset using a bespoke virtual environment and training DL classifiers. This approach, including object detection and moment, was named as M3. Then, we tried to classify steering angle using stereo vision along with object detection and momentum, which we later named M4.

The final dataset acquired for M3 can be shared upon request. The dataset weights 48.66 Gb in total, which prevented us to share it online. Yet, the results and pretrained models using general classifiers from TorchVision are available below:

Model Weights Accuracy Recall Precision Sensitivity F1-score Weight
Resnet50 Gdrive 0.794 0.793 0.966 0.792 0.794 90
Resnet101 Gdrive 0.812 0.812 0.969 0.787 0.812 162.7
EfficientNetv2m Gdrive 0.831 0.831 0.972 0.833 0.831 203.1
ConvNext Gdrive 0.736 0.735 0.956 0.734 0.736 334.2
ResNext Gdrive 0.824 0.826 0.971 0.825 0.824 331.9

Simulator

The project RecyclingRush required a simulator developed in Unity with the primary goal of resembling an underwater environment where a virtual agent collects water duckweeds. The simulator incorporates advanced features to enhance the agent's perception of the environment, such as the integration of Python-based deep learning models and stereo vision.

CAD Design Simulator

Installation

System Requirements

  • Unity: The simulator was created using version 2020.3.30f1, but is executable with newer versions.
  • Python: We used python's version 3.11.3. It is suggestable to use conda or pyenv to install Python and its dependencies and avoid collisions.
  • Windows: Note that the simulator was developed and tested with Windows 11.

Installation Steps

  1. Clone this repository to your local machine.

    git clone git@github.com:EdwinTSalcedo/RecyclingRush.git recyclingrush
    cd recyclingrush/vehicle_simulator
  2. Install the python dependencies for the detection models:

    pip install -r servers/requirements.txt
  3. Additionally, ensure that the following Python packages are installed:

     opencv-python-headless
     numpy
     matplotlib
     torch
     torchvision
     Pillow
     requests
     ultralytics

Note that the folders:

  • RecyclingRush contains the pre-packaged simulator, which is runnable using the executable file UnityCrashHandler64.exe.

  • Autonomous aquatic robot: contains the simulator's editable files.

  • servers: contains python scripts that accept input data and returns the estimated angle as for M1, M2, M3 and M4 self-driving modalities.

  1. Run the pre-packaged simulator (.exe) located in the "RecyclingRush" folder. This will create a folder named "!RecyclingRush" in your "Documents" directory.

  2. Inside the newly created "!RecyclingRush" folder, copy the entire "servers" folder. This step is crucial for both the simulator and the program to utilize the Python models.

  3. Load the folder RecyclingRush (inside recyclingrush/vehicle_simulator) with Unity.

Open Unity and load the Unity project by navigating to the "Robot_acuatico_autonomo" folder. If the project does not automatically initialize, follow these steps: - Click on "File" in the Unity editor. - Choose "Open Project." - Navigate to the "Robot_acuatico_autonomo" folder and open the project.

Usage

Three different modes are available in the main menu of the simulator, each of which serves a specific function in model training and evaluation.

Autonomous Driving: Used for data collection during autonomous driving model training. It captures image sequences, considering both the angle and speed of the vehicle.

Validation: Allows manual image capture, establishing a database for model validation. Useful for evaluating model performance in controlled and specific scenarios.

Evaluation: The activation of the four modes in this environment allows the evaluation of duckweed detection and steering models. It facilitates comprehensive evaluations and performance analysis under various conditions.

Main Menu

Self-driving environment

The self-driving environment is set in an aquatic setting with various elements simulating a lake, featuring aquatic plants and boats. The purpose of this environment is to capture a sequence of images, taking into account the boat's angle and speed at any given moment. The interface comprises two interactive elements and two informational elements.

Interface Elements:

  • Menu: This button allows users to navigate back to the main menu.

  • Record/Stop: Activates or deactivates the image capture process.

  • Angle: Displays the current steering angle of the boat. This information is reflected in the title of the captured image.

  • Velocity: Shows the speed of the boat at the specific moment. The velocity information is also included in the image title.

Main Menu

Validation Environment

The validation environment is designed to simulate a port in the aquatic lake, resulting in a more organized layout of elements. The primary objective is to manually capture images chosen by the user. Additionally, users can adjust the camera to capture different types of images, both underwater and head-on. The interface features four buttons for user interaction.

Interface Elements:

  • Menu: This button allows users to return to the main menu.

  • Capture: This button is responsible for capturing and saving the selected image.

  • Up: Adjusts the camera to face upwards.

  • Down: Adjusts the camera to face downwards.

Main Menu

Evaluation Environment

The evaluation environment is designed for model testing and stereo camera usage to optimize the collection of duckweed and avoid obstacles efficiently. It displays a well-organized section of the lake for data collection, particularly focusing on the duckweed gathered over specific time intervals. The environment allows adjusting robot modalities with different characteristics.

Interface Elements:

  • Menu: Returns to the main menu.

  • Restart: Resets the robot's journey to test different modalities or changes.

  • Reduce: Slows down the environment's time to capture journey details more effectively.

  • Increase: Speeds up the environment's time to accelerate the journey if necessary.

  • Camera: Changes the user's perspective on the boat.

  • Modality Selection: A dropdown menu presenting 5 options:

    • Manual: Default modality, allowing user manipulation of robot movement.

    • Modality 1: Utilizes only the duckweed detection model with a drastic angle change.

    • Modality 2: Utilizes only the duckweed detection model but with momentum for smoother angle changes.

    • Modality 3: Considers two models - duckweed detection and direction detection, resulting in smoothed angle changes.

    • Modality 4: Incorporates the duckweed detection model and uses a stereo camera to determine obstacles to avoid.

Main Menu

File Overview

Files C#

File Type Description
CameraRotation.cs Camera Adjust the vertical rotation of the camera in the validation environment to optimize image capture.
Cameras.cs Camera Switch between three camera types to change the perspective point for visualization.
CaptureImage.cs Camera Capture an image from the validation environment and save it in the computer's documents section.
ImageCapture.cs Camera Capture a series of images in the self-driving environment, including data such as robot speed and angle, and save the content in the computer's documents.
Depthclient.cs Clients Process three images simultaneously in each evaluation environment cycle for duckweed and obstacle detection using stereo vision techniques.
Effnetclient.cs Clients Process one image in each evaluation environment cycle for duckweed and obstacle detection using trained models.
DuckweedCounter.cs DataEvaluation Count the remaining and collected duckweeds in the evaluation environment.
TimeManager.cs DataEvaluation Track the elapsed time since the start of the evaluation environment journey, and provide controls to accelerate or decelerate it.
AngleText.cs Interface Display the robot's driving angle in the self-driving environment.
TextController.cs Interface Activate the sequence image capture process in the self-driving environment and rename the button accordingly.
UI_HoldButton.cs Interface Add an additional function to the buttons to detect continuous pressing.
VelocityScript.cs Interface Display the robot's speed in the self-driving environment.
ClearConsole.cs Load Menu Clear the Unity console each time an environment is initiated to prevent saturation
LoadingScreenController.cs Load Menu Asynchronously initiate the environment, generating elements before starting.
Closeprogram.cs Main Menu Close the application.
Create_documents.cs Main Menu Create a container folder for files generated in each environment.
Gameagain.cs Main Menu Save the startup configuration of the evaluation environment.
SelectWorld.cs Main Menu Change the scenario as specified.
Modality1.cs
Modality2.cs
Modality3.cs
Modality4.cs
Servers Activate their respective servers based on the chosen mode in the evaluation environment.
Nox.cs System Deactivate all elements in the current scenario.
Rest.cs System Activate the stereo detection client.
AutonomousMovement.cs Vehicle Modify vehicle movement based on data received from the server.
Floats.cs Vehicle Simulate aquatic movement for the robot.
Movement.cs Vehicle Handle user-initiated vehicle movement.

Files Python

File Type Description
Servermodality1.py
Servermodality2.py
Servermodality3.py
Depthserver.py
Servers The servers responsible for handling the processing of program images operate by utilizing models and stereo vision to detect duckweeds and their orientations.

Files Model

File Type Description
Best.pt Yolov8 The YOLOv8 model tasked with detecting lentils in the processed images.
Steering_estimator.pt Resnet The ResNet model designed to determine the correct angle for the robot to avoid obstacles.

Citing

If you find our work useful in your project, please consider to cite the following paper. Give RecyclingRush a star ⭐ on GitHub and share it with your friends and colleagues. With your support, we can continue to innovate in the field of autonomous vehicles for water quality management.

@article{salcedo2024,
  title={Towards Continuous Floating Invasive Plant Removal Using Unmanned Surface Vehicles and Computer Vision},
  author={Salcedo, Edwin and Uchani, Yamil and Mamani, Misael and Fernandez, Mariel},
  journal={IEEE Access},
  year={2024},
  publisher={IEEE},
  doi={10.1109/ACCESS.2024.3351764},
  url={https://doi.org/10.1109/ACCESS.2024.3351764}
}

About

RecyclingRush ♻️: Towards Continuous Floating Invasive Plant Removal Using Unmanned Surface Vehicles and Computer Vision, IEEE Access 2024.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •