AK_ACQS is a software solution for data acquisition in fruit orchards using a sensor system boarded on a terrestrial vehicle. It allows the coordination of computers and sensors through the sending of remote commands via a GUI. At the same time, it adds an abstraction layer on library stack of each sensor, facilitating its integration. This software solution is supported by a local area network (LAN), which connects computers and sensors from different manufacturers ( cameras of different technologies, GNSS receiver) for in-field fruit yield testing.
- Pre-requisites.
- Functionalities developed.
- Files and folder description.
- Development tools and environment.
This section explains aspects related to the pre-requisites needed to run AK_ACQS, hardware and software used in the experiments.
The following figure shows an example configuration, which will be used in other sections as a starting point for the explanations.
- MSI Modern 15 A10RBS-484XES (New Taipei, Zhonghe Dist) Computer 1.
- Jetson Xavier NX (NVIDIA, Santa Clara, America) Computer 2.
- LAN (Local Area Network) to connect computers.
- Azure Kinect DK camera connected to the computer.
- GNSS receiver Ardusimple SimpleRTK2B – Basic Starter Kit .
- Stereolab ZED 2 camera connected to the computer.
- Canonical Ubuntu 20.04.
- Jetson Pack.
- SDK Azure Kinect installed.
- Stereolabs SDK installed.
- pyk4a library installed. If the operating system is Windows, follow this steps. You can find test basic examples with pyk4a here.
The functionalities of AK_ACQS consist of remotely enabling and disabling clients, taking snapshots, starting and stopping video recordings, as well as logging latitude and longitude coordinates during the video recording time.
- [ENABLE REMOTE CLIENTS] allows the user to send an attention call to the devices to configure them in the initial state of listening to orders.
- [TAKE CAPTURES] makes it easy for the user to capture short videos, automatically starting and stopping video or snapshot recording.
- [START RECORDING/ STOP RECORDING] is the functionality that allows to send start and stop recording messages to all connected clients. Remote clients managing a GNSS receiver will start/stop operations for coordinate capture.
- [DISABLE REMOTE CLIENTS], with this function the user remotely turns off the devices. These will stop operating when receiving and processing the message. The recorded files are stored on the host computers, just like the data collected by the GNSS receiver.
| Folder | Description |
|---|---|
| docs/ | Documentation and explanations. |
| remote_client_generic/ | This is a template for future development, it is a dummy client that can be used to test the server connection. It can also be used to extend the functionalities of other devices. |
| remote_client_ak/ | Code to manage Azure Kinect DK camera |
| remote_client_zed/ | Code to manage ZED 2 camera and Ardusimple GNSS receiver |
| remote_management_console/ | Desktop GUI based on Tkinter library. Offers the possibility of sending instructions to the remote devices. |
| server_rest_api/ | The server acts as an intermediary in the management of messages between remote clients and the management console, and stores information about of the instructions sent and received. It uses SQLite database. |
From Linux systems, run this script at the command line to automatically create directory hierarchies for the project's development environments. Then use the "create_env_xxxxxx.sh" scripts for each component to load the Python dependencies.
git clone https://github.com/GRAP-UdL-AT/ak_acquisition_system.git
Documents and explanations accompanying AK_ACQS.
| Folder | Description |
|---|---|
| REST API test/ | Explanations about testing REST API. |
| Notes for developers | Explanations about development environment. |
| Azure Kinect camera setup | Instructions to set up the Azure Kinect camera. |
| Stereolabs ZED 2 camera setup | Instructions to set up the ZED 2 camera. |
Distribution of AK_ACQS components.
| Package type | Package | Url | Description |
|---|---|---|---|
| Virtual environment | N/A | N/A | All components of this software run in separate virtual environments. |
The following table shows each component of AK_ACQS and the operating systems on which they have been tested.
| Component | Linux | Windows | Jetpack |
|---|---|---|---|
| remote_client_generic/ | YES | YES | YES |
| remote_client_ak/ | YES | YES | N/T |
| remote_client_zed/ | YES | N/T | YES |
| remote_management_console/ | YES | YES | N/T |
| server_rest_api/ | YES | N/T | YES |
References:
- N/T --> The software has not yet been tested under the operating system at the time of publication.
- YES --> Tested in operating system.
- NO --> Support for this operating system is not yet available.
This project is contributed by GRAP-UdL-AT. Please contact authors to report bugs juancarlos.miranda@udl.cat
If you find this code useful, please consider citing:
@article{MIRANDA2022101231,
title = {AKFruitData: A dual software application for Azure Kinect cameras to acquire and extract informative data in yield tests performed in fruit orchard environments},
journal = {SoftwareX},
volume = {20},
pages = {101231},
year = {2022},
issn = {2352-7110},
doi = {https://doi.org/10.1016/j.softx.2022.101231},
url = {https://www.sciencedirect.com/science/article/pii/S2352711022001492},
author = {Juan Carlos Miranda and Jordi Gené-Mola and Jaume Arnó and Eduard Gregorio},
keywords = {RGB-D camera, Data acquisition, Data extraction, Fruit yield trials, Precision fructiculture},
abstract = {The emergence of low-cost 3D sensors, and particularly RGB-D cameras, together with recent advances in artificial intelligence, is currently driving the development of in-field methods for fruit detection, size measurement and yield estimation. However, as the performance of these methods depends on the availability of quality fruit datasets, the development of ad-hoc software to use RGB-D cameras in agricultural environments is essential. The AKFruitData software introduced in this work aims to facilitate use of the Azure Kinect RGB-D camera for testing in field trials. This software presents a dual structure that addresses both the data acquisition and the data creation stages. The acquisition software (AK_ACQS) allows different sensors to be activated simultaneously in addition to the Azure Kinect. Then, the extraction software (AK_FRAEX) allows videos generated with the Azure Kinect camera to be processed to create the datasets, making available colour, depth, IR and point cloud metadata. AKFruitData has been used by the authors to acquire and extract data from apple fruit trees for subsequent fruit yield estimation. Moreover, this software can also be applied to many other areas in the framework of precision agriculture, thus making it a very useful tool for all researchers working in fruit growing.}
}
This work is a result of the RTI2018-094222-B-I00 project (PAgFRUIT) granted by MCIN/AEI and by the European Regional Development Fund (ERDF). This work was also supported by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya under Grant 2017-SGR-646. The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and Fons Social Europeu (FSE) are also thanked for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. The authors would also like to thank the Institut de Recerca i Tecnologia Agroalimentàries (IRTA) for allowing the use of their experimental fields, and in particular Dr. Luís Asín and Dr. Jaume Lordán who have contributed to the success of this work.
