Bringing Semantics to the Vineyard: An Approach on Deep Learning-Based Vine Trunk Detection

Abstract

The development of robotic solutions in unstructured environments brings several challenges, mainly in developing safe and reliable navigation solutions. Agricultural environments are particularly unstructured and, therefore, challenging to the implementation of robotics. An example of this is the mountain vineyards, built-in steep slope hills, which are characterized by satellite signal blockage, terrain irregularities, harsh ground inclinations, and others. All of these factors impose the implementation of precise and reliable navigation algorithms, so that robots can operate safely. This work proposes the detection of semantic natural landmarks that are to be used in Simultaneous Localization and Mapping algorithms. Thus, Deep Learning models were trained and deployed to detect vine trunks. As significant contributions, we made available a novel vine trunk dataset, called VineSet, which was constituted by more than 9000 images and respective annotations for each trunk. VineSet was used to train state-of-the-art Single Shot Multibox Detector models. Additionally, we deployed these models in an Edge-AI fashion and achieve high frame rate execution. Finally, an assisted annotation tool was proposed to make the process of dataset building easier and improve models incrementally. The experiments show that our trained models can detect trunks with an Average Precision up to 84.16% and our assisted annotation tool facilitates the annotation process, even in other areas of agriculture, such as orchards and forests. Additional experiments were performed, where the impact of the amount of training data and the comparison between using Transfer Learning and training from scratch were evaluated. In these cases, some theoretical assumptions were verified.

Visible and Thermal Image-Based Trunk Detection with Deep Learning for Forestry Mobile Robotics

“…In forestry and agricultural mobile robotics, the robot visual perception of the environment is a matter of the utmost importance, where several challenges can appear, such as trees, bushes, boulders, holes and rough terrain (with mud, rocks, etc.). In the agricultural context, there are plenty of works and studies related to image-based woody trunk detection, specially for performing SLAM [ 17 , 18 , 19 , 20 ]. On the other hand, in the forestry context, there are some works that focused on image-based forest tree detection.…”

A Review of the Challenges of Using Deep Learning Algorithms to Support Decision-Making in Agricultural Activities

“…The lightweight models enable to use mobile and low cost edge devices such as Jetson nano (https://developer.nvidia. com/embedded/jetson-nano-developer-kit, accessed on 8 November 2021), Raspberrypi (https://www.raspberrypi.com/, accessed on 15 November 2021) and Google TPU Coral (https://coral.ai/, accessed on 8 November 2021) to deploy the trained model in robots and use them in the field [11,18] (Figure 3).…”

“…Aguiar et al [72] trained SSD MobilenetV1 and the Inception model to detect Grape Bunch in Mid Stage and early stages and then transferred the trained model to the TPU-Edge device to investigate the temporal accuracy of the model. The same strategy in [18] was used to collect the dataset and it is publicly available (https://doi.org/10.5281/zenodo. 5114142, accessed on 3 November 2021) with 1929 vineyard images and their annotation.…”

Unimodal and Multimodal Perception for Forest Management: Review and Dataset

“…The detection of vegetables or fruits are also important to count them and estimate the production yield [110][111][112][113]. Similarly to forestry contexts, some works are about disease detection and monitoring [114,115], and others are focused on detection woody trunks, weeds, and general obstacles in crops for navigation [116][117][118][119], operation purposes [120,121], and cleaning tasks [122,123]. Another application of perception systems in precision agriculture is characterising, monitoring, and phenotyping vegetative cultures using stereo vision [124,125], point clouds [126,127], satellite imagery [128], low-altitude aerial images [129], or multispectral imagery [130].…”