Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN

Automation of agricultural processes requires systems that can accurately detect and classify produce in real industrial environments that include variation in fruit appearance due to illumination, occlusion, seasons, weather conditions, etc. In this paper we combine a visual processing approach inspired by colour-opponent theory in humans with recent advancements in one-stage deep learning networks to accurately, rapidly and robustly detect ripe soft fruits (strawberries) in real industrial settings and using standard (RGB) camera input. The resultant system was tested on an existent data-set captured in controlled conditions as well our new real-world data-set captured on a real strawberry farm over two months. We utilise F 1 score, the harmonic mean of precision and recall, to show our system matches the state-of-the-art detection accuracy ( F 1 : 0.793 vs. 0.799) in controlled conditions; has greater generalisation and robustness to variation of spatial parameters (camera viewpoint) in the real-world data-set ( F 1 : 0.744); and at a fraction of the computational cost allowing classification at almost 30fps. We propose that the L*a*b*Fruits system addresses some of the most pressing limitations of current fruit detection systems and is well-suited to application in areas such as yield forecasting and harvesting. Beyond the target application in agriculture this work also provides a proof-of-principle whereby increased performance is achieved through analysis of the domain data, capturing features at the input level rather than simply increasing model complexity.

show abstract

“…Yu et al [34] 1900 Side on (Close) Chen et al [35] 12526 Aerial Lamb and Chuah [36] 4550 Ground Ge et al [37] -Side on Sa et al [8] 122…”

Section: Methods # Images Availability Viewpoint Multi Spectra Controlmentioning

confidence: 99%

Lab*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks

Kirk

Cielniak

Mangan

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Mask R-CNN extends the target detection framework of Faster R-CNN [45] by adding a masking branch at the end of the model [38]. This process ensures that each output instance segments the proposal box using a fully connected (FC) layer to ensure that the segmentation is parallel to the target detection.…”

Section: Structure Of Mask R-cnn Modementioning

confidence: 99%

“…The weed target in this study is small, especially in the central area. The size of the weeds changes throughout the growth cycle [38].…”

Section: Roi Generation and Roialign (Rpn+roialign)mentioning

confidence: 99%

“…Although a deeper number of network layers may lead to higher accuracy, the deeper network layers will result in lower model training and detection speeds. However, since the residual network does not increase the parameters of the model, it can effectively reduce the problem of training degradation and improve the model convergence[38]. Therefore, this paper uses ResNet50 and ResNet101 combined with the FPN as the backbone network to extract the features of weed images.…”

mentioning

confidence: 99%

“…A total of fifteen anchor points were selected, and the most likely target area was selected for segmentation and detection. ROIAlign uses bilinear interpolation to extract the corresponding features of each ROI on the feature map[38], calculates the exact value of each position, summarizes it by the maximum pooling method, adjusts the dimensions of each ROI to meet the FC requirement, and finally sends it to the FC layer and FCN for target classification, the bounding boxes and the mask.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Method for Obtaining Phenotypic Traits of Weeds in Complex Field Environments Based on Instance Segmentation

Quan

Mao

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: Weeds are the biggest threat to crop growth, and leaf age and central area of weeds are important phenotypic traits of weeds. They have an important role in understanding the morphological structure of weeds, guiding precision for target weeding and reducing the use of herbicides. However, it is still a substantial challenge to obtain weed types, leaf age and central area in the complex field conditions of light changes, variation in appearance of plants, leaf occlusion. The latest developments in deep learning provide new tools for solving challenging computer vision tasks.Results: In this study, we present a weed phenotype segmentation method based on Mask R-CNN that obtains weed types, leaf age and central area in the complex field conditions. By shooting three different angles of the main weeds (Solanum nigrum, Barnyard grass, and Abutilon theophrasti Medicus) in the field through mobile devices, two datasets (data enhancement and without data enhancement) were produced and used as input to the network, two backbone networks was tested, namely ResNet50 and ResNet101, and the detection results and instance segmentation results of the model were evaluated. The results showed that data enhancement can improve the performance of the model. In the case of data enhancement, the F1 value with ResNet101 as the backbone network was 0.9214, the mAP scores were 0.6932 and 0.5244 (for IOU thresholds of 0.5 and 0.7, respectively), the mIOU reached 0.585, and the best segmentation performance example was obtained. Furthermore, the weed image taken from the top view angle compared to the other two angles achieved the highest detection accuracy.Conclusion: Mask R-CNN can achieve accurate segmentation of weeds to obtain the types, leaf age and central area of weeds. Data enhancement and the weed image taken from the top view angle can help to improve the performance of the model. This dataset and research results may provide important resources for the development of precision agriculture in the future.

show abstract

A modified Mask region‐based convolutional neural network approach for the automated detection of archaeological sites on high‐resolution light detection and ranging‐derived digital elevation models in the North German Lowland

Bonhage

Eltaher

Raab

et al. 2021

Archaeological Prospection

View full text Add to dashboard Cite

Due to complicated backgrounds and unclear target orientation, automated object detection is difficult in the field of archaeology. Most of the current convolutional neural network (CNN) object-oriented detection techniques are based on a faster region-based CNN (R-CNN) and other one-stage detectors that often lack adequate processing speeds and detection accuracies. Recently, the two-stage detector Mask R-CNN technique achieved impressive results in object detection and instance segmentation problems and was successfully applied in the analysis of archaeological airborne laser scanning (ALS) data. In this study, we outline a modified Mask R-CNN technique that reliably and efficiently detects relict charcoal hearth (RCH) sites on light detection and ranging (LiDAR) data-based digital elevation models (DEMs).Using image augmentation and image preprocessing steps combined with the deep learning-based adaptive gradient method with a dynamic bound on the learning rate (AdaBound) optimization technique, we could improve the model's accuracy and significantly reduce its training time. We use DEMs based on high-resolution LiDAR data and the visualization for archaeological topography (VAT) technique that give images with a very strong contrast of the terrain and the outline of the sites of interest in the North German Lowland. Therefore, the model can identify RCH sites with an average recall of 83% and an average precision of 87%. Techniques such as the modified Mask R-CNN method outlined here will help to greatly improve our knowledge about archaeological site densities in the realm of historical charcoal production and past human-landscape interactions. This method provides an accurate, time-efficient and bias-free large-scale site mapping option not only for the North German Lowland but potentially for other landscapes as well.

show abstract

Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN

Cited by 462 publications

References 14 publications

Lab*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks

Lab*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks

Method for Obtaining Phenotypic Traits of Weeds in Complex Field Environments Based on Instance Segmentation

A modified Mask region‐based convolutional neural network approach for the automated detection of archaeological sites on high‐resolution light detection and ranging‐derived digital elevation models in the North German Lowland

Contact Info

Product

Resources

About

Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN

Cited by 462 publications

References 14 publications

L*a*b*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks

L*a*b*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks

Method for Obtaining Phenotypic Traits of Weeds in Complex Field Environments Based on Instance Segmentation

A modified Mask region‐based convolutional neural network approach for the automated detection of archaeological sites on high‐resolution light detection and ranging‐derived digital elevation models in the North German Lowland

Contact Info

Product

Resources

About

Lab*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks

Lab*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks