Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards

Abstract

Ground vehicles equipped with monocular vision systems are a valuable source of high resolution image data for precision agriculture applications in orchards. This paper presents an image processing framework for fruit detection and counting using orchard image data. A general purpose image segmentation approach is used, including two feature learning algorithms; multi-scale Multi-Layered Perceptrons (MLP) and Convolutional Neural Networks (CNN). These networks were extended by including contextual information about how the image data was captured (metadata), which correlates with some of the appearance variations and/or class distributions observed in the data. The pixel-wise fruit segmentation output is processed using the Watershed Segmentation (WS) and Circular Hough Transform (CHT) algorithms to detect and count individual fruits. Experiments were conducted in a commercial apple orchard near Melbourne, Australia. The results show an improvement in fruit segmentation performance with the inclusion of metadata on the previously benchmarked MLP network. We extend this work with CNNs, bringing agrovision closer to the state-of-the-art in computer vision, where although metadata had negligible influence, the best pixel-wise F1-score of 0.791 was achieved. The WS algorithm produced the best apple detection and counting results, with a detection F1-score of 0.858. As a final step, image fruit counts were accumulated over multiple rows at the orchard and compared against the post-harvest fruit counts that were obtained from a grading and counting machine. The count estimates using CNN and WS resulted in the best performance for this dataset, with a squared correlation coefficient of r 2 = 0.826.

Semantic mapping for orchard environments by merging two‐sides reconstructions of tree rows

“…There has been a lot of recent study on yield estimation for specialty crops (Bargoti & Underwood, ; Das et al, ; Häni, Roy, & Isler, a,b; Roy & Isler, ,b, ; Q. Wang, Nuske, Bergerman, & Singh, ). Most of the existing systems rely on external sensors to register the fruits from a single side or both sides of the row.…”

Various Techniques Used In Defect Segmentation and Disease Inspection in Fruits and Vegetables: A Survey

“…Bargoti et al [7] uses this machine learning technique to train the object detector using apple images dataset which contains apple images of different variety. They have used this model to effectively count the number of Apples in an Image and thereby estimating the Apple's yield.The classifier they made takes as input a contextual window around individual pixels from the raw RGB image, with the windows sampled at different image scales.…”

“…Overfitting is addressed by using L2 minimization penalty. Apart from using MLP Bargoti et al [7] have used CNNs as well for detecting and counting the number of Apples in the Image. CNNs are feed forward neural networks, which combine and merge several different types of forward propagating layers, with convolutional layers playing an important role.…”

Image Based Mango Fruit Detection, Localisation and Yield Estimation Using Multiple View Geometry

“…This is a problem that is mentioned in the vast majority of the literature. Progress over the last five years from the machine vision community with convolutional neural networks (CNNs) has led to highly accurate fruit detection in colour imagery [15,16,17], and so arguably, the focus should shift towards fruit counting systems, which are designed to acquire and process orchard imagery in such a way that delivers the highest accuracy compared to the actual field and harvest fruit counts.…”

“…In the latter case, a process of calibration to manual field-counts can be done, which has proven to be accurate for some canopy types, including trellised apple orchards [5,6,15], almond orchards [7] and vineyards [18,19]. The calibration process requires manual field or harvest counts, which is labour intensive and would ideally be repeated every year.…”