Occluded Apple Fruit Detection and Localization with a Frustum-Based Point-Cloud-Processing Approach for Robotic Harvesting

Abstract

Precise localization of occluded fruits is crucial and challenging for robotic harvesting in orchards. Occlusions from leaves, branches, and other fruits make the point cloud acquired from Red Green Blue Depth (RGBD) cameras incomplete. Moreover, an insufficient filling rate and noise on depth images of RGBD cameras usually happen in the shade from occlusions, leading to the distortion and fragmentation of the point cloud. These challenges bring difficulties to position locating and size estimation of fruit for robotic harvesting. In this paper, a novel 3D fruit localization method is proposed based on a deep learning segmentation network and a new frustum-based point-cloud-processing method. A one-stage deep learning segmentation network is presented to locate apple fruits on RGB images. With the outputs of masks and 2D bounding boxes, a 3D viewing frustum was constructed to estimate the depth of the fruit center. By the estimation of centroid coordinates, a position and size estimation approach is proposed for partially occluded fruits to determine the approaching pose for robotic grippers. Experiments in orchards were performed, and the results demonstrated the effectiveness of the proposed method. According to 300 testing samples, with the proposed method, the median error and mean error of fruits’ locations can be reduced by 59% and 43%, compared to the conventional method. Furthermore, the approaching direction vectors can be correctly estimated.

Immature Green Apple Detection and Sizing in Commercial Orchards Using YOLOv8 and Shape Fitting Techniques

“…Several studies in the past explored sphere fitting approach in 3D environment to estimate fruit size of peaches [34], guava [35], apples [36], pomegranate [37], tomato [38] and citrus [39]. Likewise, ellipsoid fitting technique have been explored to estimate the size of non-spherical fruits such as banana [40], watermelon [41] and mushroom [42].…”

A review on the application of computer vision and machine learning in the tea industry

“…BS cameras use the parallax principle to calculate the depth indirectly through the algorithm. While RGB-D cameras measure depth information more directly through laser, which is mainly divided into cameras based on the structured light (SL) principle, cameras based on the time of Flight (ToF) principle, and active infrared stereo (AIRS) cameras (Li T. et al, 2022).…”

An Improved Apple Object Detection Method Based on Lightweight YOLOv4 in Complex Backgrounds

“…The data presented in Table 6 were derived at IoU = 0.50:0.95. The subscripts are defined as follows: S means small target (area ≤ 32 2 ), M means medium target (32 2 < area ≤ 96 2 ), L means large target (area > 96 2 ), and area denotes the number of pixels [57]. The experiments detailed in this section were conducted to validate the enhanced effect of our model on the small target detection ability.…”

“…The fresh fruit industry is labor-intensive, with apple-picking labor costs accounting for over 50% of total costs [1,2]. As the aging population expands, the working population of China is shrinking, and labor costs are rising.…”