Semantic mapping for orchard environments by merging two‐sides reconstructions of tree rows

Abstract

Measuring semantic traits for phenotyping is an essential but labor‐intensive activity in horticulture. Researchers often rely on manual measurements which may not be accurate for tasks, such as measuring tree volume. To improve the accuracy of such measurements and to automate the process, we consider the problem of building coherent three‐dimensional (3D) reconstructions of orchard rows. Even though 3D reconstructions of side views can be obtained using standard mapping techniques, merging the two side‐views is difficult due to the lack of overlap between the two partial reconstructions. Our first main contribution in this paper is a novel method that utilizes global features and semantic information to obtain an initial solution aligning the two sides. Our mapping approach then refines the 3D model of the entire tree row by integrating semantic information common to both sides, and extracted using our novel robust detection and fitting algorithms. Next, we present a vision system to measure semantic traits from the optimized 3D model that is built from the RGB or RGB‐D data captured by only a camera. Specifically, we show how canopy volume, trunk diameter, tree height, and fruit count (FC) can be automatically obtained in real orchard environments. The experiment results from multiple data sets quantitatively demonstrate the high accuracy and robustness of our method.

A comparative study of fruit detection and counting methods for yield mapping in apple orchards

“…For fruit counting, the CNN approach was more accurate for both single‐image data sets and yield estimation. Additionally, together with our recent work (Dong et al, ; Roy, Dong, et al, ), we presented a complete system for yield estimation. The classical segmentation method combined with the CNN based counting approach achieved yield accuracies ranging from 95.56% to 97.83% compared to the harvested ground truth.…”

“…This component takes the single‐side reconstructions, and multiview fruit counts as inputs. It merges the input reconstructions from both sides using semantic information (Dong et al, ; Roy, Dong, et al, ), eliminates double counting owing to fruits visible from both sides of the tree, and outputs the total fruit count for the captured portion of the row. Again, for completion, this component is briefly reviewed in Section 4.4.…”

“…Many approaches strive to solve this problem-Some rely on specialized sensors and hardwares (Das et al, 2015;Gongal et al, 2016), and some correlate fruit counts from a single side to ground truth (Stein et al, 2016). In contrast to these works, we follow the approach proposed by Dong, Roy, and Isler (2018) and Roy, Dong, et al (2018). Their system detects and counts apples in individual images, and the fruits are tracked across an image sequence using estimated camera motion.…”

“…The tracking component takes the entire sequence of images from a single side of a tree row and the per‐frame fruit detections and counts as input. We use the previously proposed method in Dong et al () and Roy, Dong, et al () for fruit tracking. For completion, we review this component in Section 4.4.…”

Monocular Camera Based Fruit Counting and Mapping With Semantic Data Association

“…Roy et al develop a four-step 3D reconstruction method which first roughly aligns 3D point cloud two-side view of fruit tree row, then generates semantic representation with deep learning-based trunk segmentations and further refines two-view alignment with this data. At backend it uses the 3-D point cloud and pre-detected fruits from [7] to give both visual count and tree height and size estimation for harvest count estimation [14].…”