Research on Spatial Positioning System of Fruits to be Picked in Field Based on Binocular Vision and SSD Model

Abstract

The accurate fruit recognition in the field was one of the key technologies of fruit picking agricultural robots. An improved Single Shot Multi-Box Detector (SSD) model based on the color and morphological characteristics of fruit was proposed in this paper when aimed at the large collection workload and low secondary transfer efficiency of fruit such as palm fruit, durian, pineapple and other fruits grown in a complex field environment. A binocular depth camera RealSense D435i was used to collect images of the fruit to be picked in the field. Meanwhile, the MobileNet was replaced with the VGG16 basic network based on the Tensor-flow deep learning framework to reduce the amount of convolution operations for extracting image features in the SSD model, and a spatial positioning system for pineapple fruit was designed. Furtherly, experiments showed that the improved SSD depth detection model had a smaller size and it was more convenient to be deployed on the mobile end of agricultural robots, which the model had a high accuracy in the effective recognition of the fruits to be picked under the weed occlusion and overlapping scenes. The frame rate of the video reading and detection for the binocular depth camera reached 16.74 Frames Per Second (FPS), which had good robustness and real-time, and a good solution for the automatic picking of agricultural picking robots could be provided in the field.

Multi-Target Recognition of Bananas and Automatic Positioning for the Inflorescence Axis Cutting Point

“… Santos et al (2020) detected grapes using three networks, namely, Mask R-CNN, YOLOv2, and YOLOv3, with the F1-score being 0.91. Zhang et al (2021) described a modified SSD detector based on fruit color and morphological features. The frame rate of the stereo depth camera for detecting palm fruits, durian fruits, and pineapples reached 16.71 frames per second ( Zhang et al, 2021 ).…”

“… Zhang et al (2021) described a modified SSD detector based on fruit color and morphological features. The frame rate of the stereo depth camera for detecting palm fruits, durian fruits, and pineapples reached 16.71 frames per second ( Zhang et al, 2021 ). Wang et al (2021) described a modified YOLOv3-Litchi model for detecting densely distributed lychee fruits in a large visual scene, where the mean precision was 87.43%.…”

Identification and picking point positioning of tender tea shoots based on MR3P-TS model

“…In order to apply the model to the actual mechanized tea picking task, the model can be deployed to the development board, and the corresponding mechanical structure and complete visual recognition system can be designed for the tea picking machine to realize the fine picking of tender shoots. At present, some literature has been reported to use binocular depth cameras to collect field fruit images, and design a fruit spatial positioning system (Zhang et al, 2021). Some other researchers have proposed methods of connecting the manipulator with the tea picking machine, which provides theoretical support for this line of thinking (Yang et al, 2021).…”

A Study on Long-Close Distance Coordination Control Strategy for Litchi Picking

“…The advent of depth cameras offers an economical solution for building three-dimensional (3D) optical coordinate measurement systems. Compact dimensions, low cost, and ease of secondary development yield such cameras, an increasingly popular hardware option for fruit-picking robots [23]. Liu et al proposed a strategy for recognizing and locating citrus fruits in a close shot-based strategy on a Realsense F200 Camera (Intel Corp., Santa Clara, CA, USA).…”