Fast Method of Detecting Tomatoes in a Complex Scene for Picking Robots

Xu, Zhi-Feng; Jia, Rui-Sheng; Liu, Yanbo; Zhao, Chaoyue; Sun, Hong-Mei

doi:10.1109/access.2020.2981823

Cited by 51 publications

(25 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, machine learning, and especially deep learning, techniques in fruit detection has been increasingly used and tested [8,[15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32]. Unlike conventional methods, machine learning is a more robust and accurate alternative with a better response to problems such as occlusion and green tomato detection.…”

Section: State-of-the-art 21 Literature Reviewmentioning

confidence: 99%

“…In the most recent SoA, the interest in DL strategies has been growing [27][28][29][30][31][32]. This interest is due to the higher computability rate of the most recent computers and new edge computing devices dedicated to running DL models, as the TPU.…”

Section: State-of-the-art 21 Literature Reviewmentioning

confidence: 99%

“…This interest is due to the higher computability rate of the most recent computers and new edge computing devices dedicated to running DL models, as the TPU. Among the different DL architectures, the You Only Look Once (YOLO) models [40,41] seem to be the most common ones [27,28]. However, Convolutional Neural Network (CNN) structures also have their place due to their high accuracy, besides the long inference time [10].…”

Section: State-of-the-art 21 Literature Reviewmentioning

confidence: 99%

See 2 more Smart Citations

Evaluating the Single-Shot MultiBox Detector and YOLO Deep Learning Models for the Detection of Tomatoes in a Greenhouse

Magalhães

Castro

Moreira

et al. 2021

Sensors

View full text Add to dashboard Cite

The development of robotic solutions for agriculture requires advanced perception capabilities that can work reliably in any crop stage. For example, to automatise the tomato harvesting process in greenhouses, the visual perception system needs to detect the tomato in any life cycle stage (flower to the ripe tomato). The state-of-the-art for visual tomato detection focuses mainly on ripe tomato, which has a distinctive colour from the background. This paper contributes with an annotated visual dataset of green and reddish tomatoes. This kind of dataset is uncommon and not available for research purposes. This will enable further developments in edge artificial intelligence for in situ and in real-time visual tomato detection required for the development of harvesting robots. Considering this dataset, five deep learning models were selected, trained and benchmarked to detect green and reddish tomatoes grown in greenhouses. Considering our robotic platform specifications, only the Single-Shot MultiBox Detector (SSD) and YOLO architectures were considered. The results proved that the system can detect green and reddish tomatoes, even those occluded by leaves. SSD MobileNet v2 had the best performance when compared against SSD Inception v2, SSD ResNet 50, SSD ResNet 101 and YOLOv4 Tiny, reaching an F1-score of 66.15, an mAP of 51.46 and an inference time of 16.44ms with the NVIDIA Turing Architecture platform, an NVIDIA Tesla T4, with 12 GB. YOLOv4 Tiny also had impressive results, mainly concerning inferring times of about 5ms.

show abstract

Section: State-of-the-art 21 Literature Reviewmentioning

confidence: 99%

Section: State-of-the-art 21 Literature Reviewmentioning

confidence: 99%

Section: State-of-the-art 21 Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the Single-Shot MultiBox Detector and YOLO Deep Learning Models for the Detection of Tomatoes in a Greenhouse

Magalhães

Castro

Moreira

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…The detection of pomelo [44], kiwi fruit [45], waxberry [46], guava [47], and other fruits have been gradually concerned; With the development of deep learning, fruit flower detection, which is difficult to the traditional algorithm, has been emerging [48][49][50][51]. In the detection of vegetables, the improvement in the bounding box and the detection rate is the research focus of the tomato detection network [52][53][54]; Based on deep neural network, excellent results have been achieved in cucumber fruit length estimation [55], sweet pepper detection [56], date fruit variety and maturity judgment [57] and other aspects. In recent years, deep convolutional neural network has been applied in the banana plantation.…”

Section: B Research On Fruit and Vegetable Detectionmentioning

confidence: 99%

Fast and Accurate Detection of Banana Fruits in Complex Background Orchards

Duan

Zou

et al. 2020

IEEE Access

View full text Add to dashboard Cite

The detection of banana fruits is an important part of intelligent management in the banana plantation. To detect the banana fruit quickly and accurately in the complex orchard environment, this paper proposes a method based on the latest deep learning algorithm to detect the banana fruit. Using a monocular camera, we applied the YOLOv4 neural network algorithm to extract the deep features of banana fruits, realizing accurate detection of different banana sizes. The detection algorithm achieved a 99.29% detection rate, the average execution time was 0.171s, the shortest execution time was 0.135s, and the AP was 0.9995. Moreover, the detection results were discussed with the YOLOv3 algorithm and the machine learning algorithm. Compared with the machine learning algorithm, deep learning algorithm was superior to both detection accuracy and detection time. YOLOv4 had higher detection confidence and higher detection rate than YOLOv3. The results show that the proposed method could realize the fast detection of different varieties and different maturity in banana plantations, under different illumination and occlusion conditions, and provide information for banana picking, maturity and yield estimation. INDEX TERMS Banana detection, orchard environment, deep learning, green fruit, YOLOv4. L.Fu et al.: Fast and Accurate Detection of Banana Fruits in Complex Background Orchards L.Fu et al.: Fast and Accurate Detection of Banana Fruits in Complex Background Orchards

show abstract

“…Liu et al propose an improved tomato detection model based on YOLOv3 and use a circular bounding box for tomato localization, which contributes a 0.65% improvement on F1 score compared with the rectangular bounding box method [ 26 ]. Zhifeng et al propose an improved YOLOv3-tiny method to detect mature red tomatoes, the F1-Score is 12% higher than the original YOLOv3-tiny model [ 27 ]. To avoid poor performance of the trained model caused by insufficient diversity of dataset, in the image acquisition process, the above studies consider different environment conditions including light, weather, angles, occlusions, and so on.…”

Section: Introductionmentioning

confidence: 99%

Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach

Zhao

Liu

et al. 2021

Sensors

View full text Add to dashboard Cite

Since the mature green tomatoes have color similar to branches and leaves, some are shaded by branches and leaves, and overlapped by other tomatoes, the accurate detection and location of these tomatoes is rather difficult. This paper proposes to use the Mask R-CNN algorithm for the detection and segmentation of mature green tomatoes. A mobile robot is designed to collect images round-the-clock and with different conditions in the whole greenhouse, thus, to make sure the captured dataset are not only objects with the interest of users. After the training process, RestNet50-FPN is selected as the backbone network. Then, the feature map is trained through the region proposal network to generate the region of interest (ROI), and the ROIAlign bilinear interpolation is used to calculate the target region, such that the corresponding region in the feature map is pooled to a fixed size based on the position coordinates of the preselection box. Finally, the detection and segmentation of mature green tomatoes is realized by the parallel actions of ROI target categories, bounding box regression and mask. When the Intersection over Union is equal to 0.5, the performance of the trained model is the best. The experimental results show that the F1-Score of bounding box and mask region all achieve 92.0%. The image acquisition processes are fully unobservable, without any user preselection, which are a highly heterogenic mix, the selected Mask R-CNN algorithm could also accurately detect mature green tomatoes. The performance of this proposed model in a real greenhouse harvesting environment is also evaluated, thus facilitating the direct application in a tomato harvesting robot.

show abstract

Fast Method of Detecting Tomatoes in a Complex Scene for Picking Robots

Cited by 51 publications

References 24 publications

Evaluating the Single-Shot MultiBox Detector and YOLO Deep Learning Models for the Detection of Tomatoes in a Greenhouse

Evaluating the Single-Shot MultiBox Detector and YOLO Deep Learning Models for the Detection of Tomatoes in a Greenhouse

Fast and Accurate Detection of Banana Fruits in Complex Background Orchards

Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach

Contact Info

Product

Resources

About