Intelligent robots for fruit harvesting: recent developments and future challenges

Zhou, Hongyu; Wang, Xing; Au, Wesley; Kang, Hanwen; Chen, Chao

doi:10.1007/s11119-022-09913-3

Cited by 96 publications

(37 citation statements)

References 106 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Results show that imbalance class problems could lead to severe performance degeneration in large-scale dataset training. Increasing the minimum number of fruits point in 2 No. pts is the minimum fruit points in training samples.…”

Section: Experiments On Training Strategymentioning

confidence: 99%

“…With proper extrinsic calibration of the cameras and LiDARs, the point cloud with texture information of the real world can be obtained. Semantic scene understanding is a fundamental task in many agricultural scenarios, such as growth monitoring and robotics perception tasks [2]- [4]. However, semantic Identify applicable funding agency here.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards

Kang¹,

Wang²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Semantic segmentation is a fundamental task for agricultural robots to understand the surrounding environments in natural orchards. The recent development of the LiDAR techniques enables the robot to acquire accurate range measurements of the view in the unstructured orchards. Compared to RGB images, 3D point clouds have geometrical properties. By combining the LiDAR and camera, rich information on geometries and textures can be obtained. In this work, we propose a deep-learning-based segmentation method to perform accurate semantic segmentation on fused data from a LiDAR-Camera visual sensor. Two critical problems are explored and solved in this work. The first one is how to efficiently fused the texture and geometrical features from multi-sensor data. The second one is how to efficiently train the 3D segmentation network under severely imbalance class conditions. Moreover, an implementation of 3D segmentation in orchards including LiDAR-Camera data fusion, data collection and labelling, network training, and model inference is introduced in detail. In the experiment, we comprehensively analyze the network setup when dealing with highly unstructured and noisy point clouds acquired from an apple orchard. Overall, our proposed method achieves 86.2% mIoU on the segmentation of fruits on the high-resolution point cloud (100k-200k points). The experiment results show that the proposed method can perform accurate segmentation in real orchard environments.

show abstract

Section: Experiments On Training Strategymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards

Kang¹,

Wang²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, these achievements in robotic harvesting have yet to secure the level of productivity required to see widespread adoption. In the state-of-the-art, robotic harvesters encounter significant challenges in the presence of unstructured features such as occlusions and obstructions from leaves [ 11 ], twigs, and branches [ 12 ]. Branches and twigs could affect the accessibility of the target fruit but could be avoided by well-tuned robot path planning and manipulation [ 13 ].…”

Section: Introductionmentioning

confidence: 99%

Learning-Based Slip Detection for Robotic Fruit Grasping and Manipulation under Leaf Interference

Zhou

Xiao

Kang

et al. 2022

Sensors

Self Cite

View full text Add to dashboard Cite

Robotic harvesting research has seen significant achievements in the past decade, with breakthroughs being made in machine vision, robot manipulation, autonomous navigation and mapping. However, the missing capability of obstacle handling during the grasping process has severely reduced harvest success rate and limited the overall performance of robotic harvesting. This work focuses on leaf interference caused slip detection and handling, where solutions to robotic grasping in an unstructured environment are proposed. Through analysis of the motion and force of fruit grasping under leaf interference, the connection between object slip caused by leaf interference and inadequate harvest performance is identified for the first time in the literature. A learning-based perception and manipulation method is proposed to detect slip that causes problematic grasps of objects, allowing the robot to implement timely reaction. Our results indicate that the proposed algorithm detects grasp slip with an accuracy of 94%. The proposed sensing-based manipulation demonstrated great potential in robotic fruit harvesting, and could be extended to other pick-place applications.

show abstract

“…Robotic harvesting of fruits has shown significant progress in recent development of agricultural industry [1]. A general procedure for robotic fruit picking requires the localisation then detachment from trees [2]. Under the highly complex and unstructured environments in orchards, the accuracy of the fruit localisation is crucial to the performance of the robotic harvesting.…”

Section: Introductionmentioning

confidence: 99%

Accurate Fruit Localisation for Robotic Harvesting using High Resolution LiDAR-Camera Fusion

Kang¹,

Wang²,

Chen³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Accurate depth-sensing plays a crucial role in securing a high success rate of robotic harvesting in natural orchard environments. Solid-state LiDAR (SSL), a recently introduced LiDAR technique, can perceive high-resolution geometric information of the scenes, which can be potential utilised to receive accurate depth information. Meanwhile, the fusion of the sensory information from LiDAR and camera can significantly enhance the sensing ability of the harvesting robots. This work introduces a LiDAR-camera fusion-based visual sensing and perception strategy to perform accurate fruit localisation for a harvesting robot in the apple orchards. Two SOTA extrinsic calibration methods, target-based and targetless-based, are applied and evaluated to obtain the accurate extrinsic matrix between the LiDAR and camera. With the extrinsic calibration, the point clouds and color images are fused to perform fruit localisation using a one-stage instance segmentation network. Experimental shows that LiDAR-camera achieves better quality on visual sensing in the natural environments. Meanwhile, introducing the LiDAR-camera fusion largely improves the accuracy and robustness of the fruit localisation. Specifically, the standard deviations of fruit localisation by using LiDAR-camera at 0.5 m, 1.2 m, and 1.8 m are 0.245, 0.227, and 0.275 cm respectively. These measurement error is only one one fifth of that from Realsense D455. Lastly, we have attached our visualised point cloud 3 to demonstrate the highly accurate sensing method.

show abstract

Intelligent robots for fruit harvesting: recent developments and future challenges

Cited by 96 publications

References 106 publications

Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards

Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards

Learning-Based Slip Detection for Robotic Fruit Grasping and Manipulation under Leaf Interference

Accurate Fruit Localisation for Robotic Harvesting using High Resolution LiDAR-Camera Fusion

Contact Info

Product

Resources

About