A Proposal for Automatic Fruit Harvesting by Combining a Low Cost Stereovision Camera and a Robotic Arm

Abstract

This paper proposes the development of an automatic fruit harvesting system by combining a low cost stereovision camera and a robotic arm placed in the gripper tool. The stereovision camera is used to estimate the size, distance and position of the fruits whereas the robotic arm is used to mechanically pickup the fruits. The low cost stereovision system has been tested in laboratory conditions with a reference small object, an apple and a pear at 10 different intermediate distances from the camera. The average distance error was from 4% to 5%, and the average diameter error was up to 30% in the case of a small object and in a range from 2% to 6% in the case of a pear and an apple. The stereovision system has been attached to the gripper tool in order to obtain relative distance, orientation and size of the fruit. The harvesting stage requires the initial fruit location, the computation of the inverse kinematics of the robotic arm in order to place the gripper tool in front of the fruit, and a final pickup approach by iteratively adjusting the vertical and horizontal position of the gripper tool in a closed visual loop. The complete system has been tested in controlled laboratory conditions with uniform illumination applied to the fruits. As a future work, this system will be tested and improved in conventional outdoor farming conditions.

Sensors and systems for fruit detection and localization: A review

“…The major challenges in localizing fruit are the displacement of fruit by wind or other factors during imaging, and occlusion of fruit (Sarig, 1993). Numerous studies have been carried out to locate fruit in tree canopies with reasonable successes (Harrell et al, , 1990Slaughter and Harrell, 1989;Whittaker et al, 1987;Kondo and Kawamura, 1983;Mehta and Burks, 2014;Font et al, 2014). The sensors and methods used in the past for fruit localization in the agricultural environment are critically reviewed in the following sub-sections.…”

“…Image disparity is then converted to distance to objects from the camera using relative camera locations and orientations, and focal lengths of the cameras (Shapiro and Stockman, 2001). Researchers have widely used this method to simultaneously identify and locate fruit for harvesting or crop-load estimation (Font et al, 2014;Tanigaki et al, 2008;Kitamura and Oka, 2005). Plebe and Grasso (2001) installed stereo cameras in two arms of an orange harvesting robot.…”

Robust Grape Cluster Detection in a Vineyard by Combining the AdaBoost Framework and Multiple Color Components

“…Jimenez et al [4] adopted a laser-based computer vision system for fruit detection, which is based on an infrared laser range-finder sensor that provides range and reflectance images. Wang et al [5] and Font et al [6] used a binocular stereo vision to recognize and locate fruits in an unstructured environment with varying illumination. To detect apples in the canopy, Feng et al [7] designed a machine vision system consisting of a time-of-flight (ToF) camera and a digital color charge coupled device (CCD) camera.…”

A review of key techniques of vision-based control for harvesting robot

“…With the calculated movement duration of the end-effector, it can eliminate the time waiting for the oscillation to decay. Font et al (2014) also investigated a vision control strategy by combining open-loop visual control and visual closed-loop control. A stereovision camera mounted on a robot arm could acquire the initial fruit location.…”