Robotized Plant Probing: Leaf Segmentation Utilizing Time-of-Flight Data

Abstract

Abstract-Supervision of long-lasting extensive botanic experiments is a promising robotic application that some recent technological advances have made feasible. Plant modelling for this application has strong demands, particularly in what concerns 3D information gathering and speed. This paper shows that Time-ofFlight (ToF) cameras achieve a good compromise between both demands. A new method is proposed to segment plant images into their composite surface patches by combining a hierarchical segmentation of the infrared intensity image, provided by the ToF camera, with quadratic surface fitting using ToF depth data. Leaf models are fitted to the segments and used to find candidate leaves for probing. The candidate leaves are ranked, and then the robot-mounted camera moves closer to selected leaves to validate their suitability to being sampled. Some ambiguities arising from leaves overlap or occlusions are cleared up in this way. Suitable leaves are then probed using a special cutting tool also mounted on the robot arm. The work is a proof-of-concept that dense infrared data combined with sparse depth as provided by a ToF camera yields a good enough 3D approximation for automated cutting of leaf discs for experimentation purposes.

ToF cameras for active vision in robotics

“…Color vision is helpful to extract some relevant features, but it is not well-suited for providing the structural/geometric information indispensable for robot interaction with plants. 3D cameras are, thus, a good complement, since they directly provide depth images [55]. Moreover, plant data acquired from a given viewpoint are often partial due to self-occlusions, thus planning the best next viewpoint becomes an important requirement.…”

A cognitive architecture for automatic gardening

“…Third, a correction is performed to merge segments corresponding to the same leaf. This correction is based on the convex hull approximation using depth information, which is derived from the disparity map obtained by stereo matching [2]. The accuracy of the perception system depends on the inclination of the leaf and is measured as the validity (correlation between the measured contour and a leaf model) of the extracted segments.…”

“…The architecture integrates artificial intelligence techniques for decision-making with robotics techniques for sensing and acting. Low-level robotic approaches comprises short-term interaction of the robot with plants entailing 3D model acquisition of deformable objects (leaves) [2] and robot-arm manipulation approaches for active vision and individual treatment application [3].…”

3D Sensor planning framework for leaf probing

“…However, sensors in the measuring chamber are in a pre-defined position, and measurements are sometimes obtained from an inadequate point of view. Additionally, one of the main limitations is the difficulty to measure or perform actions that require contact with the plant, such as chlorophyll measurement or the extraction of disk samples for DNA analysis [4].…”

“…The first step is usually specified by a botanical expert that defines a criterion to choose the leaf, for example the biggest one, or the i-th leaf from the stem. The last step, the measuring action, has been already presented in previous works [4]. This paper focuses on a method for solving the second step.…”