Autonomous Leaf Picking Using Deep Learning and Visual-Servoing

Ahlin, Konrad; Joffe, Benjamin; Hu, Ai-Ping; McMurray, Gary; Sadegh, Nader

doi:10.1016/j.ifacol.2016.10.033

Cited by 35 publications

(17 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the previous paper [5], the "Monoscopic Depth Analysis" approach was discussed: a method by which the leaf position was determined through multiple images in multiple camera locations. This method is quite reliable.…”

Section: A Previous Method: Single Leaf Approachmentioning

confidence: 99%

“…Our prior work [5] demonstrated the applicability of Deep Neural Networks for the leaf detection problem. In particular, complex and variable leaf appearance, challenging backgrounds, and changing natural lighting make devising any hand-crafted features very difficult for this problem.…”

Section: A Detectionmentioning

confidence: 99%

“…This procedure combines what [3] refers to as "Open-loop visual control" and "Visual Servo Control". The position of the leaf is estimated over a series of images, using techniques such as Image-Based Visual Servoing (IBVS) [4] and Monocular Depth Estimation [5] to control for accumulated error. We perform filtering of the candidates and compute confidence metrics during the procedure to control the scanning and picking approach steps of the process.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Vision-guided robotic leaf picking

Joffe¹,

Ahlin²,

Hu³

et al. 2018

EasyChair Preprints

Self Cite

View full text Add to dashboard Cite

Abstract-The key focus of precision agriculture is the ability to monitor the health of individual plants. While visual monitoring was made easier by incorporating autonomous ground and aerial vehicles, regular sampling of leaves and soil to perform chemical analysis is still necessary. The focus of this work is to propose a leaf sampling system that uses a monocular camera and a 6-DOF robot arm to detect, track, and pick healthy and unhealthy leaves without prior knowledge about the plant. The approach focuses on the challenges of operating in the real world with the aim of increasing the robustness of the system.

show abstract

Section: A Previous Method: Single Leaf Approachmentioning

confidence: 99%

Section: A Detectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Vision-guided robotic leaf picking

Joffe¹,

Ahlin²,

Hu³

et al. 2018

EasyChair Preprints

Self Cite

View full text Add to dashboard Cite

show abstract

“…A depth sensor was fixed to acquire the top-view the whole plant and to facilitate motion planning. Ahlin et al (2016) used deep convolutional neural network to detect plant leaves in the images acquired by a RGB camera mounted on the end-effector. Sparse feature points between frames were used to compute 3D location of the leaves for guiding the endeffector.…”

Section: Introductionmentioning

confidence: 99%

<i>Robotic 3D Plant Perception and Leaf Probing with Collision-Free Motion Planning for Automated Indoor Plant Phenotyping</i>

Bao¹,

Tang²,

Shah³

2017

2017 Spokane, Washington July 16 - July 19, 2017

View full text Add to dashboard Cite

Various instrumentation devices for plant physiology study such as chlorophyll fluorimeter and Raman spectrometer require leaf probing with accurate probe positioning and orientation with respect to leaf surface. In this work, we aimed to automate this process with a Kinect V2 sensor, a high-precision 2D laser profilometer, and a 6-axis robotic manipulator in a high-throughput manner. The relatively wide field of view and high resolution of Kinect V2 allowed rapid capture of the full 3D environment in front of the robot. Given the number of plants, the location and size of each plant were estimated by K-means clustering. A real-time collision-free motion planning framework based on Probabilistic Roadmap was adopted to maneuver the robotic manipulator without colliding with the plants. Each plant was scanned from top with the short-range profilometer to obtain a high-precision point cloud where potential leaf clusters were extracted by region growing segmentation. Each leaf segment was further partitioned into small patches by Voxel Cloud Connectivity Segmentation. Only the small patches with low root mean square values of plane fitting were used to compute probing poses. To evaluate probing accuracy, a square surface was scanned at various angles and its centroid was probed perpendicularly with a probing position error of 1.5 mm and a probing angle error of 0.84 degrees on average. Our growth chamber leaf probing experiment showed that the average motion planning time was 0.4 seconds and the average traveled distance of tool center point was 1 meter.

show abstract

“…A depth sensor was fixed above the plant to acquire the top-view image of the whole plant for motion planning. Ahlin et al (2016) used a deep convolutional neural network to detect plant leaves in the images acquired by an RGB camera mounted on the end effector of a robotic manipulator. Sparse feature points between frames were used to compute the 3D location of the leaves for guiding the end effector.…”

mentioning

confidence: 99%

3D Perception-Based Collision-Free Robotic Leaf Probing for Automated Indoor Plant Phenotyping

Bao¹,

Shah²,

Tang³

2018

Transactions of the ASABE

View full text Add to dashboard Cite

Various instrumentation devices for plant physiology study such as spectrometer, chlorophyll fluorimeter, and Raman spectroscopy sensor require accurate placement of their sensor probes toward the leaf surface to meet specific requirements of probe-to-target distance and orientation. In this work, a Kinect V2 sensor, a highprecision 2D laser profilometer, and a six-axis robotic manipulator were used to automate the leaf probing task. The relatively wide field of view and high resolution of Kinect V2 allowed rapid capture of the full 3D environment in front of the robot. The location and size of each plant were estimated by k-means clustering where "k" was the user-defined number of plants. A real-time collision-free motion planning framework based on Probabilistic Roadmaps was adapted to maneuver the robotic manipulator without colliding with the plants. Each plant was scanned from the top with the short-range profilometer to obtain high-precision 3D point cloud data. Potential leaf clusters were extracted by a 3D region growing segmentation scheme. Each leaf segment was further partitioned into small patches by a Voxel Cloud Connectivity Segmentation method. Only the patches with low root mean square errors of plane fitting were used to compute leaf probing poses of the robot. Experiments conducted inside a growth chamber mock-up showed that the developed robotic leaf probing system achieved an average motion planning time of 0.4 seconds with an average end-effector travel distance of 1.0 meter. To examine the probing accuracy, a square surface was scanned at different angles, and its centroid was probed perpendicularly. The average absolute probing errors of distance and angle were 1.5 mm and 0.84 degrees, respectively. These results demonstrate the utility of the proposed robotic leaf probing system for automated non-contact deployment of spectroscopic sensor probes for indoor plant phenotyping under controlled environmental conditions. This manuscript is in press. It has been accepted for publication in Transactions of the ASABE. When the final, edited version is posted online this in-press version will be removed. Example citation: Authors. Year. Article title. Trans. ASABE (in press). DOI number. The DOI for this manuscript is https://doi.org/10.13031/trans.12653. It will remain the same after publication. ABSTRACT. Various instrumentation devices for plant physiology study such as spectrometer, chlorophyll fluorimeter, and Raman

show abstract

Autonomous Leaf Picking Using Deep Learning and Visual-Servoing

Cited by 35 publications

References 5 publications

Vision-guided robotic leaf picking

Vision-guided robotic leaf picking

<i>Robotic 3D Plant Perception and Leaf Probing with Collision-Free Motion Planning for Automated Indoor Plant Phenotyping</i>

3D Perception-Based Collision-Free Robotic Leaf Probing for Automated Indoor Plant Phenotyping

Contact Info

Product

Resources

About

Autonomous Leaf Picking Using Deep Learning and Visual-Servoing

Cited by 35 publications

References 5 publications

Vision-guided robotic leaf picking

Vision-guided robotic leaf picking

&lt;i&gt;Robotic 3D Plant Perception and Leaf Probing with Collision-Free Motion Planning for Automated Indoor Plant Phenotyping&lt;/i&gt;

3D Perception-Based Collision-Free Robotic Leaf Probing for Automated Indoor Plant Phenotyping

Contact Info

Product

Resources

About

<i>Robotic 3D Plant Perception and Leaf Probing with Collision-Free Motion Planning for Automated Indoor Plant Phenotyping</i>