Simulation and deep learning on point clouds for robot grasping

Abstract

Purpose In this work, the authors aim to provide a set of convenient methods for generating training data, and then develop a deep learning method based on point clouds to estimate the pose of target for robot grasping. Design/methodology/approach This work presents a deep learning method PointSimGrasp on point clouds for robot grasping. In PointSimGrasp, a point cloud emulator is introduced to generate training data and a pose estimation algorithm, which, based on deep learning, is designed. After trained with the emulation data set, the pose estimation algorithm could estimate the pose of target. Findings In experiment part, an experimental platform is built, which contains a six-axis industrial robot, a binocular structured-light sensor and a base platform with adjustable inclination. A data set that contains three subsets is set up on the experimental platform. After trained with the emulation data set, the PointSimGrasp is tested on the experimental data set, and an average translation error of about 2–3 mm and an average rotation error of about 2–5 degrees are obtained. Originality/value The contributions are as follows: first, a deep learning method on point clouds is proposed to estimate 6D pose of target; second, a convenient training method for pose estimation algorithm is presented and a point cloud emulator is introduced to generate training data; finally, an experimental platform is built, and the PointSimGrasp is tested on the platform.

One Policy to Dress Them All: Learning to Dress People with Diverse Poses and Garments

“…Some of these works propose new algorithms for imitation learning from point clouds for grasping and manipulating tools or articulated objects [52,45,8,33]. Some recent works explore applying RL with point cloud observations for grasping or dexterous hand manipulation with rigid objects [46,40,32,20]. Our work differs from these as we apply RL from point cloud observations for deformable cloth manipulation.…”

“…Policy and Q function Architecture: Most prior works [32,40,46] that train RL policies with point cloud observation use a classification-type PointNet-like [37,38] network architecture for the policy, which encodes the whole partial point cloud to a single action vector. There have been some recent works showing that instead of compressing the whole point cloud into a single action vector, inferring the action from a dense output leads to better performance [56,49,57,50,45,8,15,5].…”

“…We then use a classification-type neural network architecture (i.e., classification-type PointNet++) to output a scalar Q value. This Q function architecture has also been used in prior work [46], although with a different policy architecture. We compare to other ways of representing the Q function in experiments and show that this one works the best.…”

Learning Human-to-Robot Handovers from Point Clouds

“…Chao et al [2] introduce HandoverSim, a benchmark to evaluate handover policies in simulation. GA-DDPG [40] propose a vision-based method for grasping static objects, which can be deployed for H2R handovers. However, their method has difficulties in dynamic scenes with humans.…”

Development of Multi-Robotic Arm System for Sorting System Using Computer Vision

“…Recently, Deep Convolutional Neural Network (ConvNet or CNN), has been applied in many object recognition tasks which outstanding image recognition capabilities [53][54][55][56]. In [53], Li and Chang propose an innovative automation system for visual placement and precision positioning of the workpiece using a mobile manipulator.…”