A spatial information inference method for programming by demonstration of assembly tasks by integrating visual observation with CAD model

Abstract

Purpose In robot programming by demonstration (PbD) of small parts assembly tasks, the accuracy of parts poses estimated by vision-based techniques in demonstration stage is far from enough to ensure a successful execution. This paper aims to develop an inference method to improve the accuracy of poses and assembly relations between parts by integrating visual observation with computer-aided design (CAD) model. Design/methodology/approach In this paper, the authors propose a spatial information inference method called probabilistic assembly graph with optional CAD model, shorted as PAGC*, to achieve this task. Then an assembly relation extraction method from CAD model is designed, where different assembly relation descriptions in CAD model are summarized into two fundamental relations that are colinear and coplanar. The relation similarity, distance similarity and rotation similarity are adopted as the similar part matching criterions between the CAD model and the observation. The knowledge of part in CAD is used to correct that of the corresponding part in observation. The likelihood maximization estimation is used to infer the accurate poses and assembly relations based on the probabilistic assembly graph. Findings In the experiments, both simulated data and real-world data are applied to evaluate the performance of the PAGC* model. The experimental results show the superiority of PAGC* in accuracy compared with assembly graph (AG) and probabilistic assembly graph without CAD model (PAG). Originality/value The paper provides a new approach to get the accurate pose of each part in demonstration stage of the robot PbD system. By integrating information from visual observation with prior knowledge from CAD model, PAGC* ensures the success in execution stage of the PbD system.

Contact-consistent visual object pose estimation for contact-rich robotic manipulation tasks

“…They proposed a spatial information inference method (called PAGC⋆) to improve the accuracy by integrating visual observation with CAD model. Relation (collinear and coplanar), distance and rotation similarity are used to match observations with CAD and the likelihood maximization estimation is used to infer the accurate poses and assembly relations based on the probabilistic assembly graph (Zhou et al , 2020).…”

Cloud robotic grasping of Gaussian mixture model based on point cloud projection under occlusion

“…Cloud robotics generally have strong remote processing ability, which also puts forward certain requirements for the robot’s environmental adaptability. In the existing visual inspection process of robot grasping or assembling, most of the information on the desktop is obtained by a fixed-angle camera (Wu et al , 2020; Lin et al , 2019; Liu and Qiao, 2009; Zhou et al , 2020) or moving a camera mounted on the arm to a fixed position (Monica and Aleotti, 2020). Especially in some scenes that require human-computer interaction, the camera may be installed far away from the desktop to obtain the information around the robot (Wang et al , 2017).…”