Industrial robot programming by demonstration using stereoscopic vision and inertial sensing

Abstract

Purpose The purpose of this paper is to present a programming by demonstration (PbD) system based on 3D stereoscopic vision and inertial sensing that provides a cost-effective pose tracking system, even during error-prone situations, such as camera occlusions. Design/methodology/approach The proposed PbD system is based on the 6D Mimic innovative solution, whose six degrees of freedom marker hardware had to be revised and restructured to accommodate an IMU sensor. Additionally, a new software pipeline was designed to include this new sensing device, seeking the improvement of the overall system’s robustness in stereoscopic vision occlusion situations. Findings The IMU component and the new software pipeline allow the 6D Mimic system to successfully maintain the pose tracking when the main tracking tool, i.e. the stereoscopic vision, fails. Therefore, the system improves in terms of reliability, robustness, and accuracy which were verified by real experiments. Practical implications Based on this proposal, the 6D Mimic system reaches a reliable and low-cost PbD methodology. Therefore, the robot can accurately replicate, on an industrial scale, the artisan level performance of highly skilled shop-floor operators. Originality/value To the best of the authors’ knowledge, the sensor fusion between stereoscopic images and IMU applied to robot PbD is a novel approach. The system is entirely designed aiming to reduce costs and taking advantage of an offline processing step for data analysis, filtering and fusion, enhancing the reliability of the PbD system.

Robots trends and megatrends: artificial intelligence and the society

“…Object grasping and manipulation rely on computer vision and reinforced learning (He et al , 2019; de Souza et al , 2022). Object release is based on a priori models and real-time information.…”

A novel visual tracking scheme for unstructured indoor environments

“…The majority of visual tracking systems only employ monocular vision, which is prone to high false detection rates due to the absence of depth information in two-dimensional images [19]. To overcome this limitation, stereoscopic vision has been proposed as a solution, which utilizes two cameras to provide depth information, leading to a reduction in the number of false detections [20]. However, the use of stereoscopic vision comes with an increased computational cost, which may restrict its use in smaller platforms.…”

Robot programming by demonstration with a monocular RGB camera

“…wearable gloves; Zhu et al , 2020) or markers (e.g. motion capture; Mueller et al , 2019; Ferreira et al , 2016 ; de Souza et al , 2021a) to the demonstrator and/or the interacting objects. Instead of attaching sensors, the vision-based imitation approach without the extra effort of interactive training and the need for any special equipment is considered a more natural and low-cost approach.…”