An Augmented Reality Based Human-Robot Interaction Interface Using Kalman Filter Sensor Fusion

Abstract

In this paper, the application of Augmented Reality (AR) for the control and adjustment of robots has been developed, with the aim of making interaction and adjustment of robots easier and more accurate from a remote location. A LeapMotion sensor based controller has been investigated to track the movement of the operator hands. The data from the controller allows gestures and the position of the hand palm’s central point to be detected and tracked. A Kinect V2 camera is able to measure the corresponding motion velocities in x, y, z directions after our investigated post-processing algorithm is fulfilled. Unreal Engine 4 is used to create an AR environment for the user to monitor the control process immersively. Kalman filtering (KF) algorithm is employed to fuse the position signals from the LeapMotion sensor with the velocity signals from the Kinect camera sensor, respectively. The fused/optimal data are sent to teleoperate a Baxter robot in real-time by User Datagram Protocol (UDP). Several experiments have been conducted to test the validation of the proposed method.

Multiple Kinect based system to monitor and analyze key performance indicators of physical training

“…The problem of accurately tracking the 3D motion of a monocular camera in a known 3D environment and dynamically estimating the 3D camera location is described in [32], suggesting a fully automated landmark-based camera calibration to initialize the motion estimation and employ extended Kalman filtering techniques to track landmarks and to estimate the camera location. Kalman filter also was used in other studies such as [13,33,39,44,48] for reducing the noise in the acquired signals. Several studies have demonstrated that Kalman filter has achieved the best denoising performance when compared to other filter-based approaches [14].…”

“…The reliability of Kinect V2 is not lower as that of other (high cost) motion tracking systems and Kinect can be used as a reliable and valid clinical measurement tool [33,46,68]. However, such studies typically used simple poses such as standing, walking, sit down and stand up, and for more complex poses such as performing different kinds of physical exercises, the accuracy reliability still could improved using multiple Kinect sensors rather than a single sensor, which was demonstrated to fail, for example, for tracking a lying person [43].…”

Vehicle-in-the-Loop in Global Coordinates for Advanced Driver Assistance System

“…The 3D visualization module is programmed with Unreal Engine 4, developed by Epic Games, which allows the development of games efficiently. In recent years, it has been widely used outside of gaming, especially for visualization of computer simulation [41][42][43]. Unreal Engine 4 also has a programming language called Blueprint, similar to MATLAB/Simulink.…”

Towards Digital Twin Implementation for Assessing Production Line Performance and Balancing

“…A detailed description of digitalization processes of factories, according to the Industry 4.0 paradigm, is proposed by Belli et al [37], together with an application developed for monitoring the production and the quality control. Short and Twiddle [38] developed the digitalization of a water industry application, while Li et al [39] developed an AR application for the control and adjustment of robots during a human-robot interaction based working task. They implemented the DT of human hands by means of LeapMotion sensor and a Kinect V2 camera.…”