Closed-Loop Hybrid Gaze Brain-Machine Interface Based Robotic Arm Control with Augmented Reality Feedback

Abstract

Brain-machine interface (BMI) can be used to control the robotic arm to assist paralysis people for performing activities of daily living. However, it is still a complex task for the BMI users to control the process of objects grasping and lifting with the robotic arm. It is hard to achieve high efficiency and accuracy even after extensive trainings. One important reason is lacking of sufficient feedback information for the user to perform the closed-loop control. In this study, we proposed a method of augmented reality (AR) guiding assistance to provide the enhanced visual feedback to the user for a closed-loop control with a hybrid Gaze-BMI, which combines the electroencephalography (EEG) signals based BMI and the eye tracking for an intuitive and effective control of the robotic arm. Experiments for the objects manipulation tasks while avoiding the obstacle in the workspace are designed to evaluate the performance of our method for controlling the robotic arm. According to the experimental results obtained from eight subjects, the advantages of the proposed closed-loop system (with AR feedback) over the open-loop system (with visual inspection only) have been verified. The number of trigger commands used for controlling the robotic arm to grasp and lift the objects with AR feedback has reduced significantly and the height gaps of the gripper in the lifting process have decreased more than 50% compared to those trials with normal visual inspection only. The results reveal that the hybrid Gaze-BMI user can benefit from the information provided by the AR interface, improving the efficiency and reducing the cognitive load during the grasping and lifting processes.

Real-time Continuous Hand Motion Myoelectric Decoding by Automated Data Labeling

“…In addition, an indirect measurement method typified by computer vision has excellent recognition performance with the help of deep learning [2] , but the recognition accuracy is affected by intensity of surrounding light, image background, and obstacle occlusion. Recently, bio-signal based hand recognition has become a hotspot [3][4][5] , which has the advantages of comfortable wearing, unrestricted hand shape and outdoor usage. Since the motion of the hand and wrist is mainly innervated by the forearm muscles, one of the mainstream research directions is to extract potential motion intentions from the forearm muscles signals.…”

Semi-Autonomous Robotic Arm Reaching With Hybrid Gaze–Brain Machine Interface

“…In case the obstacle were to be knocked away by the end-effector in a run, it would then be relocated to the fixed coordinates in the next run. In our previous work, we investigated how to improve the robot grasping performance with the hybrid gaze-BMI control (Zeng et al, 2017). Thereby, we will focus on improving the reaching performance in the current study, and the grasping task will be completed automatically.…”

“…In this work, the hybrid gaze-BMI operated in two modes. In stage 1, as in Frisoli et al (2012), McMullen et al (2014), and Zeng et al (2017), the user exploited the interface in a discrete selection mode to specify the intended target location for the system by firstly gazing at the center of the target object in GUI and then issuing the confirmation once the posterior probability value for the MI state exceeded the threshold (0.6 in our experiment). After a successful target object selection indicated by the augmented reality (AR) feedback (illustrated in subsection "Camera, GUI and Computer Vision"), the hybrid gaze-BMI automatically entered the continuous-valued velocity control mode in stage 2 (the horizontal reaching).…”

“…However, one of the main limitations of gaze tracking is that the input may be intention-free (without necessarily selecting there), even though the user stares at somewhere on the computer screen. To this end, the hybrid gaze-BMI has been proposed to predict the web user's click intention (Slanzi et al, 2017), explicit selection of the target in robot reaching (Frisoli et al, 2012), and ability to carry out grasping (McMullen et al, 2014;Zeng et al, 2017) and pushing tasks (Schiatti et al, 2017). In these studies, the continuous-valued position of an object is firstly indicated by the user in the 2D space through the gaze-tracking device, and this is then confirmed once the MI state is detected with a simple "MI vs. rest" two-class BMI.…”

“…One line of the paradigms triggers a fully pre-specified autonomous takeover when a specific mental state, e.g., the motor imagery (MI) state or a response state to a visual stimulus, is detected by BMI (McMullen et al, 2014;Beckerle et al, 2017;Zhang et al, 2017). We have recently presented a semiautonomous robotic system with similar BMI triggers to initiate a subtask in a sequence (Zeng et al, 2017). While these systems are effective at accomplishing the task since a high amount of autonomy can outperform direct BMI user control, users instead preferred to have more control (Kim et al, 2012).…”