Persistent Human–Machine Interfaces for Robotic Arm Control Via Gaze and Eye Direction Tracking

Abstract

Recent advances in sensors and electronics have enabled electrooculogram (EOG) detection systems for capturing eye movements. However, EOG signals are susceptible to the sensor's skin‐contact quality, limiting the precise detection of eye angles and gaze. Herein, a two‐camera eye‐tracking system and a data classification method for persistent human–machine interfaces (HMIs) are introduced. Machine‐learning technology is used for a continuous real‐time classification of gaze and eye directions, to precisely control a robotic arm. In addition, a deep‐learning algorithm for classifying eye directions is developed and the pupil center‐corneal reflection method of an eye tracker for gaze tracking is utilized. A supervisory control and data acquisition architecture that can be universally applied to any screen‐based HMI task are used by the system. It is shown in the study that the classification algorithm using deep learning enables exceptional accuracy (99.99%) with the number of actions per command (≥64), the highest performance compared to other HMI systems. Demonstrating real‐time control of a robotic arm captures the unique advantages of the precise eye‐tracking system for playing chess and manipulating dice. Overall, this paper shows the HMI system's potential for remote control of surgery robots, warehouse systems, and construction tools.

The role of bio-inspired micro-/nano-structures in flexible tactile sensors

“…108–110 In addition, in the field of human–machine interactions, sensors can detect various signals associated with human movement and convert them into control signals for electronic devices, such as manipulating a robotic arm or a prosthetic limb. 111–113 Furthermore, soft robots equipped with tactile sensors are widely used in flexible grasping, medical rehabilitation, and obstacle detection by mimicking the activities of living organisms. 114,115 In this section, we provide an overview of these applications.…”

Evaluation of an adaptive hybrid tongue-brain control framework by individuals with amyotrophic lateral sclerosis

“…Sunny et al achieved a similar level of control using commercial eye tracking equipment [53]. Several other studies have similarly gained some control over the robot arm using eye tracking [1,29,11]. However, eye-tracking technology can have problems with certain eye types, pupil color, glasses, contact lenses, and ambient lighting [14,12].…”