Gaze-based teleprosthetic enables intuitive continuous control of complex robot arm use: Writing & drawing

Abstract

Abstract-Eye tracking is a powerful mean for assistive technologies for people with movement disorders, paralysis and amputees. We present a highly intuitive eye tracking-controlled robot arm operating in 3-dimensional space based on the user's gaze target point that enables tele-writing and drawing. The usability and intuitive usage was assessed by a "tele"writing experiment with 8 subjects that learned to operate the system within minutes of first time use. These subjects were naive to the system and the task and had to write three letters on a white board with a white board pen attached to the robot arm's endpoint. The instructions are to imagine they were writing text with the pen and look where the pen would be going, they had to write the letters as fast and as accurate as possible, given a letter size template. Subjects were able to perform the task with facility and accuracy, and movements of the arm did not interfere with subjects ability to control their visual attention so as to enable smooth writing. On the basis of five consecutive trials there was a significant decrease in the total time used and the total number of commands sent to move the robot arm from the first to the second trial but no further improvement thereafter, suggesting that within writing 6 letters subjects had mastered the ability to control the system. Our work demonstrates that eye tracking is a powerful means to control robot arms in closed-loop and real-time, outperforming other invasive and non-invasive approaches to Brain-MachineInterfaces in terms of calibration time (<2 minutes), training time (<10 minutes), interface technology costs. We suggests that gaze-based decoding of action intention may well become one of the most efficient ways to interface with robotic actuators -i.e. Brain-Robot-Interfaces -and become useful beyond paralysed and amputee users also for the general teleoperation of robotic and exoskeleton in human augmentation.

Towards free 3D end-point control for robotic-assisted human reaching using binocular eye tracking

“…Consequently we propose [3] how tracking the gaze position in 3D using our custom developed binocular 3D eye trackers (GT3D), provides a considerably richer signal directly relevant for robotic, exoskeleton and prosthetic control, overcoming the limitations of conventional BMI systems. We have since shown that such eye tracking can be used for precise robotic end-point control and manipulation in small 3D volumes [11] and for near instant ability to perform teleoperation tasks as painting in an oil brush technique [12], or navigate a wheelchair without requiring to interact with a user interface [13]. Here we demonstrate a novel technique for 3D calibration that enables us to quickly and accurately calibrate a UR-10 robotic arm in 3D space.…”

Controlling a robotic arm for functional tasks using a wireless head-joystick: A case study of a child with congenital absence of upper and lower limbs

“…In addition to head control methods discussed here, several alternate control interfaces to manual joysticks have been developed for individuals with severe movement impairments. These include sip-and-puff systems, voice control [24], gaze control [25] and tongue control [26]. These interfaces typically involve some tradeoff between (i) the number of control dimensions (e.g., a device that only allows control of 1 or 2 dimensions would require frequent 'switching' to control a high DOF robotic arm), (ii) the type of control (e.g., discrete controls are possible using voice commands but are less intuitive and precise relative to continuous control like a joystick), and (iii) the 'invasiveness' of the device both in terms of its physical attributes (e.g., whether it is easily wearable, wireless etc.)…”

Playing the piano with a robotic third thumb: Assessing constraints of human augmentation

“…Haddadin and Croft, 2016;Maurtua et al, 2016;Shafti et al, 2019a), or equipping them with additional arms to perform several tasks concurrently (e.g. Dziemian et al, 2016;Parietti and Asada, 2016). The latter fits within a particular area of human augmentation robotics which is referred to as supernumerary robotics.…”