Humans Can Integrate Augmented Reality Feedback in Their Sensorimotor Control of a Robotic Hand

Abstract

Abstract-Tactile feedback is pivotal for grasping and manipulation in humans. Providing functionally effective sensory feedback to prostheses users is an open challenge. Past paradigms were mostly based on vibroor electrotactile stimulations. However, the tactile sensitivity on the targeted body parts (usually the forearm) is greatly less than that of the hand/fingertips, restricting the amount of information that can be provided through this channel. Visual feedback is the most investigated technique in motor learning studies, where it showed positive effects in learning both simple and complex tasks; however, it was not exploited in prosthetics due to technological limitations. Here, we investigated if visual information provided in the form of augmented reality (AR) feedback can be integrated by able-bodied participants in their sensorimotor control of a pick-and-lift task while controlling a robotic hand. For this purpose, we provided visual continuous feedback related to grip force and hand closure to the participants. Each variable was mapped to the length of one of the two ellipse axes visualized on the screen of wearable single-eye display AR glasses. We observed changes in behavior when subtle (i.e., not announced to the participants) manipulation of the AR feedback was introduced, which indicated that the participants integrated the artificial feedback within the sensorimotor control of the task. These results demonstrate that it is possible to deliver effective information through AR feedback in a compact and wearable fashion. This feedback modality may be exploited for delivering sensory feedback to amputees in a clinical scenario.

Usability Assessments for Augmented Reality Motor Rehabilitation Solutions: A Systematic Review

“…Within the 32 selected articles, there were a total of 806 participants with an average age of 39.89 years, distributed through a proportion of 37% men, 23% women, and 40% unknown ( Figure 5). Of the 32 articles, 14 did not report the participants' ages, 2 reported within an age range (20-30 years and 27-35 years) [26,37], and 3 articles reported the mean ages per group [22,28,29]. From the included papers, 3 presented study protocols [38][39][40] to be completed during the research.…”

“…On the other hand, 8 of the studies (23%) aimed to investigate users who suffered injuries due to traumatic accidents, such as cerebral and spinal cord [43] and hip fracture in the elderly caused by fall [40]. Others involved several disabilities and included unique occurrences, such as muscular dystrophy fluctuations in human locomotion [44], motor coordination [33], mastectomy and stroke [29], Alzheimer [22], prosthesis users [37], Parkinson's [41], phantom limb pain [45], and different developmental difficulties (mildly intellectually retarded; cerebral palsy; moderate multiple disabilities, weak legs, and low vision) [23], and only one study [31] did not indicate the disability altogether (Figure 9).…”

Augmented Reality for Robotics: A Review

“…Another AR-based touchless interaction system allowed the users to interact with the AR environment via hand/feet gestures in [91]. In the work of Clemente et al [92], an AR-based vision system was designed to deliver feedback to the sensory-motor control module of a robotic system during object manipulation. The system utilized two wearable devices: haptic data glove and MR goggles to facilitate user interaction during remote collaborative work with a robotic hand.…”

An Education Application for Teaching Robot Arm Manipulator Concepts Using Augmented Reality

“…Nowadays, augmented reality represents a potential solution to problems in several areas such as robotics [3], teaching [4], mobile apps [5], or medicine [6]. For instance, Clemente et al [7] proposed an improvement in sensing feedback of a robotic hand by visual elements. Although other options are typically used as sensors in prostheses such as vibro-or electrotactile stimulations, one advantage of AR implementation is the increase in sensibility resolution and easiness to adapt in the real world.…”