Visual Cues to Improve Myoelectric Control of Upper Limb Prostheses

Abstract

The instability of myoelectric signals over time complicates their use to control highly articulated prostheses. To address this problem, studies have tried to combine surface electromyography with modalities that are less affected by the amputation and environment, such as accelerometry or gaze information. In the latter case, the hypothesis is that a subject looks at the object he or she intends to manipulate and that knowing this object's affordances allows to constrain the set of possible grasps. In this paper, we develop an automated way to detect stable fixations and show that gaze information is indeed helpful in predicting hand movements. In our multimodal approach, we automatically detect stable gazes and segment an object of interest around the subject's fixation in the visual frame. The patch extracted around this object is subsequently fed through an off-the-shelf deep convolutional neural network to obtain a high level feature representation, which is then combined with traditional surface electromyography in the classification stage. Tests have been performed on a dataset acquired from five intact subjects who performed ten types of grasps on various objects as well as in a functional setting. They show that the addition of gaze information increases the classification accuracy considerably. Further analysis demonstrates that this improvement is consistent for all grasps and concentrated during the movement onset and offset.

On the Visuomotor Behavior of Amputees and Able-Bodied People During Grasping

“…The success of this approach relies however on the ability to distinguish informative fixations from those that are not necessarily related to any grasp intent. Gigli et al (2018) attempted to address this problem by including the onset of the arm movement as an additional condition, which we have shown here to shorten the window of opportunity. Also the method that is used to detect fixations may shorten this window.…”

“…Several studies have attempted to explore this proactivity to help disabled people, such as in a robot assistant scenario (Admoni and Srinivasa, 2016; Koochaki and Najafizadeh, 2018; Saran et al, 2018). Another compelling use-case is the control of dexterous upper-limb prostheses (Castellini and Sandini, 2006; Markovic et al, 2014, 2015; Gigli et al, 2018), where deciphering the grasp intent from myoelectric activations alone can be challenging. The integration of gaze and vision as contextual information could be helpful especially during the initial transient phase of a movement.…”

Kinematic synergies of hand grasps: a comprehensive study on a large publicly available dataset

“…Including reach-to-grasp and release phases into the models can as well improve the prostheses, making them more similar to real hands and possibly empowered with multis-sensorial inputs (e.g. [44–46]). Finally, if the results are matched with muscular correspondents, they may lead to real natural and continuous myoelectric control of robotic hands, a challenge not yet achieved to the best of our knowledge.…”

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

“…This type of sensor fusion which combines vision and proprioceptive information is intensively used in biomedical applications, such as in the transradial prosthetic domain, to improve control performance (Markovic et al, 2014(Markovic et al, , 2015, or to focus on recognizing objects during grasping to adjust the movements (Došen et al, 2010). This last task can also use Convolutional Neural Networks (CNNs) as feature extractors (Ghazaei et al, 2017;Gigli et al, 2018).…”