Integrating computer vision to prosthetic hand control with sEMG: Preliminary results in grasp classification

Abstract

The myoelectric prosthesis is a promising tool to restore the hand abilities of amputees, but the classification accuracy of surface electromyography (sEMG) is not high enough for real-time application. Researchers proposed integrating sEMG signals with another feature that is not affected by amputation. The strong coordination between vision and hand manipulation makes us consider including visual information in prosthetic hand control. In this study, we identified a sweet period during the early reaching phase in which the vision data could yield a higher accuracy in classifying the grasp patterns. Moreover, the visual classification results from the sweet period could be naturally integrated with sEMG data collected during the grasp phase. After the integration, the accuracy of grasp classification increased from 85.5% (only sEMG) to 90.06% (integrated). Knowledge gained from this study encourages us to further explore the methods for incorporating computer vision into myoelectric data to enhance the movement control of prosthetic hands.

A Perspective on Prosthetic Hands Control: From the Brain to the Hand

“…• sEMG signals are nonstationary and can vary due to physiological and environmental factors like fatigue, sweat, electrode displacement, and arm position. Such changes can impact the signals' amplitude, frequency, and morphology, thereby affecting the control system's accuracy and stability [146,147]. Therefore, adaptive and robust methods are needed to cope with these changes.…”

“…Additionally, the human brain can use more sophisticated and flexible learning rules than deep learning models and can balance between stability and plasticity. The human brain controls hand movements by sending signals from the motor cortex to the spinal cord and the muscles, which are coordinated by the sensory feedback and the cerebellum [147,158]. Conversely, deep learning models control prosthetic hand movements by sending commands from the classifier to the actuator, based on sEMG signals and the control algorithm [19,159,160].…”

OBT-Trace: An approach to trace and recognition object motion through prosthetic hand gestures

“…Wang, Shuo [4] this paper proposed a computer vision-based approach for object recognition and interaction. Using a combination of depth sensing and image processing, the system achieved an object recognition accuracy of 92% and precise control over grasping and releasing objects.…”