Towards Sonomyography-Based Real-Time Control of Powered Prosthesis Grasp Synergies

Bimbraw, Keshav; Fox, Elizabeth; Weinberg, Gil; Hammond, Frank L.

doi:10.1109/embc44109.2020.9176483

Cited by 11 publications

(14 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It would be worthwhile to check out the classifier performance for bigger data-sets and to see how deep learning based classification performs on ultrasound data in comparison with the traditional SVC. For real time control applications, it would be necessary to evaluate the classification in an on-the-fly training and evaluation paradigm, similar to the 4 state discretized classifier trained in [19]. We trained our SVC model on data acquired in a way that motion artifacts are minimised, and the subjects were supervised in a way that they would be consistent with their movements for a particular hand configuration movement set which lead to obtaining 100% classification accuracy on our combined speed data.…”

Section: Discussion and Future Workmentioning

confidence: 99%

“…While sEMG can be used to give an estimate of the muscle activation for hand and finger movements, ultrasound imaging of the forearm can be used to visualize the muscles which can be used for hand state prediction using image processing techniques. Ultrasound imaging of the forearm, or Sonomyography, has been explored as an alternative sensing modality that can capture both muscle configuration and movement [19]- [21]. It has been shown to be capable of identifying different hand gestures and finger movements by analyzing the image obtained from ultrasound data with a combination of image processing and classification algorithms [19], [22].…”

Section: A Related Workmentioning

confidence: 99%

“…Ultrasound imaging of the forearm, or Sonomyography, has been explored as an alternative sensing modality that can capture both muscle configuration and movement [19]- [21]. It has been shown to be capable of identifying different hand gestures and finger movements by analyzing the image obtained from ultrasound data with a combination of image processing and classification algorithms [19], [22]. Sonomyography has also been used to classify several grasp types and controlling robotic mechanisms [19].…”

Section: A Related Workmentioning

confidence: 99%

“…It has been shown to be capable of identifying different hand gestures and finger movements by analyzing the image obtained from ultrasound data with a combination of image processing and classification algorithms [19], [22]. Sonomyography has also been used to classify several grasp types and controlling robotic mechanisms [19]. Akhlagi et al demonstrated classification of various hand gestures using Nearest Neighbor classification algorithms [23].…”

Section: A Related Workmentioning

confidence: 99%

“…SVMs are a class of classical machine learning algorithms based on supervised learning which are used for image classification and regression problems. For ultrasound forearm image classification to predict hand configurations, SVC and other classical machine learning algorithms have been explored in the past [19], [25]. A support vector classifier constructs hyperplanes in high-dimensional spaces, wherein a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class.…”

Section: A Hand Configuration Classificationmentioning

confidence: 99%

See 4 more Smart Citations

Prediction of Metacarpophalangeal joint angles and Classification of Hand configurations based on Ultrasound Imaging of the Forearm

Bimbraw¹,

Nycz²,

Matt³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

With the advancement in computing and robotics, it is necessary to develop fluent and intuitive methods for interacting with digital systems, augmented/virtual reality (AR/VR) interfaces, and physical robotic systems. Hand movement recognition is widely used to enable this interaction. Hand configuration classification and Metacarpophalangeal (MCP) joint angle detection are important for a comprehensive reconstruction of the hand motion. Surface electromyography (sEMG) and other technologies have been used for the detection of the hand motions. Ultrasound images of the forearm offer a way to visualize the internal physiology of the hand from a musculoskeletal perspective. Recent work has shown that these images can be classified using machine learning to predict various hand configurations. In this paper, we propose a Convolutional Neural Network (CNN) based deep learning pipeline for predicting the MCP joint angles. We supplement our results by using a Support Vector Classifier (SVC) to classify the ultrasound information into several predefined hand configurations based on activities of daily living (ADL). Ultrasound data from the forearm was obtained from 6 subjects who were instructed to move their hands according to predefined hand configurations relevant to ADLs. Motion capture data was acquired as the ground truth for hand movements at different speeds (0.5 Hz, 1 Hz, & 2 Hz) for the index, middle, ring, and pinky fingers. We were able to get promising SVC classification results on a subset of our collected data set. We demonstrated a correspondence between the predicted MCP joint angles and the actual MCP joint angles for the fingers, with an average root mean square error of 7.35 degrees. We implemented a low latency (6.25 -9.1 Hz) pipeline for the prediction of both MCP joint angles and hand configuration estimation aimed at real-time control of digital devices, AR/VR interfaces, and physical robots.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%