Surface Electromyography–Based Recognition, Synthesis, and Perception of Prosodic Subvocal Speech

Abstract

Purpose This study aimed to evaluate a novel communication system designed to translate surface electromyographic (sEMG) signals from articulatory muscles into speech using a personalized, digital voice. The system was evaluated for word recognition, prosodic classification, and listener perception of synthesized speech. Method sEMG signals were recorded from the face and neck as speakers with ( n = 4) and without ( n = 4) laryngectomy subvocally recited (silently mouthed) a speech corpus comprising 750 phrases (150 phrases with variable phrase-level stress). Corpus tokens were then translated into speech via personalized voice synthesis ( n = 8 synthetic voices) and compared against phrases produced by each speaker when using their typical mode of communication ( n = 4 natural voices, n = 4 electrolaryngeal [EL] voices). Naïve listeners ( n = 12) evaluated synthetic, natural, and EL speech for acceptability and intelligibility in a visual sort-and-rate task, as well as phrasal stress discriminability via a classification mechanism. Results Recorded sEMG signals were processed to translate sEMG muscle activity into lexical content and categorize variations in phrase-level stress, achieving a mean accuracy of 96.3% ( SD = 3.10%) and 91.2% ( SD = 4.46%), respectively. Synthetic speech was significantly higher in acceptability and intelligibility than EL speech, also leading to greater phrasal stress classification accuracy, whereas natural speech was rated as the most acceptable and intelligible, with the greatest phrasal stress classification accuracy. Conclusion This proof-of-concept study establishes the feasibility of using subvocal sEMG-based alternative communication not only for lexical recognition but also for prosodic communication in healthy individuals, as well as those living with vocal impairments and residual articulatory function. Supplemental Material https://doi.org/10.23641/asha.14558481

Prediction of Voice Fundamental Frequency and Intensity from Surface Electromyographic Signals of the Face and Neck

“…The goal of this study was to determine the feasibility of using sEMG signals of the face and neck to predict two primary attributes of linguistic prosody: voice f o and intensity. This study builds on our primary work in using sEMG activity for silent speech recognition (i.e., identifying the words in a message; [ 14 , 15 ]) and for classifying basic manipulations in prosody (i.e., identifying how the words in a message are conveyed; [ 18 ]). Taking this past work into account, the current study successfully demonstrates efficacy in using sEMG as an alternative method for detecting prosody via continuous estimates of f o and intensity.…”

“…Unsurprisingly, our single-speaker models performed better than the multi-speaker counterparts, as sEMG signals are speaker-dependent due to skin-electrode impedances, skin and adipose thickness, as well as differences in muscle activation during speech. Indeed, most prior works in this area focus on single-speaker models for this very reason (e.g., [ 18 , 25 , 31 , 69 ]). We argue that the overall performance of the multi-speaker models is still promising, as our results provide preliminary evidence of predicting f o and intensity within 10% and 15% error, respectively.…”

“…Instead, SSIs leverage other physiological signals to infer information about speech content and reconstruct this content as text or audible outputs [ 6 ]. Different approaches have included ultrasound and optical cameras [ 7 – 9 ], electropalatographic [ 10 ], or electromagnetic [ 11 ] devices for tracking tongue and lip movements; non-audible murmur microphones for detecting resonance in the vocal tract [ 12 , 13 ]; surface electromyography (sEMG) of articulatory muscles or the larynx (e.g., [ 14 – 18 ]); and motor cortex implants [ 19 ], electroencephalography [ 20 ] or electrocorticography (ECoG; [ 21 ]) to track speech-related brain activity.…”

“…Using sEMG for alternative communication provides a noninvasive, easy-to-use alternative to EcoG- and PMA-based SSIs. Preliminary studies have shown the promise of sEMG-based SSIs to recognize a range of utterances including individual phonemes, isolated words, and even continuous speech with relatively high accuracy (e.g., [ 14 , 17 , 18 , 24 , 25 ]). Subsequent preliminary studies have begun to incorporate prosodic features in their sEMG-based SSI systems.…”

“…Subsequent preliminary studies have begun to incorporate prosodic features in their sEMG-based SSI systems. By tracking articulatory muscle activity, sEMG-based SSIs from Johner et al [ 26 ] and Vojtech et al [ 18 ] were able to successfully distinguish emphasized words and questions from normal statements, demonstrating F1 scores of 0.68 and 0.92, respectively. While these studies demonstrated the ability of an sEMG-based SSI to detect prosodic features in speech, the metrics used may lack objectivity due to the large phonetic variation in how a word can be emphasized both within and across people [ 27 ].…”