A Wireless Brain-Machine Interface for Real-Time Speech Synthesis

Abstract

BackgroundBrain-machine interfaces (BMIs) involving electrodes implanted into the human cerebral cortex have recently been developed in an attempt to restore function to profoundly paralyzed individuals. Current BMIs for restoring communication can provide important capabilities via a typing process, but unfortunately they are only capable of slow communication rates. In the current study we use a novel approach to speech restoration in which we decode continuous auditory parameters for a real-time speech synthesizer from neuronal activity in motor cortex during attempted speech.Methodology/Principal FindingsNeural signals recorded by a Neurotrophic Electrode implanted in a speech-related region of the left precentral gyrus of a human volunteer suffering from locked-in syndrome, characterized by near-total paralysis with spared cognition, were transmitted wirelessly across the scalp and used to drive a speech synthesizer. A Kalman filter-based decoder translated the neural signals generated during attempted speech into continuous parameters for controlling a synthesizer that provided immediate (within 50 ms) auditory feedback of the decoded sound. Accuracy of the volunteer's vowel productions with the synthesizer improved quickly with practice, with a 25% improvement in average hit rate (from 45% to 70%) and 46% decrease in average endpoint error from the first to the last block of a three-vowel task.Conclusions/SignificanceOur results support the feasibility of neural prostheses that may have the potential to provide near-conversational synthetic speech output for individuals with severely impaired speech motor control. They also provide an initial glimpse into the functional properties of neurons in speech motor cortical areas.

Brain-Computer Interfaces in Medicine

“…105,106 Current research seeks to restore speech by implanting the device in the speech motor area and decoding phonemes from imagined speech. 107,108 In recent studies, 2 patients with stereotactic depth electrodes implanted in the hippocampus before epilepsy surgery were able to use signals from these electrodes to accurately control a P300-based BCI speller. 12,16 Ongoing studies in a number of laboratories are working toward achieving natural control of devices such as a prosthetic arm using electrode microarrays implanted in the motor cortex or other cortical areas of nonhuman primates.…”

Brain-Machine-Brain Interface

“…Recently, Guenther et al (2009) implanted a single "locked-in" patient with a neurotrophic electrode in a speech-related cortical region of the left precentral gyrus. The subject was able to learn to modulate activity on this electrode to control a speech synthesizer and produce four vowel sounds.…”

“…Sensory neuroprostheses and sensory substitution systems for the restoration of hearing (Merzenich et al, 1974;Fallon et al, 2008) and vision (Bach-y-Rita et al, 1969, 1998Bach-y-Rita & Kercel, 2003;Dobelle et al, 1974;Dagnelie, 2008;Cohen, 2007) walking ) and putative systems that combine both speech production (Guenther et al, 2009;Brumberg et al, 2010) and hearing (Merzenich et al, 1974;Fallon et al, 2008;Peterson et al, 2010).…”

Reliability of brain computer interface language sample transcription procedures

“…Invasive BCI systems where the electrode array is surgically implanted have been used for elicited production of isolated phonemes with a speech synthesizer [6]. Invasive BCI systems have never been used for production of connected speech or generative language.…”