Although brain-computer interfaces (BCIs) can be used in several different ways to restore communication, communicative BCI has not approached the rate or success of natural human speech. Electrocorticography (ECoG) has precise spatiotemporal resolution that enables recording of brain activity that is distributed over a wide area of cortex, such as during speech production. In this study, we investigated words that span the entire set of phonemes in the General American accent using ECoG with 4 subjects. We classified phonemes with up to 36% accuracy when classifying all phonemes and up to 63% accuracy for a single phoneme. Further, misclassified phonemes follow articulation organization described in phonology literature, aiding classification of whole words. Precise temporal alignment to phoneme onset was crucial for classification success. We identified specific spatiotemporal features that aid classification, which could guide future applications. Word identification was equivalent to information transfer rates as high as 3.0 bits/s (33.6 words/min), supporting pursuit of speech articulation for BCI control.

Objective-Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. Approach-Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is wellsuited to work with the small amount of data available from each participant. Main results-In a study with six participants, we achieved correlations up to r = 0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. Significance-To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.

Neural interfaces that directly produce intelligible speech from brain activity would allow people with severe impairment from neurological disorders to communicate more naturally. Here, we record neural population activity in motor, premotor and inferior frontal cortices during speech production using electrocorticography (ECoG) and show that ECoG signals alone can be used to generate intelligible speech output that can preserve conversational cues. To produce speech directly from neural data, we adapted a method from the field of speech synthesis called unit selection, in which units of speech are concatenated to form audible output. In our approach, which we call Brain-To-Speech, we chose subsequent units of speech based on the measured ECoG activity to generate audio waveforms directly from the neural recordings. Brain-To-Speech employed the user's own voice to generate speech that sounded very natural and included features such as prosody and accentuation. By investigating the brain areas involved in speech production separately, we found that speech motor cortex provided more information for the reconstruction process than the other cortical areas.

Objective. Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. Approach. Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. Main results. In a study with six participants, we achieved correlations up to r = 0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. Significance. To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.

An electroencephalographic (EEG) brain-computer interface (BCI) internet browser was designed and evaluated with 10 healthy volunteers and three individuals with advanced amyotrophic lateral sclerosis (ALS), all of whom were given tasks to execute on the internet using the browser. Participants with ALS achieved an average accuracy of 73% and a subsequent information transfer rate (ITR) of 8.6 bits/min and healthy participants with no prior BCI experience over 90% accuracy and an ITR of 14.4 bits/min. We define additional criteria for unrestricted internet access for evaluation of the presented and future internet browsers, and we provide a review of the existing browsers in the literature. The P300-based browser provides unrestricted access and enables free web surfing for individuals with paralysis.

Speech is a critical form of human communication and is central to our daily lives. Yet, despite decades of study, an understanding of the fundamental neural control of speech production remains incomplete. Current theories model speech production as a hierarchy from sentences and phrases down to words, syllables, speech sounds (phonemes), and the actions of vocal tract articulators used to produce speech sounds (articulatory gestures). Here, we investigate the cortical representation of articulatory gestures and phonemes in ventral precentral and inferior frontal gyri in men and women. Our results indicate that ventral precentral cortex represents gestures to a greater extent than phonemes, while inferior frontal cortex represents both gestures and phonemes. These findings suggest that speech production shares a common cortical representation with that of other types of movement, such as arm and hand movements. This has important implications both for our understanding of speech production and for the design of brain-machine interfaces to restore communication to people who cannot speak.

Background. Abnormal muscle co-activation contributes to impairment after stroke. We developed a myoelectric computer interface (MyoCI) training paradigm to reduce abnormal co-activation. MyoCI provides intuitive feedback about muscle activation patterns, enabling decoupling of these muscles. Objective. To investigate tolerability and effects of MyoCI training of 3 muscle pairs on arm motor recovery after stroke, including effects of training dose and isometric versus movement-based training. Methods. We randomized chronic stroke survivors with moderate-to-severe arm impairment to 3 groups. Two groups tested different doses of isometric MyoCI (60 vs 90 minutes), and one group tested MyoCI without arm restraint (90 minutes), over 6 weeks. Primary outcome was arm impairment (Fugl-Meyer Assessment). Secondary outcomes included function, spasticity, and elbow range-of-motion at weeks 6 and 10. Results. Over all 32 subjects, MyoCI training of 3 muscle pairs significantly reduced impairment (Fugl-Meyer Assessment) by 3.3 ± 0.6 and 3.1 ± 0.7 (P < 10 −4) at weeks 6 and 10, respectively. Each group improved significantly from baseline; no significant differences were seen between groups. Participants' lab-based and home-based function also improved at weeks 6 and 10 (P ≤ .01). Spasticity also decreased over all subjects, and elbow range-of-motion improved. Both moderately and severely impaired patients showed significant improvement. No participants had training-related adverse events. MyoCI reduced abnormal co-activation, which appeared to transfer to reaching in the movement group. Conclusions. MyoCI is a well-tolerated, novel rehabilitation tool that enables stroke survivors to reduce abnormal co-activation. It may reduce impairment and spasticity and improve arm function, even in severely impaired patients.

Corneal potential maps obtained in response to full-field flash stimuli are altered in eyes with scotomas in the central and far-peripheral retina. The meERG approach yields useful spatial information following a single brief flash, analogous to body-surface potential maps used to evaluate heart and brain.