Machine translation of cortical activity to text with an encoder-decoder framework

Abstract

<jats:title>Abstract</jats:title><jats:p>A decade after the first successful attempt to decode speech directly from human brain signals, accuracy and speed remain far below that of natural speech or typing. Here we show how to achieve high accuracy from the electrocorticogram at natural-speech rates, even with few data (on the order of half an hour of spoken speech). Taking a cue from recent advances in machine translation and automatic speech recognition, we train a recurrent neural network to map neural signals directly to word sequences (sentences). In particular, the network first encodes a sentence-length sequence of neural activity into an abstract representation, and then decodes this representation, word by word, into an English sentence. For each participant, training data consist of several spoken repeats of a set of some 30-50 sentences, along with the corresponding neural signals at each of about 250 electrodes distributed over peri-Sylvian speech cortices. Average word error rates across a validation (held-out) sentence set are as low as 7% for some participants, as compared to the previous state of the art of greater than 60%. Finally, we show how to use transfer learning to overcome limitations on data availability: Training certain components of the network under multiple participants’ data, while keeping other components (e.g., the first hidden layer) “proprietary,” can improve decoding performance—despite very different electrode coverage across participants.</jats:p>

Artificial intelligence for clinical decision support in neurology

“…A recent study found that AI can transform a person’s brain waves recorded during speech production into real text. Makin et al (2020) studied four people with epilepsy who underwent brain surgery and had implanted electrodes directly over the inferior frontal cortices where words and speech are produced. The four epilepsy patients read sentences aloud, and brain signals from intracranial electrodes recorded during speech production were the inputs into an encoder recurrent neural network often used for data containing temporal information such as spontaneous brain activity.…”

Evaluation of Hyperparameter Optimization in Machine and Deep Learning Methods for Decoding Imagined Speech EEG

“…To date, it has been considered in relatively few BCI studies, as the field has favored MI, P300 or SSVEP paradigms (see [ 8 ] for review). However, a DS-BCI offers the possibility of a more naturalistic form of communication [ 9 , 10 , 11 ], relying on neural recordings corresponding to units of language rather than some unrelated brain activity [ 12 , 13 ]. One recent study has demonstrated the potential for spoken sentences to be synthesized from neural activity [ 14 ] and another has shown speech reconstruction directly from the auditory cortex while subjects listened to overt speech [ 15 ].…”

Silent Speech Interfaces for Speech Restoration: A Review

“…Following a similar approach, a 'brain-to-text' system was presented in [33] to decode speech from ECoG signals, which was able to achieve word and phone error rates below 25% and 50%, respectively. More recently, in [210], seq2seq models were used to decode speech from ECoG signals, achieving WERs as low as 3%. In [47], a pilot study showed that ECoG recordings from temporal areas can be used to synthesise speech in real time.…”