Decoding spoken English phonemes from intracortical electrode arrays in dorsal precentral gyrus

Abstract

<jats:title>Abstract</jats:title><jats:sec><jats:title>Objective</jats:title><jats:p>To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of classifiers trained to discriminate a comprehensive basis set for speech: 39 English phonemes. We classified neural correlates of spoken-out-loud words in the “hand knob” area of precentral gyrus, which we view as a step towards the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak.</jats:p></jats:sec><jats:sec><jats:title>Approach</jats:title><jats:p>Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode’s binned action potential counts or high-frequency local field potential power. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes’ onset times.</jats:p></jats:sec><jats:sec><jats:title>Main results</jats:title><jats:p>A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while a recurrent neural network classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively.</jats:p></jats:sec><jats:sec><jats:title>Significance</jats:title><jats:p>The ability to decode a comprehensive set of phonemes using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.</jats:p></jats:sec>

Brain–Machine Interfaces: Closed-Loop Control in an Adaptive System

“…Naturalistic typing, however, might require an interface more similar to touch-typing and decoding neural signals, such as finger movements (166) or handwriting gestures (7). Future BMIs may even decode speech directly (167). Solving the design challenges of user interfaces with limited bandwidth can be addressed independently of BMI hardware trials and can also benefit users ineligible for current clinical BMIs.…”

Acoustic inspired brain-to-sentence decoder for logosyllabic language

“…Most previous studies on language prosthesis use Electrocorticography (ECoG) electrodes 6,11,13,[19][20][21] or Utah array electrodes 4,10,22 to acquire cortical signals. However, some studies show that the subcortical signals have potential to enhance the effectiveness of decoder 23,24 .…”

Speech Production in Intracranial Electroencephalography: iBIDS Dataset

“…Additionally, ECoG is less affected by movement artifacts than non-invasive measures of neural activity. Other studies have used intracortical microarrays to decode speech [35][36][37][38] or a neurotrophic electrode 39 to synthesize formant frequencies 40,41 from the motor cortex. An alternative measure of intracranial neural activity is stereotactic EEG (sEEG), in which electrode shafts are implanted into the brain through small burr holes 42 .…”