Decoding Speech With Integrated Hybrid Signals Recorded From the Human Ventral Motor Cortex

Abstract

Restoration of speech communication for locked-in patients by means of brain computer interfaces (BCIs) is currently an important area of active research. Among the neural signals obtained from intracranial recordings, single/multi-unit activity (SUA/MUA), local field potential (LFP), and electrocorticography (ECoG) are good candidates for an input signal for BCIs. However, the question of which signal or which combination of the three signal modalities is best suited for decoding speech production remains unverified. In order to record SUA, LFP, and ECoG simultaneously from a highly localized area of human ventral sensorimotor cortex (vSMC), we fabricated an electrode the size of which was 7 by 13 mm containing sparsely arranged microneedle and conventional macro contacts. We determined which signal modality is the most capable of decoding speech production, and tested if the combination of these signals could improve the decoding accuracy of spoken phonemes. Feature vectors were constructed from spike frequency obtained from SUAs and event-related spectral perturbation derived from ECoG and LFP signals, then input to the decoder. The results showed that the decoding accuracy for five spoken vowels was highest when features from multiple signals were combined and optimized for each subject, and reached 59% when averaged across all six subjects. This result suggests that multi-scale signals convey complementary information for speech articulation. The current study demonstrated that simultaneous recording of multi-scale neuronal activities could raise decoding accuracy even though the recording area is limited to a small portion of cortex, which is advantageous for future implementation of speech-assisting BCIs.

Electrocorticogram (ECoG) Is Highly Informative in Primate Visual Cortex

“…We add to this literature by performing a direct comparison across simultaneously recorded scales, and showing that not only is ECoG efficient, but it outperforms other signals, at least in our recording conditions. Our results are consistent with recent attempts at comparing cortical signals using specialized arrays (Toda et al, 2011;Miyakawa and Hasegawa, 2013;Ibayashi et al, 2018) that have also shown that ECoGs have high decoding accuracy.…”

“…In humans, EEG has been the popular choice for BMIs because it is noninvasive, but it suffers from poor signal-to-noise ratio (Blankertz et al, 2006;McFarland et al, 2010;Sereshkeh et al, 2017;Padfield et al, 2019). More recently, BMIs based on spiking activity, LFPs, and ECoG signals have also been used (Moran, 2010;Filippini et al, 2017;Slutzky and Flint, 2017;Ibayashi et al, 2018), including human subjects with paraplegia (Aflalo et al, 2015;Bouton et al, 2016;Milekovic et al, 2018Milekovic et al, , 2019. However, an objective comparison of the usefulness of these signals is lacking because of differences in recording setup, brain area and resolution, behavioral task, species, and so on.…”

Multi-scale neural decoding and analysis

“…[195]). Existing strategies for integration include using ECoG arrays fabricated with holes through which intracortical probes can be inserted [195][196][197], performing sparse ECoG measurements without a monolithic array that provide space to insert probes [198], placing multi-unit arrays underneath ECoG arrays [199], and custom devices with mixed electrode formats [200]. Spatial registration of the two measurements may be critical for analysis and interpretation of signals and is therefore an important consideration in methodological design.…”

Non-invasive real-time access to spatial attention information from 3T fMRI BOLD signals

“…Thanks to advances in feature detection and machine learning approaches (Abraham et al, 2014), considerable progress has been made in both animal (Astrand et al, 2016;Branco et al, 2017;Brown et al, 1998;Padmanaban et al, 2018;Tremblay et al, 2015) and human research fields (Glaser et al, 2020;Grootswagers et al, 2017;Tong and Pratte, 2012). For example, sensory and motor brain signals can now be decoded with high accuracy using both invasive (Bouton et al, 2016;Branco et al, 2017;Hatsopoulos et al, 2004;Ibayashi et al, 2018) and non-invasive techniques (Kamitani and Tong, 2005;Schwarz et al, 2017;Wen et al, 2018).…”