The neural processes that underlie your ability to read and understand this sentence are unknown. Sentence comprehension occurs very rapidly, and can only be understood at a mechanistic level by discovering the precise sequence of underlying computational and neural events. However, we have no continuous and online neural measure of sentence processing with high spatial and temporal resolution. Here we report just such a measure: intracranial recordings from the surface of the human brain show that neural activity, indexed by γ-power, increases monotonically over the course of a sentence as people read it. This steady increase in activity is absent when people read and remember nonword-lists, despite the higher cognitive demand entailed, ruling out accounts in terms of generic attention, working memory, and cognitive load. Response increases are lower for sentence structure without meaning ("Jabberwocky" sentences) and word meaning without sentence structure (word-lists), showing that this effect is not explained by responses to syntax or word meaning alone. Instead, the full effect is found only for sentences, implicating compositional processes of sentence understanding, a striking and unique feature of human language not shared with animal communication systems. This work opens up new avenues for investigating the sequence of neural events that underlie the construction of linguistic meaning.ow does a sequence of sounds emerging from one person's mouth create a complex meaning in another person's mind? Although we have long known where language is processed in the brain (1-3), we still know almost nothing about how neural circuits extract and represent the meaning of a sentence. A powerful method for addressing this question is intracranial recording of neural activity directly from the cortical surface in neurosurgery patients (i.e., electrocorticography or ECoG) (4, 5). Although opportunities for ECoG data collection are rare, determined by clinical-not scientific-priorities, they nonetheless offer an unparalleled combination of spatial and temporal resolution, and further provide direct measures of actual neural activity, rather than indirect measures via blood flow (as in PET, fMRI, and near infrared spectroscopy/optical imaging). ECoG data are particularly valuable for the study of uniquely human functions like language, where animal models are inadequate. Here we used ECoG to identify the neural events that occur online as the meaning of a sentence is extracted and represented.Prior intracranial recording studies of language have largely focused on speech perception and production (e.g., refs. 6-11) and word-level processes (e.g., refs. 12-26). However, the most distinctive feature of human language is its compositionality: the ability to create and understand complex meanings from novel combinations of words structured into phrases and sentences (27). As a first step toward understanding the neural basis of sentence comprehension, we recorded intracranial responses while participants read sentences and...

It has long been speculated whether communication between humans and machines based on natural speech related cortical activity is possible. Over the past decade, studies have suggested that it is feasible to recognize isolated aspects of speech from neural signals, such as auditory features, phones or one of a few isolated words. However, until now it remained an unsolved challenge to decode continuously spoken speech from the neural substrate associated with speech and language processing. Here, we show for the first time that continuously spoken speech can be decoded into the expressed words from intracranial electrocorticographic (ECoG) recordings.Specifically, we implemented a system, which we call Brain-To-Text that models single phones, employs techniques from automatic speech recognition (ASR), and thereby transforms brain activity while speaking into the corresponding textual representation. Our results demonstrate that our system can achieve word error rates as low as 25% and phone error rates below 50%. Additionally, our approach contributes to the current understanding of the neural basis of continuous speech production by identifying those cortical regions that hold substantial information about individual phones. In conclusion, the Brain-To-Text system described in this paper represents an important step toward human-machine communication based on imagined speech.

Many people affected by debilitating neuromuscular disorders such as amyotrophic lateral sclerosis (ALS), brainstem stroke, or spinal cord injury, are impaired in their ability to, or even unable to, communicate. A Brain-Computer Interface (BCI) uses brain signals, rather than muscles, to re-establish communication with the outside world. One particular BCI approach is the so-called "P300 matrix speller" that was first described by Farwell and Donchin in 1988. It has been widely assumed that this method does not depend on the ability to focus on the desired character, because it was thought that it relies primarily on the P300 evoked potential and minimally if at all on other EEG features such as the visual evoked potential (VEP). This issue is highly relevant for clinical application of this BCI method, because eye movements may be impaired or lost in the relevant user population.This study investigated to what extent performance in a "P300" speller BCI depends on eye gaze. We evaluated the performance of 17 healthy subjects using a "P300" matrix speller during two conditions. In one condition ("letter"), the subjects focused their eye gaze on the intended letter, while in the second condition ("center"), subjects focused eye gaze on a fixation cross that was located in the center of the matrix.The results show that the performance of the "P300" matrix speller in normal subjects depends in considerable measure on gaze direction. They thereby disprove a widespread assumption in BCI research, and suggest that this BCI might function more effectively for people who retain some eye-movement control. The applicability of these findings to people with severe neuromuscular disabilities (particularly in eye-movements) remains to be determined.

Language is one of the defining abilities of humans. Many studies have characterized the neural correlates of different aspects of language processing. However, the imaging techniques typically used in these studies were limited in either their temporal or spatial resolution. Electrocorticographic (ECoG) recordings from the surface of the brain combine high spatial with high temporal resolution, and thus could be a valuable tool for the study of neural correlates of language function. In this study, we defined the spatiotemporal dynamics of ECoG activity during a word repetition task in nine human subjects. ECoG was recorded while each subject overtly or covertly repeated words that were presented either visually or auditorily. ECoG amplitudes in the high gamma (HG) band confidently tracked neural changes associated with stimulus presentation and with the subject's verbal response. Overt word production was primarily associated with HG changes in the superior and middle parts of temporal lobe, Wernicke's area, the supramarginal gyrus, Broca's area, premotor cortex (PMC), primary motor cortex. Covert word production was primarily associated with HG changes in superior temporal lobe and the supramarginal gyrus. Acoustic processing from both auditory stimuli as well as the subject's own voice resulted in HG power changes in superior temporal lobe and Wernicke's area. In summary, this study represents a comprehensive characterization of overt and covert speech using electrophysiological imaging with high spatial and temporal resolution. It thereby complements the findings of previous neuroimaging studies of language and thus further adds to current understanding of word processing in humans.

Auditory perception and auditory imagery have been shown to activate overlapping brain regions. We hypothesized that these phenomena also share a common underlying neural representation. To assess this, we used electrocorticography intracranial recordings from epileptic patients performing an out loud or a silent reading task. In these tasks, short stories scrolled across a video screen in two conditions: subjects read the same stories both aloud (overt) and silently (covert). In a control condition the subject remained in a resting state. We first built a high gamma (70–150 Hz) neural decoding model to reconstruct spectrotemporal auditory features of self-generated overt speech. We then evaluated whether this same model could reconstruct auditory speech features in the covert speech condition. Two speech models were tested: a spectrogram and a modulation-based feature space. For the overt condition, reconstruction accuracy was evaluated as the correlation between original and predicted speech features, and was significant in each subject (p < 10−5; paired two-sample t-test). For the covert speech condition, dynamic time warping was first used to realign the covert speech reconstruction with the corresponding original speech from the overt condition. Reconstruction accuracy was then evaluated as the correlation between original and reconstructed speech features. Covert reconstruction accuracy was compared to the accuracy obtained from reconstructions in the baseline control condition. Reconstruction accuracy for the covert condition was significantly better than for the control condition (p < 0.005; paired two-sample t-test). The superior temporal gyrus, pre- and post-central gyrus provided the highest reconstruction information. The relationship between overt and covert speech reconstruction depended on anatomy. These results provide evidence that auditory representations of covert speech can be reconstructed from models that are built from an overt speech data set, supporting a partially shared neural substrate.

Functional mapping of eloquent cortex is often necessary prior to invasive brain surgery, but current techniques that derive this mapping have important limitations. In this paper, we demonstrate the first comprehensive evaluation of a rapid, robust, and practical mapping system that uses passive recordings of electrocorticographic (ECoG) signals. This mapping procedure is based on the BC12000 and SIGFRIED technologies that we have been developing over the past several years. In our study, we evaluated ten epilepsy patients from four different institutions and compared the results of our procedure to the results derived using electrical cortical stimulation (ECS) mapping. The results show that our procedure derives a functional motor cortical map in only a few minutes. They also show a substantial concurrence with the results derived using ECS mapping. Specifically, compared to ECS maps, a next-neighbor evaluation showed no false negatives, and only 0.46% and 1.10% false positives for hand and tongue maps, respectively. In summary, we demonstrate the first comprehensive evaluation of a practical and robust mapping procedure that could become a new tool tor planning or invasive brain surgeries.

Objective-Brain-computer interface (BCI) technology can provide severely disabled people with non-muscular communication. For those most severely disabled, limitations in eye mobility or visual acuity may necessitate auditory BCI systems. The present study investigates the efficacy of the use of six environmental sounds to operate a 6×6 P300 Speller.Methods-A two-group design was used to ascertain whether participants benefited from visual cues early in training. Group A (N=5) received only auditory stimuli during all 11 sessions, whereas Group AV (N=5) received simultaneous auditory and visual stimuli in initial sessions after which the visual stimuli were systematically removed. Stepwise linear discriminant analysis determined the matrix item that elicited the largest P300 response and thereby identified the desired choice.Results-Online results and offline analyses showed that the two groups achieved equivalent accuracy. In the last session, eight of ten participants achieved 50% or more, and four of these achieved 75% or more, online accuracy (2.8% accuracy expected by chance). Mean bit rates averaged about 2 bits/min, and maximum bit rates reached 5.6 bits/min.Conclusions-This study indicates that an auditory P300 BCI is feasible, that reasonable classification accuracy and rate of communication are achievable, and that the paradigm should be further evaluated with a group of severely disabled participants who have limited visual mobility.Significance-With further development, this auditory P300 BCI could be of substantial value to severely disabled people who cannot use a visual BCI.

Motor cortex (M1) has been thought to form a continuous somatotopic homunculus extending down the precentral gyrus from foot to face representations1,2, despite evidence for concentric functional zones3 and maps of complex actions4. Here, using precision functional magnetic resonance imaging (fMRI) methods, we find that the classic homunculus is interrupted by regions with distinct connectivity, structure and function, alternating with effector-specific (foot, hand and mouth) areas. These inter-effector regions exhibit decreased cortical thickness and strong functional connectivity to each other, as well as to the cingulo-opercular network (CON), critical for action5 and physiological control6, arousal7, errors8 and pain9. This interdigitation of action control-linked and motor effector regions was verified in the three largest fMRI datasets. Macaque and pediatric (newborn, infant and child) precision fMRI suggested cross-species homologues and developmental precursors of the inter-effector system. A battery of motor and action fMRI tasks documented concentric effector somatotopies, separated by the CON-linked inter-effector regions. The inter-effectors lacked movement specificity and co-activated during action planning (coordination of hands and feet) and axial body movement (such as of the abdomen or eyebrows). These results, together with previous studies demonstrating stimulation-evoked complex actions4 and connectivity to internal organs10 such as the adrenal medulla, suggest that M1 is punctuated by a system for whole-body action planning, the somato-cognitive action network (SCAN). In M1, two parallel systems intertwine, forming an integrate–isolate pattern: effector-specific regions (foot, hand and mouth) for isolating fine motor control and the SCAN for integrating goals, physiology and body movement.