Encoding and Decoding Models in Cognitive Electrophysiology

Abstract

Cognitive neuroscience has seen rapid growth in the size and complexity of data recorded from the human brain as well as in the computational tools available to analyze this data. This data explosion has resulted in an increased use of multivariate, model-based methods for asking neuroscience questions, allowing scientists to investigate multiple hypotheses with a single dataset, to use complex, time-varying stimuli, and to study the human brain under more naturalistic conditions. These tools come in the form of “Encoding” models, in which stimulus features are used to model brain activity, and “Decoding” models, in which neural features are used to generated a stimulus output. Here we review the current state of encoding and decoding models in cognitive electrophysiology and provide a practical guide toward conducting experiments and analyses in this emerging field. Our examples focus on using linear models in the study of human language and audition. We show how to calculate auditory receptive fields from natural sounds as well as how to decode neural recordings to predict speech. The paper aims to be a useful tutorial to these approaches, and a practical introduction to using machine learning and applied statistics to build models of neural activity. The data analytic approaches we discuss may also be applied to other sensory modalities, motor systems, and cognitive systems, and we cover some examples in these areas. In addition, a collection of Jupyter notebooks is publicly available as a complement to the material covered in this paper, providing code examples and tutorials for predictive modeling in python. The aim is to provide a practical understanding of predictive modeling of human brain data and to propose best-practices in conducting these analyses.

Tracking Temporal Hazard in the Human Electroencephalogram Using a Forward Encoding Model

“…To test how the brain tracks temporal hazard, we applied an encoding model approach (Lalor et al, 2006, 2009; Naselaris et al, 2011; Mesgarani et al, 2014; Holdgraf et al, 2017) to track temporal hazard in human time-domain EEG, recorded during an auditory foreperiod paradigm (Herbst and Obleser, 2017). The encoding models could distinguish between the different conditions of temporal hazard (see Fig.…”

Decoding and encoding models reveal the role of mental simulation in the brain representation of meaning

“…The score can be also negative if the model is worse than random guessing. Voxels that had positive variance explained values were identified for further analysis (Miyawaki et al, 2008;Holdgraf et al, 2017) for each participant, ROI and condition. To estimate the empirical chance level performance of the encoding models, a random shuffling was added to the training phase during cross-validation before model fitting.…”

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex

“…Moreover, ECoG recordings offer a higher signal‐to‐noise ratio in the fast dynamic signal range above 60 Hz than MEG and EEG recordings (Quandt et al., ). Combining ECoG recordings with statistical modelling techniques recent studies was able to reveal neural mechanisms specific of speech coding in the human brain with high spatial and temporal resolution (Chang et al., ; Holdgraf et al., , ; Mesgarani & Chang, ; Mesgarani, Cheung, Johnson, & Chang, ). Zion Golumbic et al.…”

“…Previous research has suggested that the two different dynamic bands serve different computational functions in speech processing (Giraud & Poeppel, ). The HG band responses reflect local neural processing (Lachaux, Axmacher, Mormann, Halgren, & Crone, ; Ray & Maunsell, ) and auditory feature processing in primary and higher cortices (Holdgraf et al., , ; Ozker et al., ). The role of LF band activity (from 6 Hz to 30 Hz) in sentence processing might be related to semantic processing (Wang et al., ), disruption of the perceptual status‐quo (Engel & Fries, ) and sentence unification (Bastiaansen, Magyari, & Hagoort, ).…”

“…Moreover, ECoG recordings offer a higher signal-to-noise ratio in the fast dynamic signal range above 60 Hz than MEG and EEG recordings (Quandt et al, 2012). Combining ECoG recordings with statistical modelling techniques recent studies was able to reveal neural mechanisms specific of speech coding in the human brain with high spatial and temporal resolution (Chang et al, 2010;Holdgraf et al, 2016Holdgraf et al, , 2017Mesgarani & Chang, 2012;Mesgarani, Cheung, Johnson, & Chang, 2014). Zion Golumbic et al (2013) showed with ECoG that high gamma responses in STG track the auditory speech envelope during audiovisual speech perception but did not investigate visual lip movement tracking.…”