A multivariate investigation of visual word, face, and ensemble processing: Perspectives from EEG‐based decoding and feature selection

Abstract

Dan Nemrodov and Shouyu Ling are contributed equally to this work.The University of Toronto has filed a U.S. patent application that includes portions of the method for feature selection described here. Adrian Nestor and Dan Nemrodov are co-inventors on this patent. AbstractRecent investigations have focused on the spatiotemporal dynamics of visual recognition by appealing to pattern analysis of EEG signals. While this work has established the ability to decode identity-level information (such as the identity of a face or of a word) from neural signals, much less is known about the precise nature of the signals that support such feats, their robustness across visual categories, or their consistency across human participants. Here, we address these questions through the use of EEG-based decoding and multivariate feature selection as applied to three visual categories: words, faces and face ensembles (i.e., crowds of faces). Specifically, we use recursive feature elimination to estimate the diagnosticity of time and frequencybased EEG features for identity-level decoding across three datasets targeting each of the three categories. We then relate feature diagnosticity across categories and across participants while, also, aiming to increase decoding performance and reliability. Our investigation shows that word and face processing are similar in their reliance on spatiotemporal information provided by occipitotemporal channels. In contrast, ensemble processing appears to also rely on central channels and exhibits a similar profile with word processing in the frequency domain. Further, we find that feature diagnosticity is stable across participants and is even capable of supporting cross-participant feature selection, as demonstrated by systematic boosts in decoding accuracy and feature reduction. Thus, our investigation sheds new light on the nature and the structure of the information underlying identity-level visual processing as well as on its generality across categories and participants.

Putting people in context: ERP responses to bodies in natural scenes

“…The contribution of spatiotemporal features to decoding was estimated with the aid of support vector machine (SVM) weights. This method for feature selection is instrumental in the effort to decrease the dimensionality of relevant observations, to boost discriminability and to estimate feature diagnosticity [ 44 , 45 ]. The SVM procedure is able to linearly separate classes with a hyperplane within a high-dimensional representation of their data points.…”

Varying Stimulus Duration Reveals Consistent Neural Activity and Behavior for Human Face Individuation

“…S5); see also Rossion, Retter & Liu-Shuang, 2020). Previous EEG studies taking this approach have produced conflicting results: it remains debated whether FI is captured at the (peak of the) occipito-temporal facesensitive N170 component (Heisz et al, 2006;Jacques & Rossion, 2007;Nemrodov et al, 2019), on post-200 ms components such as the N250 (Schweinberger & Neumann, 2016), or even possibly at earlier latencies (e.g., Seeck et al, 1997;Nemrodov et al, 2016;Dobs et al, 2019).…”