Neural correlates of face perception modeled with a convolutional recurrent neural network

Abstract

Objective: Event-related potential (ERP) sensitivity to faces is predominantly characterized by an N170 peak that has greater amplitude and shorter latency when elicited by human faces than images of other objects. We aimed to develop a computational model of visual ERP generation to study this phenomenon which consisted of a three-dimensional convolutional neural network (CNN) connected to a recurrent neural network (RNN).
Approach: The CNN provided image representation learning, complimenting sequence learning of the RNN for modeling visually-evoked potentials. We used open-access data from ERP CORE (40 subjects) to develop the model, generated synthetic images for simulating experiments with a generative adversarial network, then collected additional data (16 subjects) to validate predictions of these simulations. For modeling, visual stimuli presented during ERP experiments were represented as sequences of images (time x pixels). These were provided as inputs to the model. By filtering and pooling over spatial dimensions, the CNN transformed these inputs into sequences of vectors that were passed to the RNN. The ERP waveforms evoked by visual stimuli were provided to the RNN as labels for supervised learning. The whole model was trained end-to-end using data from the open-access dataset to reproduce ERP waveforms evoked by visual events. 
Main results: Cross-validation model outputs strongly correlated with open-access (r = 0.98) and validation study data (r = 0.78). Open-access and validation study data correlated similarly (r = 0.81). Some aspects of model behavior were consistent with neural recordings while others were not, suggesting promising albeit limited capacity for modeling the neurophysiology of face-sensitive ERP generation.
Significance: The approach developed in this work is potentially of significant value for visual neuroscience research, where it may be adapted for multiple contexts to study computational relationships between visual stimuli and evoked neural activity.

Localized estimation of electromagnetic sources underlying event-related fields using recurrent neural networks

“…They have previously been used to simulate changes in auditory evoked responses that occur between states [9], which could be adapted to estimate how state changes influence the dynamics of underlying source signals. Simulations can also be performed to examine how the output of an RNN changes as its input representations are varied; for example, manipulating represented stimulus duration, frequency or intensity [10], altering representations of sound loudness [11], or changing input images to models of visually-evoked potentials [13]. Similar manipulations could be applied to models developed for localized source estimation to investigate what aspects of the modelling paradigm influence estimated source signals.…”

“…Recurrent neural networks (RNNs) have recently been developed for modelling auditory-evoked epidural field potentials from mice [9,10], auditory event-related potentials from humans [11], and visual event-related potentials from humans [12,13]. These models learn transformations between input representations of physical events and average event-related neural signals, which can be probed with simulations and examined using data analysis techniques.…”

“…These models learn transformations between input representations of physical events and average event-related neural signals, which can be probed with simulations and examined using data analysis techniques. By fitting labels from evoked neural activity in a supervised learning paradigm, hidden units of the RNN produce signals that can be interpreted as activity of putative (non-localized) sources [9][10][11][12][13]. In the present study, we expand this toolset by incorporating the lead field matrix into the network architecture to impose spatial priors on estimated source signals from the RNN.…”

Localized estimation of electromagnetic sources underlying event-related fields using recurrent neural networks

“…Recurrent neural networks (RNNs) have recently been developed for modelling auditory-evoked epidural field potentials from mice [9,10], auditory event-related potentials from humans [11], and visual event-related potentials from humans [12,13]. These models learn transformations between input representations of physical events and average event-related neural signals, which can be probed with simulations and examined using data analysis techniques.…”

“…These models learn transformations between input representations of physical events and average event-related neural signals, which can be probed with simulations and examined using data analysis techniques. By fitting labels from evoked neural activity in a supervised learning paradigm, hidden units of the RNN produce signals that can be interpreted as activity of putative (non-localised) sources [9][10][11][12][13]. In the present study, we expand this toolset by incorporating the lead field matrix from a volumetric source space forward conduction model into the network architecture to impose spatial priors on estimated source signals from the RNN.…”