The Time Course of Face Representations during Perception and Working Memory Maintenance

Abstract

Successful social communication requires accurate perception and maintenance of invariant (face identity) and variant (facial expression) aspects of faces. While numerous studies investigated how face identity and expression information is extracted from faces during perception, less is known about the temporal aspects of the face information during perception and working memory (WM) maintenance. To investigate how face identity and expression information evolve over time, I recorded electroencephalography (EEG) while participants were performing a face WM task where they remembered a face image and reported either the identity or the expression of the face image after a short delay. Using multivariate event-related potential (ERP) decoding analyses, I found that the two types of information exhibited dissociable temporal dynamics: Although face identity was decoded better than facial expression during perception, facial expression was decoded better than face identity during WM maintenance. Follow-up analyses suggested that this temporal dissociation was driven by differential maintenance mechanisms: Face identity information was maintained in a more “activity-silent” manner compared to facial expression information, presumably because invariant face information does not need to be actively tracked in the task. Together, these results provide important insights into the temporal evolution of face information during perception and WM maintenance.

Attractive serial dependence overcomes repulsive neuronal adaptation

“…Furthermore, correct motor responses were directly yoked to the stimulus so any behavioral tendency to report seeing the same stimulus on successive trials could be due to motor priming rather than stimulus based serial dependence (e.g., a “stay” bias). Related work has shown the ability to decode the previous stimulus from EEG activity patterns [ 31 – 33 ], but it is important to note that our study also showed robust decoding of the previous stimulus that did not also correspond with an attractive bias in the neural representation of the current stimulus (Figs 2F and 2G and S4). This is because the representations of current and past stimuli are not necessarily stored using the same code.…”

Sensory readout accounts for adaptation

“…a “stay” bias). Related work has shown the ability to decode the previous stimulus from EEG activity patterns (Fornaciai and Park, 2018; Bae and Luck, 2019; Bae, 2021) but it is important to note that our study also showed robust decoding of the previous stimulus that did not also correspond with an attractive bias in the neural representation of the current stimulus (Figure 2F-G, S4). This is because the representations of current and past stimuli are not necessarily stored using the same code.…”

“…This is consistent with our model, and it suggests that some representation of information about stimulus history should be a minimum requirement for an aware decoding scheme. The identity of the previous stimulus for spatial position and angle has been shown to be decodable from the spiking activity of single units in the frontal eye field (FEF) and large-scale activity patterns in human EEG and MEG (Papadimitriou et al, 2016; Fornaciai and Park, 2018; Bae and Luck, 2019; Bae, 2021; Hajonides et al, 2021). We additionally demonstrate that information about the previous trial is encoded in patterns of fMRI activity in human visual cortex (Figure 2F), but not in a sensory-like code (Figure S4AB).…”

“…This neural finding, however, is challenging to interpret as consecutive trials were always the same or orthogonal orientations which, by definition, cannot distinguish attractive from repulsive biases. Related studies decoding past stimuli from EEG activity do not measure how current stimulus representations are biased, precluding a connection to behavioral biases (Fornaciai and Park, 2018; Bae and Luck, 2019; Bae, 2021).…”

Beyond the brain-computer interface: Decoding brain activity as a tool to understand neuronal mechanisms subtending cognition and behavior

“…In this context, neural decoding has shown to be a powerful tool to describe the temporal dynamics of coding information, describing the evolution of information decoding performance in time. For example, Bae (2020) studied the different time course differences between face identification (recognition) and face expression (emotion) processes using EEG recordings. In this study, participants performed a working memory task in which they were asked to remember a face image presented in the screen and, after a short delay, they were asked to report either the identity or the expression of the face.…”