Multivariate Pattern Classification of Facial Expressions Based on Large-Scale Functional Connectivity

Abstract

It is an important question how human beings achieve efficient recognition of others’ facial expressions in cognitive neuroscience, and it has been identified that specific cortical regions show preferential activation to facial expressions in previous studies. However, the potential contributions of the connectivity patterns in the processing of facial expressions remained unclear. The present functional magnetic resonance imaging (fMRI) study explored whether facial expressions could be decoded from the functional connectivity (FC) patterns using multivariate pattern analysis combined with machine learning algorithms (fcMVPA). We employed a block design experiment and collected neural activities while participants viewed facial expressions of six basic emotions (anger, disgust, fear, joy, sadness, and surprise). Both static and dynamic expression stimuli were included in our study. A behavioral experiment after scanning confirmed the validity of the facial stimuli presented during the fMRI experiment with classification accuracies and emotional intensities. We obtained whole-brain FC patterns for each facial expression and found that both static and dynamic facial expressions could be successfully decoded from the FC patterns. Moreover, we identified the expression-discriminative networks for the static and dynamic facial expressions, which span beyond the conventional face-selective areas. Overall, these results reveal that large-scale FC patterns may also contain rich expression information to accurately decode facial expressions, suggesting a novel mechanism, which includes general interactions between distributed brain regions, and that contributes to the human facial expression recognition.

Abstract Representations of Emotions Perceived From the Face, Body, and Whole-Person Expressions in the Left Postcentral Gyrus

“…Then the generated parameters were used to spatially normalize the functional images into the standard Montreal Neurological Institute (MNI) space at an isotropic voxel size of 3 mm × 3 mm × 3 mm. Especially, the images in the first four runs and the functional localization run were smoothed with a 4-mm full-width at half-maximum (FWHM) Gaussian filter ( Zhang et al, 2016 ; Liang et al, 2018 ). Before the further analysis, fMRI data were fitted with a GLM to obtain regressors for all nine experimental conditions (happy face, angry face, fearful face, happy body, angry body, fearful body, happy whole-person, angry whole-person, and fearful whole-person).…”

Network Representations of Facial and Bodily Expressions: Evidence From Multivariate Connectivity Pattern Classification

“…The fMRI data were first preprocessed using SPM8 software 1 . For each run, the first five volumes were discarded to allow for T1 equilibration effects (Wang et al, 2016; Liang et al, 2018; Zhang et al, 2018). The remaining functional images were corrected for slice acquisition time and head motion.…”

“…The modeled task effects (box-car task design function convolved with the canonical hemodynamic response function) were also included as covariates to ensure that temporal correlations reflected FC and did not simply reflect task-related co-activations (Cole et al, 2019). Each of these defined confounding factors was then regressed out from the BOLD time series, and the resulting residual time series were temporally filtered using band-pass filter 0.01–0.1 Hz (Wang et al, 2016; Liang et al, 2018). The FC computation was conducted on the residual BOLD time series.…”

“…Using whole-brain FC analysis combined with MVPA, recent fMRI studies have demonstrated the successful decoding of neurological disorders and various object categories from the FC patterns (Zeng et al, 2012; Liu et al, 2015; Wang et al, 2016). Inspired by these studies, our recent study has further revealed the successful decoding of facial expressions based on the FC patterns (Liang et al, 2018). However, the potential contribution of the FC patterns in the decoding of bodily expressions remains unclear, and no neuroimaging studies have resolved the question of whether emotions perceiving from facial and bodily expressions are processed rely upon common or different neural networks.…”

Neural Activity and Decoding of Action Observation Using Combined EEG and fNIRS Measurement

“…Complex brain networks can reveal the mechanisms and characteristics of brain structure and function that cannot be discovered by past analytical methods, such as modularity, hierarchy, centrality, and the distribution of network hubs (Bullmore and Sporns, 2009). The complex brain network is a powerful approach to identify the similarities and differences of brain activation in many applications, such as brain-computer interface classification (Zhang et al, 2016), mental illness diagnosis (Fang et al, 2017; Shon et al, 2018), fatigue detection (Han et al, 2019), and emotional cognitive classification (Liang et al, 2018). However, to date, few studies have used complex brain networks to classify action observation and understanding.…”