Social Trait Information in Deep Convolutional Neural Networks Trained for Face Identification

Abstract

Faces provide information about a person's identity, as well as their sex, age, and ethnicity. People also infer social and personality traits from the face — judgments that can have important societal and personal consequences. In recent years, deep convolutional neural networks (DCNNs) have proven adept at representing the identity of a face from images that vary widely in viewpoint, illumination, expression, and appearance. These algorithms are modeled on the primate visual cortex and consist of multiple processing layers of simulated neurons. Here, we examined whether a DCNN trained for face identification also retains a representation of the information in faces that supports social‐trait inferences. Participants rated male and female faces on a diverse set of 18 personality traits. Linear classifiers were trained with cross validation to predict human‐assigned trait ratings from the 512 dimensional representations of faces that emerged at the top‐layer of a DCNN trained for face identification. The network was trained with 494,414 images of 10,575 identities and consisted of seven layers and 19.8 million parameters. The top‐level DCNN features produced by the network predicted the human‐assigned social trait profiles with good accuracy. Human‐assigned ratings for the individual traits were also predicted accurately. We conclude that the face representations that emerge from DCNNs retain facial information that goes beyond the strict limits of their training.

Convolutional neural net face recognition works in non-human-like ways

“…CNNs can certainly be trained to identify sex and race; results from the Kairos system are used here as evidence that our transforms had the expected effect. In fact it has been shown that the top-level layer of a CNN that was trained to perform identification codes not only information about sex and race but a range of other social characteristics such as being warm, impulsive or anxious [22].…”

Automated face recognition in forensic science: Review and perspectives

“…It is important to note that, despite the emergence of face recognition algorithms based on deep learning and artificial intelligence [31], the black-box effect remains. The structure of these neural networks indeed allows generating impressive results by letting the algorithm learn to detect and sort relevant information from large databases (see [32] for general structure and applications of deep learning, and [33,34] as an example of the most recent studies applied to face recognition). However, while the overall structure of these neural networks is theoretically known, the processing of features is not transparent, even for the developers.…”

A cautionary note on predicting social judgments from faces with deep neural networks

“…While this approach is informative, it turns out to be unnecessary. DCNNs that have only been trained to recognize face identity, or even object identity, without any training specifically on trait judgments, already generate features that can be used in simple regression models to predict human trait judgments of faces 17,18 . This finding is perhaps unsurprising, since, in the absence of any other context, the structural features of the face are also the only source of information that human raters have available for their trait judgments.…”

“…Past and current work highlights several specific limitations of using DCNNs to predict human trait ratings. First, inconsistent results have been found when using features from DC-NNs pre-trained for face identification versus those for object recognition 17,18 ; it is also unclear how features from different pre-trained DCNNs explain the variance in trait judgments of faces. Second, all prior studies trained and tested their models using a single dataset (the 10k US Adult Face Database 21 in Song et al 17 , and ratings for the Human ID Database 22 in Parde et al 18 ), leaving it an open question how well this approach generalizes out-of-sample (both across face databases and across human raters), a growing concern in modern machine learning for practical applications 23 .…”

“…First, inconsistent results have been found when using features from DC-NNs pre-trained for face identification versus those for object recognition 17,18 ; it is also unclear how features from different pre-trained DCNNs explain the variance in trait judgments of faces. Second, all prior studies trained and tested their models using a single dataset (the 10k US Adult Face Database 21 in Song et al 17 , and ratings for the Human ID Database 22 in Parde et al 18 ), leaving it an open question how well this approach generalizes out-of-sample (both across face databases and across human raters), a growing concern in modern machine learning for practical applications 23 . Third, recent findings show that human judgments of faces on a large number of traits can be captured by a small number (two to four) of psychological dimensions 14,24,25 , raising the possibility of generalizability across traits (a model trained to predict a particular trait from face features should also predict judgments of some other traits), an important shortcut that remains to be tested empirically.…”