Inferred vs traditional personality assessment: are we predicting the same thing?

Abstract

Machine learning methods are widely used by researchers to predict psychological characteristics from digital records. To answer whether automatic personality estimates retain the properties of the original traits, we reviewed 220 recent articles.First, we put together the predictive quality estimates from a subset of the studies which declare separation of training, validation, and testing phases, which is critical for ensuring the correctness of quality estimates in machine learning (Hastie et al. (2009); Zheng (2015); Kumar (2019)). Only 20% of the reviewed papers met this criterion. To compare the reported quality estimates, we converted them to approximate Pearson correlations. The credible upper limits for correlations between predicted and self-reported personality traits vary in a range between 0.42 and 0.48, depending on the specific trait. The achieved values are substantially below the correlations between traits measured with distinct self-report questionnaires. This suggests that we cannot readily interpret personality predictions as estimates of the original traits or expect predicted personality traits to reproduce known relationships with life outcomes regularly.Next, we complement quality estimates evaluation with evidence on psychometric properties of predicted traits. The few existing results suggest that predicted traits are less stable with time and have lower effective dimensionality than self-reported personality. The predictive text-based models perform substantially worse outside their training domains but stay above a random baseline. The evidence on the relationships between predicted traits and external variables is mixed. Predictive features are difficult to use for validation, due to the lack of prior hypotheses. Thus, predicted personality traits fail to retain important properties of the original characteristics. This calls for the cautious use and targeted validation of the predictive models.

Predicting Relationship Labels and Individual Personality Traits From Telecommunication History in Social Networks Using Hawkes Processes

“…The second baseline uses 145 features (NETSENSE) and 2212 features (NETHEALTH) self-reported in the surveys: grades, health, happiness, activity, book reading, and club membership. These are highly personal and sensitive features, and have been shown to be the upper bound of automatic personality traits prediction [40]. We find our method to outperform the first baseline and, surprisingly, very competitive with respect to the prediction upper-bound.…”

“…We also test combinations of feature sets. The red area shows the credible upper limits for correlations between predicted and self-reported personality traits [40].…”

Using Personality Detection Tools for Software Engineering Research: How Far Can We Go?

“…Because automatic personality recognition approaches are inherently data-driven, the availability of experimental datasets plays a crucial role. According to Novikov et al [83], more than 40% of studies on personality involve the collection of new datasets that remain private. The remaining studies rely on a few shared and reusable datasets.…”

“…40. In a survey of over 200 papers on personality published since 2017, Novikov et al [83] found that the reported Pearson correlation coefficients 15…”

“…The MAE and RMSE measures are also very popular. Novikov et al [83] found that 65 out of the 218 studies analyzed report personality prediction performance using either measure. Nonetheless, these metrics are not without problems.…”