To Score or Not to Score? A Simulation Study on the Performance of Test Scores, Plausible Values, and SEM, in Regression With Socio-Emotional Skill or Personality Scales as Predictors

Abstract

This article addresses a fundamental question in the study of socio-emotional skills, personality traits, and related constructs: “To score or not to score?” When researchers use test scores or scale scores (i.e., fallible point estimates of a skill or trait) as predictors in multiple regression, measurement error in these scores tends to attenuate regression coefficients for the skill and inflate those of the covariates. Unlike for cognitive assessments, it is not fully established how severe this bias can be in socio-emotional skill assessments, that is, how well test scores recover the true regression coefficients — compared with methods designed to account for measurement error: structural equation modeling (SEM) and plausible values (PV). The different types of scores considered in this study are standardized mean scores (SMS), regression factor scores (RFS), empirical Bayes modal (EBM) score, weighted maximum likelihood estimates (WLE), and expected a posteriori (EAP) estimates. We present a simulation study in which we compared these approaches under conditions typical of socio-emotional skill and personality assessments. We examined the performance of five types of test scores, PV, and SEM with regard to two outcomes: (1) percent bias in regression coefficient of the skill in predicting an outcome; and (2) percent bias in the regression coefficient of a covariate. We varied the number of items, factor loadings/item discriminations, sample size, and relative strength of the relationship of the skill with the outcome. Results revealed that whereas different types of test scores were highly correlated with each other, the ensuing bias in regression coefficients varied considerably. The magnitude of bias was highest for WLE with short scales of low reliability. Bias when using SMS or WLE test scores was sometimes large enough to lead to erroneous research conclusions with potentially adverse implications for policy and practice (up to 55% for the regression coefficient of the skill and 20% for that of the covariate). EAP, EBM, and RFS performed better, producing only small bias in some conditions. Additional analyses showed that the performance of test scores also depended on whether standardized or unstandardized scores were used. Only PV and SEM performed well in all scenarios and emerged as the clearly superior options. We recommend that researchers use SEM, and preferably PV, in studies on the (incremental) predictive power of socio-emotional skills.

Emotional Intelligence in Applied Settings: Approaches to Its Theoretical Model, Measurement, and Application

“…In this order of ideas, Cieciuch and Strus (2021) highlight in the current knowledge a comprehensive model of flexible core personality competencies determined by stable temperamental traits with a biological base that underlies many specific emotional skills. Finally, Bhaktha and Lechner (2021) presented a simulation study regarding the performance of test scores and plausible values, in regression with scales of personality or socio-emotional skills as a forecast for school performance results and professional success. 2017) examined an intervention in ES with the participation of parents of children with ASD, demonstrating significant improvements in children's social skills, parents' self-efficiency, children's behavior, and anxiety levels.…”

A bibliometric analysis and literature review on emotional skills

“…In this order of ideas, Cieciuch and Strus (2021) highlight in the current knowledge a comprehensive model of flexible core personality competencies determined by stable temperamental traits with a biological base that underlies many specific emotional skills. Finally, Bhaktha and Lechner (2021) presented a simulation study regarding the performance of test scores and plausible values, in regression with scales of personality or socio-emotional skills as a forecast for school performance results and professional success. Similarly, from the empirical studies performed, it is evident that there is a relation between how experiences mediated by music when learning languages can develop students' interpersonal and collaborative skills to become active members of a more inclusive society (Cores-Bilbao et al, 2019).…”