Why ability point estimates can be pointless: a primer on using skill measures from large-scale assessments in secondary analyses

Abstract

Measures of cognitive or socio-emotional skills from large-scale assessments surveys (LSAS) are often based on advanced statistical models and scoring techniques unfamiliar to applied researchers. Consequently, applied researchers working with data from LSAS may be uncertain about the assumptions and computational details of these statistical models and scoring techniques and about how to best incorporate the resulting skill measures in secondary analyses. The present paper is intended as a primer for applied researchers. After a brief introduction to the key properties of skill assessments, we give an overview over the three principal methods with which secondary analysts can incorporate skill measures from LSAS in their analyses: (1) as test scores (i.e., point estimates of individual ability), (2) through structural equation modeling (SEM), and (3) in the form of plausible values (PVs). We discuss the advantages and disadvantages of each method based on three criteria: fallibility (i.e., control for measurement error and unbiasedness), usability (i.e., ease of use in secondary analyses), and immutability (i.e., consistency of test scores, PVs, or measurement model parameters across different analyses and analysts). We show that although none of the methods are optimal under all criteria, methods that result in a single point estimate of each respondent’s ability (i.e., all types of “test scores”) are rarely optimal for research purposes. Instead, approaches that avoid or correct for measurement error—especially PV methodology—stand out as the method of choice. We conclude with practical recommendations for secondary analysts and data-producing organizations.

To Score or Not to Score? A Simulation Study on the Performance of Test Scores, Plausible Values, and SEM, in Regression With Socio-Emotional Skill or Personality Scales as Predictors

“…For introductions to PV methodology, see Wu (2005), von Davier 5 A potential downside of PV is that standard errors increase compared to test scores because of the additional uncertainty introduced by the imputation. However, with large sample sizes and a large number of PV, this is unlikely to et al ( 2009), Braun and von Davier (2017), and Lechner et al (2021). We provide further computational details about PV in the section 5.…”

“…Next, we briefly review these approaches. For a more in-depth treatment, we refer the reader to Lechner et al (2021).…”

“…The error variance that tarnishes test scores is likely to lead to attenuation bias when using them as predictors in multiple regression (Fuller, 2006;Schofield, 2015;Braun and von Davier, 2017;Lechner et al, 2021)-a scenario that is ubiquitous in current studies (e.g., Roberts et al, 2007;Bergner, 2020;Harzer, 2020;Wagner et al, 2020). Moreover, it may lead to false positive or false negative conclusions about incremental validity (e.g., Westfall and Yarkoni, 2016;Sengewald et al, 2018).…”

“…As a result, the true skill of each individual test taker is, by definition, unknown. Any individual point estimates of that skill-conventionally known as "test scores" or "scale scores"are but estimates and invariably contain measurement error (see Lechner et al, 2021, for an overview). The most common (though not the only possible) consequence of measurement error is that the regression coefficient of that skill will be attenuated (i.e., biased downward;Fuller, 2006) 3 .…”

Reflections on Analytical Choices in the Scaling Model for Test Scores in International Large-Scale Assessment Studies

“…where 0 is the number of items, ( ) is the item response of student on item 1, and , ) denotes a vector of item parameters for item 1. Note that error-prone ability estimates result in biased estimates of distribution parameters, particularly for the standard deviation and quantiles, and biased correlation of abilities with covariates (Lechner et al, 2021;Wu, 2005). The second obstacle in LSA studies like PISA is that not all students receive items in all ability domains (OECD, 2014; see also Frey et al, 2009).…”