Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs.

Abstract

On the basis of an empirical study of measures of constructs from the cognitive domain, the personality domain, and the domain of affective traits, the authors of this study examine the implications of transient measurement error for the measurement of frequently studied individual differences variables. The authors clarify relevant reliability concepts as they relate to transient error and present a procedure for estimating the coefficient of equivalence and stability (L. J. Cronbach, 1947), the only classical reliability coefficient that assesses all 3 major sources of measurement error (random response, transient, and specific factor errors). The authors conclude that transient error exists in all 3 trait domains and is especially large in the domain of affective traits. Their findings indicate that the nearly universal use of the coefficient of equivalence (Cronbach's alpha; L. J. Cronbach, 1951), which fails to assess transient error, leads to overestimates of reliability and undercorrections for biases due to measurement error.

Scale quality: alpha is an inadequate estimate and factor-analytic evidence is needed first of all

“…Ignoring transient error can lead to inaccurate conclusions (Chmielewski & Watson, 2009). The coefficient of equivalence and stability (CES) is a test-retest estimate that takes transient error into account (Schmidt, Le, & Ilies, 2003). We have previously explained how to compute CES using the previously mentioned R package (Crutzen, 2014).…”

Development of Experimental Army Enlisted Personnel Selection and Classification Tests and Job Performance Criteria

“…The second step was to estimate the standard deviations (SDs) for ratings of a single supervisor using a formula provided by Schmidt, Le, and Ilies (2003;equation all, p. 223). We needed these SDs to estimate the covariance matrix in the following step.…”

What sources contribute to variance in observer ratings? Using generalizability theory to assess construct validity of psychological measures

“…Other sources of variance include random error and systematic variance attributable to specific facets of the measurement method (e.g. choice of items or timing of measurement, Schmidt, Le, & Ilies, 2003). Thus, the essential question for investigators seeking to establish the validity of a given measurement procedure for a particular use (more formally, the validity of their intended interpretations of scores on a measure; Messick, 1989) is 'What constructs account for variance in test performance?'…”