How important is transient error in estimating reliability? Going beyond simulation studies.

Abstract

This article introduces a procedure for estimating reliability in which equivalent halves of a given test are systematically created and then administered a few days apart so that transient error can be included in the error calculus. The procedure not only estimates complete reliability (taking into account both specific-factor error and transient error) but also can estimate partial reliability (taking into account only specific-factor error). Scores from 6 different measuring instruments were analyzed with the procedure. The results indicate that the magnitude of transient error in real data can range from nonexistent to very large. It follows that traditional reliability estimates, using nonstaggered procedures, are inflated to the extent that transient error is present.

Coefficient alpha: Unnecessarily ambiguous; unduly ubiquitous

“…Just as the potential consequences of ignoring such error were dawning on the psychometric community (cf. Becker, 2000), along came Green (2003) and reinvented, so to speak, coefficient alpha. His test-retest alpha requires that the same test be administered on different occasions and uses the covariances of each item administered on one occasion with every other item administered on the other occasion.…”

“…It is less often mentioned that if transient error or correlated error scores are present (and it is a good bet that generally such flaws do prevail), coefficient alpha overestimates reliability. A more inclusive view would be that whether coefficient alpha is an underestimate or an overestimate depends on which one of these assumptions is more seriously violated (Becker, 2000;Komaroff, 1997). So, in the presence of transient error, obtaining an alpha even as high as one of 0,90 may be regarded as a Pyrrhic victory.…”

Recent proposals to estimate transient error within the classical test theory tradition

“…None of them requires the time-consuming construction of strictly parallel test forms as does the conventional coefficient of equivalence and stability. The procedures proposed by Becker (2000) and Schmidt et al (2003), however, do require the complicated procedure of dividing a test into two strictly parallel halves.…”

“…However, the first few years of the 21st century have seen three proposals that have been formulated within the classical test theory tradition to address this situation. The designs proposed by Becker (2000), Schmidt et al (2003) and Green (2003) are intended to investigate complete reliability in the absence of parallel test forms. However, all of them require that either the full-length test or two parallel halves be administered on two separate occasions.…”

“…Consequently, an estimate that doesn't take account of all of them, results in an overestimate. Becker (2000) uses the terms complete reliability and partial reliability to distinguish between estimates that are susceptible to both specific-factor error and transient error, on the one hand, and estimates that are affected by specific-factor error only (i.e., apart from random response error), on the other hand. As neither the coefficient of stability, nor the coefficient of equivalence, or any of the internal-consistency coefficients, are susceptible to both kinds of measurement error, all of them fail to reflect complete reliability.…”