Reliability Generalization: Moving Toward Improved Understanding and Use of Score Reliability

Abstract

Reliability generalization (RG) is a measurement meta-analytic method used to explore the variability in score reliability estimates and to characterize the possible sources of this variance. This article briefly summarizes some RG considerations. Included is a description of how reliability confidence intervals might be portrayed graphically. The article includes tabulations across various RG studies, including how frequently authors (a) report score reliabilities for their own data, (b) conduct reliability induction, or (c) do not even mention reliability.Reliability generalization (RG), a meta-analytic method used to explore variability in reliability estimates and characterize the sources of this variance, may be something like a safety pin-simple, yet quite useful. Although the analogy may be debated, there is little question that an increasing number of RG studies have been published, with researchers using this meta-analytic method to evaluate the sources of measurement error variance as they occur for scores across studies using a given instrument or sets of instruments measuring a given construct. Described as "a potentially powerful method with which to characterize and explore variance in score reliability" (Vacha-

Use of internal consistency coefficients for estimating reliability of experimental task scores

“…In particular, there is little data to evaluate the sensitivity of reliability coefficients to changes in task characteristics, respondents, and administration conditions. Vacha-Haase, Henson, and Caruso (2002) discuss a methodology for evaluating sensitivity of reliability coefficients, which they refer to as reliability generalization. Also a typology of task measures could be suggested that would link the kind of internal consistency reliability coefficients to the type of task.…”

Meta-analysis of the reliability and validity of the Anticipated Turnover Scale across studies of registered nurses in the United States

“…Reliability generalization also focuses on measurement error, an essential component of reliability, and sources for it, reliability generalization adds a way to understand the degree to which measurement error impacts the results (i.e. effect size) across studies (Vache‐Haase 1998, Vacha‐Haase et al. 2002).…”

“…Various measurement experts have reported on the problem of some researchers’ lack of reporting reliability coefficients for instruments used in their studies (Vache‐Haase 1998, Whittington 1998, Vacha‐Haase et al. 2002) as well as the importance of sample‐specific reliability estimates of scores for instruments (Nunnally & Bernstein 1994, Vache‐Haase 1998, Vacha‐Haase et al. 2002, Waltz et al.…”

The reliability of romantic love: A reliability generalization meta‐analysis

“…In recognition of this, several journals have changed their editorial policies to require researchers to consider the reliability of data in hand when publishing results (e.g., Fan & Thompson, 2001). Although evidence suggests that researchers rely on reliability induction and fail to report the reliability of their data at an alarming rate (Vacha‐Haase, Henson, & Caruso, 2002), other evidence suggests gradual and steady improvement in reporting practices (Graham, Liu, & Jeziorski, 2006).…”