Test Reliability at the Individual Level

Abstract

Reliability has a long history as one of the key psychometric properties of a test. However, a given test might not measure people equally reliably. Test scores from some individuals may have considerably greater error than others. This study proposed two approaches using intraindividual variation to estimate test reliability for each person. A simulation study suggested that the parallel tests approach and the structural equation modeling approach recovered the simulated reliability coefficients. Then in an empirical study, where forty-five females were measured daily on the Positive and Negative Affect Schedule (PANAS) for 45 consecutive days, separate estimates of reliability were generated for each person. Results showed that reliability estimates of the PANAS varied substantially from person to person. The methods provided in this article apply to tests measuring changeable attributes and require repeated measures across time on each individual. This article also provides a set of parallel forms of PANAS.

The Reliability Factor: Modeling individual reliability with multiple items from a single assessment

“…If error variance itself randomly varies across persons and groups, then equation 2 suggests that ICC reliability must necessarily vary across persons and groups. Indeed, previous work using the MELSM demonstrates that interviewers affect the reliability of respondents (Brunton-Smith, Sturgis, & Leckie, 2017), and repeatedly measured subjects demonstrate highly variable ICC reliability estimates (Hu et al, 2016).…”

“…Consequently, the treatment of reliability coefficients as unvarying is unduly restrictive. Indeed, item response theory has long permitted reliability to vary across the range of the latent factor itself (de Ayala, 2009), and previous work suggests that reliability can be estimated per-person (Brunton-Smith et al, 2017;Hu et al, 2016;Feldt & Quails, 1996;Lek & Van De Schoot, 2018) and modeled across the latent factor (Feldt & Quails, 1996;Raju, Price, Oshima, & Nering, 2007); notably, the typical population reliability estimate is merely the expected value of case-specific reliability (Raju et al, 2007).…”

“…Importantly, the approach can be readily used within a common design in psychology: cross-sectional singleassessment multivariate data. To date, individualspecific ω requires repeated measurements (Hu et al, 2016;Geldhof et al, 2014;Raykov & du Toit, 2005), which are then modeled with multilevel (Martin et al, 2019;Muthén, 1994) or multi-group latent variable models. This data is comparatively uncommon due to the greater cost and participant demands of collecting multiple variables, repeatedly or over time.…”

“…When multiple indicators are employed, the proposed method permits varying reliability to be estimated even when each case is only assessed on a single occasion. Altogether, the proposed method permits varying reliability without requiring parallel tests, tau-equivalent models (Lek & Van De Schoot, 2018), nor repeated observations (Hu et al, 2016;Geldhof, Preacher, & Zyphur, 2014). Therefore, the proposed approach may be used in the scenario commonly found in psychological research: Single observations with multiple item instruments.…”

Dynamics of Change and Change in Dynamics

“…These observations could be configured in a variety of ways, e.g., as 20 individuals with 20 observations each or as 10 individuals with 40 observations each. As with any simplistic rule-of-thumb such as suggested here, actual power will be dependent on the characteristics of the data: the reliability of the within-person measurements (Hu et al, in press); the degree of within-person equilibrium change over time; and the degree of between-person differences. The more one wishes to know about the person-oriented variables, the more data one will want from each person.…”