Applications of generalizability theory and their relations to classical test theory and structural equation modeling.

Abstract

Although widely recognized as a comprehensive framework for representing score reliability, generalizability theory (G-theory), despite its potential benefits, has been used sparingly in reporting of results for measures of individual differences. In this article, we highlight many valuable ways that G-theory can be used to quantify, evaluate, and improve psychometric properties of scores. Our illustrations encompass assessment of overall reliability, percentages of score variation accounted for by individual sources of measurement error, dependability of cut-scores for decision making, estimation of reliability and dependability for changes made to measurement procedures, disattenuation of validity coefficients for measurement error, and linkages of G-theory with classical test theory and structural equation modeling. We also identify computer packages for performing G-theory analyses, most of which can be obtained free of charge, and describe how they compare with regard to data input requirements, ease of use, complexity of designs supported, and output produced. (PsycINFO Database Record

Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED)

“… McArdle, 1994 ) may be leveraged to identify components of error sources and estimate their magnitude in more complex designs in more comprehensive and general ways than achievable with standard ANOVA-based ICC decompositions. The underlying framework for deriving the individual error components as factors of reliability is closely related to Cronbach’s generalizability theory (or G-Theory; Cronbach et al, 1972 ), which was recently expressed in a SEM framework ( Vispoel et al, 2018 ). Our approach is similar to those approaches but was derived using the power equivalence logic ( von Oertzen, 2010 ) to analytically derive effective error and reliability scores in a SEM context.…”

Evaluating the Internal Consistency of Subtraction-Based and Residualized Difference Scores: Considerations for Studies of Event-Related Potentials

“…Generalizability theory approaches circumvent the sampling variability endemic to splithalf estimates by using all ERP trials from all participants in the estimation of reliability (Baldwin, Larson, & Clayson, 2015;Clayson, Carbine, Baldwin, Olsen, & Larson, 2020;Clayson & Miller, 2017a). Generalizability theory provides a multifaceted framework for estimating score reliability that simultaneously considers multiple sources of variance (Brennan, 2001;Cronbach, Gleser, Nanda, & Rajaratnum, 1972;Vispoel, Morris, & Kilinc, 2018). ERP difference score reliability can be estimated using a multivariate extension of the univariate generalizability theory formulas (see Baldwin et al, 2015;Clayson & Miller, 2017a, for an introduction to the univariate approach).…”

Reliability from alpha to omega: a tutorial

“…These procedures (α, λ 3 , KR20) were essentially short cuts for estimating reliability. The variance decomposition procedures continued this approach but expanded to be known as generalizability theory (Cronbach et al, 1963;Gleser et al, 1965;Vispoel et al, 2018) and allow for the many reliability estimates discussed before. In order to understand these procedures, it is useful to think about what goes into the correlation between two tests or two times.…”

“…These estimates are based upon the patterns of correlations of the items within the test. An alternative approach makes use of Analysis of Variance procedures to decompose the total test variance into that due to individuals, to items, to time, relevant interactions, and to residual (Cronbach et al, 1963;Gleser et al, 1965;Shavelson et al, 1989;Vispoel et al, 2018). We have already discussed this in the context of test-retest reliability.…”