Generalizability theory: A practical guide to study design, implementation, and interpretation

Validity and reliability of physical employment standards

“…The relative proportion of variation attributed to these various factors in form of reliability coefficients (sometimes called generalizability coefficients) can then be calculated. Generalizability theory is intended to pinpoint the sources of measurement error, disentangle them, and estimate each one (Cronbach et al 1972;Webb 1981, 1991;Brennan 2001).…”

“…In short, generalizability theory considers assessment akin to an experiment wherein the proportion of variation can be calculated by using a statistical test such as intra-class correlation coefficients (ICCs) for various factors in the assessment design. For detailed step by step guidelines, readers are guided to Bloch and Norman (2012) and Briesch et al (2014).…”

“…Finally, generalizability theory allows tests to have decisions assisted by norm-referencing or criterionreferencing (Zumbo 2016). These 2 options reflect 2 types of definitions of error variance and require different types of ICCs and reliability coefficients (Bloch and Norman 2012;Briesch et al 2014). Whilst the use of ICCs has been advocated, Atkinson and Nevill (1998) have recommended that they should not be the sole statistic used, as more work is needed to define acceptable ICCs based on the realisation of definite analytical goals.…”

Calibration of communication skills items in OSCE checklists according to the MAAS-Global

“…Classical test theory only recognizes two types of variances: true variance and error variance [20,21]. Whereas in, a Generalizability Theory study, analysis will more appropriately show the contribution of each of the potential sources of error variance to the total error [22]. The Generalizability Theory (G-theory) analysisis complementary to the classical psychometric theory and consists of a Generalisability-study (G-study) and a Decision-study (D-study).…”

The state of the Journal of School Psychology: Promoting science

“…JSP is an excellent source for manuscripts focused on the "how-to" of methodological and statistical techniques that are being used by researchers. Occasionally, these papers stand on their own as general manuscripts such as the following recently published articles focused on (a) mean phase differences and generalized least squares for analyzing single-case design (Manolov & Solanas, 2013); (b) reliability of multi-category ratings scales (Parker, Vannest, & Davis, 2013); (c) single-case effect size calculation (Ross & Begeny, 2014); and (d) a practical guide to generalizability theory (Briesch, Swaminathan, Welsh, & Chafouleas, 2014). As an author, if you are considering submitting a paper focused on methodological and statistical techniques, please feel free to contact me to discuss your idea.…”