Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68

Abstract

We realize that statistics and mathematics can be either boring or fearsome to many physicians and educators, yet we believe that some foundations are necessary for a better understanding of generalizability analysis. Consequently, we have tried, wherever possible, to keep the use of equations to a minimum and to use a conversational and slightly "off-serious" style.

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

“…Generalizability (G) refers to the contribution to the overall variance in scores that can be attributed to the variable under investigation (Bloch & Norman, 2012). In the context of MMIs the G coefficient is the proportion of variance in MMI score that is attributable to differences in applicants' non-cognitive abilities.…”

Quantifying error in OSCE standard setting for varying cohort sizes: A resampling approach to measuring assessment quality

“…Despite this weight of literature, the estimation of the error in the standard setting itself has received relatively little attention, particularly in situations where generalizability theory (Bloch & Norman, 2012) may not be easily applicable; for example, in assessment where OSCE stations are not all scored on exactly the same scale.…”

Co-Worker Assessment and Physician Multisource Feedback

“…AG study uses repeated-measures analysis of variance to simultaneously quantify the variance embedded in each facet that are interacting to create error but is not captured in the inter-item correlation matrices underpinning internal consistency reliability. 31 G studies calculate variance components expressed as a coefficient (EP 2 ), where EP 2 > 0.70 is generally considered the minimum threshold suitable for MSF instruments of similar intent. 32 For all co-worker tools, G studies were performed to estimate the variance associated with different facets: the physician, the coworker, the questionnaire item, and residual error (i.e., measurement artifact).…”