Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

Abstract

Purpose Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance. Method In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability. Results Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%–51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59–.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment. Conclusions G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta

“…This was confirmed by the generalizability analysis, which showed that disagreement between assessors only accounted for 0.8% of the variance despite minimal rater training. 22 This positive finding suggests that the tool is easy and unambiguous to use.…”

Are Video Recordings Reliable for Assessing Surgical Performance? A Prospective Reliability Study Using Generalizability Theory

“…We used G theory to explore the relative contributions to the assessments' variance, 10 as recommended in modern reliability research 10,26 . In contrast to the modest level of reliability found using κ analysis, G analyses demonstrated a very high G coefficient (>0.8) 22 with most of the variance being attributed to participant performance (ie, true variance).…”

Development of peer assessment rubrics in simulation-based learning for advanced cardiac life support skills among medical students

“…This absolute coefficient was chosen because the scores contributing to each student’s grade point average (GPAX) were based on predefined criteria, not relative comparison. The established lower limit for reliability (Phi-coefficient) was 0.70 for formative examination and 0.80 for summative examination, which signified high generalizability of assessment scores [ 41 , 42 ].…”