The Utility of Peers and Trained Raters in Technical Skill-based Assessments a Generalizability Theory Study

A novel simulator-based checklist for evaluating residents' competence in cerebral angiography in China

“…Discussions have arisen in the past as to whether one should rate surgical performance in an on-site or video-based manner ( 32 – 34 ). On-site assessment enables all aspects of the procedure to be rated, and video-based assessment enables the rater to view the videos at their convenience, which has led to the increased popularity of blinded video assessment of surgical skills ( 33 ). In this study, screen recording showed the same results as on live, indicating that consistency can be guaranteed even if the same rater is reassessed over a period of time.…”

A Robust Indicator Mean-Based Method for Estimating Generalizability Theory Absolute Error and Related Dependability Indices within Structural Equation Modeling Frameworks

“…More specifically, we will consider the following three random effects designs: persons × items (pi), persons × items × occasions (pio), and persons × items × occasions × skills (pios). Partitioning of variance at the individual score level within those designs is shown in Equations ( 4)- (6).…”

“…Respondents answer items along an 8-point Likert-style metric with the response options "Definitely false" (1), "Mostly false" (2). "Moderately false" (3), "More false than true" (4), "More true than false" (5), to "Moderately true" (6), "Mostly true" (7), and "Definitely true" (8). Scales are equally balanced for positive and negative phrasing, with responses to all negatively keyed items reverse scored.…”

“…Generalizability theory (GT; [1][2][3][4]) offers an effective framework for accounting for multiple sources of measurement error when assessing the accuracy of measurement data and subsequently using that information to evaluate and improve assessment procedures. Such techniques have recently been applied to advantage in such diverse fields as education [5][6][7][8][9][10][11][12][13], psychology [14][15][16][17], business [18][19][20][21], medicine/health sciences [22][23][24][25][26][27][28][29], psychophysiology [30][31][32], athletic training [33][34][35], and many others. Although GT designs have traditionally been analyzed using analysis of variance (ANOVA) procedures, they also can be analyzed using linear mixed-effect [36,37] and structural equation models (SEMs; [14,[37][38][39][40][41][42][43]…”

“…GT can be used with both objectively and subjectively scored measures to yield indices reflecting the accuracy of scores used for either norm-or criterion-referencing purposes. However, initial uses of SEMs for conducting GT analyses were limited to derivation of Psych 2024, 6 variance components and related generalizability (G) coefficients for norm-referencing purposes that only reflected relative rather than absolute differences in scores (see, e.g., [38][39][40][41][42][43]). To address this limitation within one-and two-facet GT designs, Jorgensen [44] devised methods for estimating variance components reflecting absolute differences in scores that together with variance components for relative differences can be used to derive dependability coefficients for making criterion-referenced decisions.…”