C-SATS: Assessing Surgical Skills Among Urology Residency Applicants

Abstract

Crowd-source assessment of resident applicant surgical skills has good inter-rater agreement with expert physician raters but not with a computer-based objective motion metrics software assessment. Overall applicant rank was affected to some degree by the crowd performance rating.

Implementing assessments of robot‐assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

“…Notably, evidence supports its ability to discriminate amongst staff surgeons of differing case volume [16], as well as across a single surgeon's learning curve [13]. The vast majority of literature using the GEARS score has found it to be a reliable assessment method [4,11,16,24,25,27,28]. However, a study of robotic renal hilar dissection using oriented expert raters showing poor internal consistency [17], and Hung et al [21] found that while trainee self-assessments and faculty evaluations correlated weakly, inter-faculty reliability was better when assessing residents [intraclass correlation (ICC) = 0.77] and fellows (ICC = 0.45).…”

“…The Global Evaluative Assessment of Robotic Skills (GEARS) tool, developed by Goh et al. , has been applied to urological assessments on multiple occasions , and has the strongest validity argument supporting its use in the assessment of robotic skill. Its generic framework has allowed it to become a widely accepted method of assessment across multiple procedures and even across specialties .…”

“…Notably, evidence supports its ability to discriminate amongst staff surgeons of differing case volume , as well as across a single surgeon's learning curve . The vast majority of literature using the GEARS score has found it to be a reliable assessment method . However, a study of robotic renal hilar dissection using oriented expert raters showing poor internal consistency , and Hung et al.…”

“…As shown in Table , it is the only technical skill assessment tool that has supporting consequences evidence, having been used to both predict clinical outcomes in a retrospective case‐control study, and impact residency match‐rankings when applied to a cohort of medical students. The Global Operative Assessment of Laparoscopic Skills (GOALS) , a laparoscopic‐specific GRS that served as the underlying model for the GEARS, was also used in robotic skills assessment by Hung et al. , with the addition of two robotic‐specific domains, instrument awareness and precision, and camera awareness and precision.…”

“…Table 1 summarises the validity evidence supporting the seven non-time-based technical skill assessment tools used in urological RAS. The Global Evaluative Assessment of Robotic Skills (GEARS) tool, developed by Goh et al [4], has been applied to urological assessments on multiple occasions [4,[11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29], and has the strongest validity argument supporting its use in the assessment of robotic skill. Its generic framework has allowed it to become a widely accepted method of assessment across multiple procedures and even across specialties [4,15,19].…”