Crowdsourced Assessment of Surgical Skill Proficiency in Cataract Surgery

Abstract

OBJECTIVE: To test whether crowdsourced lay raters can accurately assess cataract surgical skills. DESIGN: Two-armed study: independent cross-sectional and longitudinal cohorts. SETTING: Washington University Department of Ophthalmology. PARTICIPANTS AND METHODS: Sixteen cataract surgeons with varying experience levels submitted cataract surgery videos to be graded by 5 experts and 300+ crowdworkers masked to surgeon experience. Cross-sectional study : 50 videos from surgeons ranging from first-year resident to attending physician, pooled by years of training. Longitudinal study : 28 videos obtained at regular intervals as residents progressed through 180 cases. Surgical skill was graded using the modified Objective Structured Assessment of Technical Skill (mOSATS). Main outcome measures were overall technical performance, reliability indices, and correlation between expert and crowd mean scores. RESULTS: Experts demonstrated high interrater reliability and accurately predicted training level, establishing construct validity for the modified OSATS. Crowd scores were correlated with (r = 0.865, p < 0.0001) but consistently higher than expert scores for first, second, and third-year residents (p < 0.0001, paired t-test). Longer surgery duration negatively correlated with training level (r = −0.855, p < 0.0001) and expert score (r = −0.927, p < 0.0001). The longitudinal dataset reproduced cross-sectional study findings for crowd and expert comparisons. A regression equation transforming crowd score plus video length into expert score was derived from the cross-sectional dataset (r 2 = 0.92) and demonstrated excellent predictive modeling when applied to the independent longitudinal dataset (r 2 = 0.80). A group of student raters who had edited the cataract videos also graded them, producing scores that more closely approximated experts than the crowd. CONCLUSIONS: Crowdsourced rankings correlated with expert scores, but were not equivalent; crowd scores overestimated technical competency, especially for novice surgeons. A novel approach of adjusting crowd scores with surgery duration generated a more accurate predictive model for surgical skill. More studies are needed before crowdsourcing can be reliably used for assessing surgical proficiency.

Using the language of surgery to enhance ophthalmology surgical education

“…oculoplastic, orbital, and strabismus procedures in ophthalmology), and that video-based assessments can only grade surgical maneuvers within the camera's field of view [ 49 ]. In addition to using artificial intelligence for OCASE-T, other efforts have been made to reduce bias when using video recordings for formal assessment, including anonymizing recordings before grading by a panel of expert surgeons and using crowdsourcing from lay people to rate performance [ 49 , 50 ].…”

A Virtual Reading Center Model Using Crowdsourcing to Grade Photographs for Trachoma: Validation Study

“…Crowdsourcing using unskilled internet users has been used to identify pathological features of diabetic retinopathy [19,20,29] and glaucoma [18,30] on fundus photographs as well as to classify surgical outcomes of ophthalmic surgery [31][32][33].…”

Dedicated Chalazion Clinic as a Tool for Early Surgical Education in Ophthalmology Residency

“…2,3 Ophthalmology residency programs provide resident surgical education through surgical exposure and primary surgical cases, and there are a wide variety of ophthalmology microsurgical assessment strategies that have been developed to evaluate resident surgical skills. [4][5][6][7][8] There have been efforts across surgical specialties to develop earlier surgical experience in training in order to better prepare residents for more complex surgical operations. 9 For example, the American Board of Surgery mandates that general surgery residents log 250 cases by the end of their second post-graduate year (PGY2).…”

“… 2 3 Ophthalmology residency programs provide resident surgical education through surgical exposure and primary surgical cases, and there are a wide variety of ophthalmology microsurgical assessment strategies that have been developed to evaluate resident surgical skills. 4 5 6 7 8 …”