A study of crowdsourced segment-level surgical skill assessment using pairwise rankings

Malpani, Anand; Vedula, S. Swaroop; Chen, Grace; Hager, Gregory D.

doi:10.1007/s11548-015-1238-6

Cited by 42 publications

(34 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We examined data from studies that evaluated surgical skills (17 studies) [29][30][31][32][33][34][35][36][37][38][39][40][41][42], generated sexual health messages (seven studies) [13,[43][44][45][46][47][48], developed systems for out-of-hospital cardiopulmonary resuscitation (six studies) [2][3][4][5][6][7], quantified malaria parasitemia (two studies) [15,49], and generated messages for smoking cessation (three studies) [50][51][52].…”

Section: Synthesizing Evidencementioning

confidence: 99%

Crowdsourcing in health and medical research: a systematic review

et al. 2020

View full text Add to dashboard Cite

Background: Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health. Methods: We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach. Results: We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardiopulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-ofhospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and laypersoninitiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled. Conclusions: Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine.

show abstract

Section: Synthesizing Evidencementioning

confidence: 99%

Crowdsourcing in health and medical research: a systematic review

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Principal components that explained the majority of variation in the data were used as nonspecific summary features for OCASE-T in some studies (69, 70, 73, 79). Generic descriptors of tool motion efficiency, such as time, path length, or movements, which may be easily computed on any surgical platform, were used for OCASE-T in several other studies (78, 79, 81, 82, 84, 85, 93, 97). Descriptors of other measures of tool motion, such as force, were used as features for OCASE-T in a few studies (54, 70, 71, 83, 88).…”

Section: Data Representations For Ocase-tmentioning

confidence: 99%

“…Each observation is assigned to the class with the largest value for its discriminant function. In another approach, boundaries to optimally separate classes in the ground truth are directly modeled (e.g., SVM) (54, 57, 78, 79, 81, 82, 84, 97). In a third approach, including what may be considered prototype-based methods (NNC and LVQ NNets), each observation is assigned a class label corresponding to its nearest prototype (55, 88).…”

Section: Algorithms For Ocase-tmentioning

confidence: 99%

“…Furthermore, OCASE-T for segments within a task may be extrapolated to task-level skill assessment. Malpani et al (81, 82) used summary features computed from tool motion data to train a SVM to determine which among a pair of maneuvers was better performed. The pairwise preferences were used to rank maneuvers, compute a percentile score for each maneuver, and use linear regression to map maneuver-level percentile scores to task-level OSATS-GRS; the authors reported a moderate (0.47) correlation between the two scores.…”

Section: Feedback Based On Ocase-tmentioning

confidence: 99%

See 1 more Smart Citation

Objective Assessment of Surgical Technical Skill and Competency in the Operating Room

Vedula

Ishii

Hager

2017

Annu. Rev. Biomed. Eng.

Self Cite

107

View full text Add to dashboard Cite

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a “surgical data science” approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.

show abstract

“…This could not have been accomplished without crowdsourcing from the academic radiology community. Examples of crowdsourcing can be found across a wide variety of disciplines, including medical education (eg, Radiopaedia) and for the evaluation of trainees [6][7][8][9].…”

Section: What We Accomplishedmentioning

confidence: 99%