Crowd-Sourced Assessment of Technical Skills: Differentiating Animate Surgical Skill Through the Wisdom of Crowds

Holst, Daniel; Kowalewski, Timothy M.; White, Lesley; Brand, Timothy C.; Harper, Jonathan D.; Sorensen, Mathew D.; Truong, Mireille; Simpson, Khara; Tanaka, Alyssa; Smith, Roger D.; Lendvay, Thomas S.

doi:10.1089/end.2015.0104

Cited by 81 publications

(59 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Global Evaluative Assessment of Robotic Skills (GEARS) tool, developed by Goh et al. , has been applied to urological assessments on multiple occasions , and has the strongest validity argument supporting its use in the assessment of robotic skill. Its generic framework has allowed it to become a widely accepted method of assessment across multiple procedures and even across specialties .…”

Section: Resultsmentioning

confidence: 99%

“…Notably, evidence supports its ability to discriminate amongst staff surgeons of differing case volume , as well as across a single surgeon's learning curve . The vast majority of literature using the GEARS score has found it to be a reliable assessment method . However, a study of robotic renal hilar dissection using oriented expert raters showing poor internal consistency , and Hung et al.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Implementing assessments of robot‐assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

et al. 2018

View full text Add to dashboard Cite

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Implementing assessments of robot‐assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

et al. 2018

View full text Add to dashboard Cite

show abstract

“…The crowd was biased toward being more critical of the top performing surgeons than the experts were. Turkers have also been asked to assess surgical skills in videos of live, porcine robot-assisted urinary bladder closures (16). In this study, Turkers agreed with expert scoring (alpha>0.90) and completed the task in less than 5 hours, more than 13 days faster than the surgeon graders (16).…”

Section: Crowdsourcing As Tool For Biomedical Researchmentioning

confidence: 99%

“…Turkers have also been asked to assess surgical skills in videos of live, porcine robot-assisted urinary bladder closures (16). In this study, Turkers agreed with expert scoring (alpha>0.90) and completed the task in less than 5 hours, more than 13 days faster than the surgeon graders (16). In a separate study, Turkers consistently generated the same rank order of lower scoring surgeons as the expert graders in reviewing videos of robotic-assisted radical prostatectomy (17).…”

Section: Crowdsourcing As Tool For Biomedical Researchmentioning

confidence: 99%

Crowdsourcing

Wang

Mudie

Brady

2016

Current Opinion in Ophthalmology

View full text Add to dashboard Cite

Purpose of review Crowdsourcing involves the use of the collective intelligence of online communities to produce solutions and outcomes for defined objectives. The use of crowdsourcing is growing in many scientific areas. Crowdsourcing in ophthalmology has been used in basic science and clinical research, however it also shows promise as a method with wide-ranging applications. This review presents current findings on the use of crowdsourcing in ophthalmology and potential applications in the future. Recent findings Crowdsourcing has been used to distinguish normal retinal images from images with diabetic retinopathy; the collective intelligence of the crowd was able to correctly classify 81% of 230 images (19 unique) for USD$1.10 per eye in 20 minutes. Crowdsourcing has also been used to distinguish normal optic discs from abnormal ones with reasonable sensitivity (83–88%), but low specificity (35–43%). Another study used crowdsourcing for quick and reliable manual segmentation of OCT images. Outside of ophthalmology, crowdsourcing has been used for text and image interpretation, language translation, and data analysis. Summary Crowdsourcing has the potential for rapid and economical data processing. Among other applications, it could be used in research settings to provide the “ground-truth” data, and in the clinical settings to relieve the burden of image processing on experts.

show abstract

“…Crowdsourcing provides data that is inexpensive, rapid, and objective due to the huge size of participant pool [12][13][14]. Holst, et al, show that crowdsourcing can be used to obtain valid performance grading of videos of urologic residents and faculty at different levels on basic robotic skills tasks [15,16]. Moreover, Holst, et al, obtained these results in hours, making the use of crowdsourcing for competency assessment of technical skills more valuable.…”

Section: Research Articlementioning

confidence: 99%

Crowd Sourced Assessment of Technical Skills (CSATS): A Scalable Assessment Tool for the Nursing Workforce

Kirsch¹,

Comstock²,

Harder³

et al. 2016

Arch Nurs Pract Care

View full text Add to dashboard Cite

Background: In the current healthcare environment, education for technical skills focuses on quality improvement that demands ongoing skill assessment. Objectively assessing competency is a complex task that, when done effectively, improves patient care. Current methods are time-consuming, expensive, and subjective. Crowdsourcing is the practice of obtaining services from a large group of people, typically the general public on an online community. CSATS (Crowd Sourced Assessment of Technical Skills) uses crowdsourcing as an innovative way to rapidly, objectively, and comprehensively assess technical skills. We hypothesized that CSATS could accurately evaluate the technical skill proficiency of nurses. Methods:An interface displaying one of 34 video-recorded nurses performing a glucometer skills test and a corresponding survey listing each required step were uploaded to an Amazon.com hosted crowdsourcing site, Mechanical Turk™. The crowd evaluated completion and sequence of the glucometer steps in each video. Results:In under 4 hours, we obtained 1,300 crowd ratings, approximately 38 per video that evaluated the user's performance based on completion and correct order of steps. The crowd identified individual performance variance, specific steps frequently missed by users, and provided feedback tailored to each user. CSATS identified 15% of nurses who would benefit from additional training. Conclusion:Our study showed that healthcare-naïve crowd workers can assess technical skill proficiency rapidly and accurately at nominal cost. CSATS may be a valuable tool to assist educators in creating targeted training curricula for nurses in need of follow up while rapidly identifying nurses whose technical skills meet expectations, thus, dramatically reducing the resource burden for training. myriad of challenges [2]. Assessing competency is time-intensive and expensive, and requires at least one certified nurse educator to teach/assess relatively simple nursing skills. In our study, over 1,400 nurses being trained at one hospital for a glucometer device received 15 minutes of didactic instruction and 5 minutes of hands on demonstration as part of seven 4-hour training sessions over the course of two months. That training period involved paying 1,400 nurses for their time and paying experts and evaluators to lead the training, all while mandating all nurses spend time in an education session rather than at the bedside. In addition to the costs and time, successful competency assessment in and of itself is challenging due to biased evaluators, inconsistent assessment tools, and no objective way to guarantee that those individuals who need more help receive it [3,4,6]. Some innovations have attempted to address the challenges of competency assessment, such as using video-based self-assessments to give participants and their evaluators accurate views of their performance [6]. The field of nursing and healthcare in general is in need of a competency assessment methodology that is reliable, fast, and affordable.Crowd ...

show abstract

Crowd-Sourced Assessment of Technical Skills: Differentiating Animate Surgical Skill Through the Wisdom of Crowds

Cited by 81 publications

References 15 publications

Implementing assessments of robot‐assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

Implementing assessments of robot‐assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

Crowdsourcing

Crowd Sourced Assessment of Technical Skills (CSATS): A Scalable Assessment Tool for the Nursing Workforce

Contact Info

Product

Resources

About