Crowd Sourced Assessment of Technical Skills (CSATS): A Scalable Assessment Tool for the Nursing Workforce

Abstract

Background: In the current healthcare environment, education for technical skills focuses on quality improvement that demands ongoing skill assessment. Objectively assessing competency is a complex task that, when done effectively, improves patient care. Current methods are time-consuming, expensive, and subjective. Crowdsourcing is the practice of obtaining services from a large group of people, typically the general public on an online community. CSATS (Crowd Sourced Assessment of Technical Skills) uses crowdsourcing as an innovative way to rapidly, objectively, and comprehensively assess technical skills. We hypothesized that CSATS could accurately evaluate the technical skill proficiency of nurses. Methods:An interface displaying one of 34 video-recorded nurses performing a glucometer skills test and a corresponding survey listing each required step were uploaded to an Amazon.com hosted crowdsourcing site, Mechanical Turk™. The crowd evaluated completion and sequence of the glucometer steps in each video. Results:In under 4 hours, we obtained 1,300 crowd ratings, approximately 38 per video that evaluated the user's performance based on completion and correct order of steps. The crowd identified individual performance variance, specific steps frequently missed by users, and provided feedback tailored to each user. CSATS identified 15% of nurses who would benefit from additional training. Conclusion:Our study showed that healthcare-naïve crowd workers can assess technical skill proficiency rapidly and accurately at nominal cost. CSATS may be a valuable tool to assist educators in creating targeted training curricula for nurses in need of follow up while rapidly identifying nurses whose technical skills meet expectations, thus, dramatically reducing the resource burden for training. myriad of challenges [2]. Assessing competency is time-intensive and expensive, and requires at least one certified nurse educator to teach/assess relatively simple nursing skills. In our study, over 1,400 nurses being trained at one hospital for a glucometer device received 15 minutes of didactic instruction and 5 minutes of hands on demonstration as part of seven 4-hour training sessions over the course of two months. That training period involved paying 1,400 nurses for their time and paying experts and evaluators to lead the training, all while mandating all nurses spend time in an education session rather than at the bedside. In addition to the costs and time, successful competency assessment in and of itself is challenging due to biased evaluators, inconsistent assessment tools, and no objective way to guarantee that those individuals who need more help receive it [3,4,6]. Some innovations have attempted to address the challenges of competency assessment, such as using video-based self-assessments to give participants and their evaluators accurate views of their performance [6]. The field of nursing and healthcare in general is in need of a competency assessment methodology that is reliable, fast, and affordable.Crowd ...