Testing an objective structured assessment technical skills tool: A pilot study

Abstract

Purpose: The purpose of this prospective pilot study was to test the feasibility, reliability and validity of an adapted version of an objective structured assessment technical skills (OSATS) tool using a criterion-reference rating scale while evaluating student registered nurse anesthetists (SRNAs) performing mask ventilation, laryngeal mask airway (LMA) insertion, and endotracheal intubation. Methods: Prior to testing, three OSATS evaluation tools were developed with nurse anesthesia faculty input. Six facultyevaluators pilot tested an adapted version of three OSATS tools for feasibility and validity with 23 SRNAs performing mask ventilation, LMA insertion, and endotracheal intubation with an airway management trainer. SRNAs had 15 minutes; five-minute each to perform mask ventilation, insert an LMA, and perform endotracheal intubation. Paired observations from six facultyevaluators determined inter-rater reliability. Separate survey questionnaires were created for faculty-evaluators and SRNAs; using three four-point Likert-type rating scales to measure feasibility and validity. Results: Descriptive statistics revealed that faculty-evaluators' found the OSATs tools to be feasible based on six out of seven somewhat favorable to strongly favorable responses. The Kendall's Coefficient of Concordance W showed statistical significance for mask ventilation and endotracheal intubation (p-value of .041 and .036). Six faculty-evaluators found the OSATs tools to have face and content validity based on six out of seven somewhat favorable responses. Twenty-three SRNAs found the OSATS tools to have face validity based on somewhat strongly favorable responses. Conclusions: An adapted version of OSATS tool for evaluating SRNAs performing mask ventilation, LMA insertion, and endotracheal intubation demonstrated feasibility, reliability, and validity. The inter-rater values obtained from Kendall's Coefficient of Concordance W in this project correlated with the Cook and Beckman report.

Development of a novel global rating scale for objective structured assessment of technical skills in an emergency medical simulation training

“…These findings highlight that the GRS is an easy to use tool and due to the high standardization in emergency medicine with systematic approaches, guidelines, procedures and algorithms, agreement amid instructors is generally given. Further studies considering the GRS to be a time efficient and feasible tool [ 33 , 34 ] support these results. Yet, even more consistent results might have been achieved with a preliminary briefing among the raters on how to use the tool or any objectives of the items of the GRS, as these may vary slightly in diverse simulation scenarios.…”