Usability Testing Via Simulation

Abstract

Objectives To inform development of a preintubation checklist for pediatric emergency departments via multicenter usability testing of a prototype checklist. Methods This was a prospective, mixed methods study across 7 sites in the National Emergency Airway Registry for Pediatric Emergency Medicine (NEAR4PEM) collaborative. Pediatric emergency medicine attending physicians and senior fellows at each site were first oriented to a checklist prototype, including content previously identified using a modified Delphi approach. Each site used the checklist in 2 simulated cases: an “easy airway” and a “difficult airway” scenario. Facilitators recorded verbalization, completion, and timing of checklist items. After each simulation, participants completed an anonymous usability survey. Structured debriefings were used to gather additional feedback on checklist usability. Comments from the surveys and debriefing were qualitatively analyzed using a framework approach. Responses informed human factors-based optimization of the checklist. Results Fifty-five pediatric emergency medicine physicians/fellows (4–13 per site) participated. Participants found the prototype checklist to be helpful, easy to use, clear, and of appropriate length. During the simulations, 93% of checklist items were verbalized and more than 80% were completed. Median time to checklist completion was 6.2 minutes (interquartile range, 4.8–7.1) for the first scenario and 4.2 minutes (interquartile range, 2.7–5.8) for the second. Survey and debriefing data identified the following strengths: facilitating a shared mental model, cognitively offloading the team leader, and prompting contingency planning. Suggestions for checklist improvement included clarifying specific items, providing more detailed prompts, and allowing institution-specific customization. Integration of these data with human factors heuristic inspection resulted in a final checklist. Conclusions Simulation-based, human factors usability testing of the National Emergency Airway Registry for Pediatric Emergency Medicine Preintubation Checklist allowed optimization prior to clinical implementation. Next steps involve integration into real-world settings utilizing rigorous implementation science strategies, with concurrent evaluation of the impact on patient outcomes and safety.