Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists

Abstract

Background We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods. Methods A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant’s technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist. Results Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance. Conclusions Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.

Making the clinical connection from textbook to bedside during MDY1: an integrative approach for medical physiology education employing human simulation

“…The implementation of computer-assisted instruction systems has traditionally been conducted in advanced training settings for the purpose of introducing novel technology resulting from new developments in science. Some of the first simulation centers included Harvard University, which originally established training for anesthesia residents and academic anesthesiology faculty members, and are still in use (20). Others have used simulation for the purposes of team building in medical settings, where multiple departmental entities come together to accomplish a common goal (i.e., surgical teams).…”

Reliability of Simulation-Based Assessment for Practicing Physicians: Performance is Context-Specific

“…Cases were speci cally designed to measure diagnostic, clinical management, and teamwork skills. A summary of the scenarios is outlined in Table 1 and full details about case development and evaluation can be found elsewhere 2,8 .…”

“…High delity simulation is increasingly used across all levels of training and is effective for improving the performance of individual physicians in a manner that is highly relevant to actual clinical performance 1 . We recently demonstrated that simulationbased assessment using complex scenarios requiring critical thinking and crisis resource management skills can be developed and administered in a standardized way at multiple sites 2 . Nevertheless, issues concerning the reliability and validity of simulation-based performance assessment scores for practicing physicians, both individually and as members of a team, still remain.…”

“…In addition, an ongoing population-wide evaluation of the abilities of physician groups could provide feedback to training programs, continuing medical education, departmental leadership, and other stakeholders. Through deeper analysis of a subset of data from the study by Weinger et al 2 , we begin to address whether simulation-based assessment, employing high-delity manikins, can provide reliable estimates of the technical and behavioral performance of practicing anesthesiologists.…”

The extracorporeal membrane oxygenation (ECMO) high-fidelity simulator: the best complementary tool to learn the technique

“…However, because troubleshooting simulations and animal testing are usually expensive and available less often, water-drill-based ECMO training might be limited. On the other hand, because it seems to be highly effective, with long-lasting take-away skills, ECMO-simulation-based apprenticeship could be particularly useful for intensivists (10,11). Using this pedagogical tool to train ECMO users could offer promising outcomes but have rarely been studied (12,13).…”