Intermethod Reliability of Real‐time Versus Delayed Videotaped Evaluation of a High‐fidelity Medical Simulation Septic Shock Scenario

Abstract

Objectives: High-fidelity medical simulation (HFMS) is increasingly utilized in resident education and evaluation. No criterion standard of assessing performance currently exists. This study compared the intermethod reliability of real-time versus videotaped evaluation of HFMS participant performance.Methods: Twenty-five emergency medicine residents and one transitional resident participated in a septic shock HFMS scenario. Four evaluators assessed the performance of participants on technical (26-item yes ⁄ no completion) and nontechnical (seven item, five-point Likert scale assessment) scorecards. Two evaluators provided assessment in real time, and two provided delayed videotape review. After 13 scenarios, evaluators crossed over and completed the scenarios in the opposite method. Real-time evaluations were completed immediately at the end of the simulation; videotape reviewers were allowed to review the scenarios with no time limit. Agreement between raters was tested using the intraclass correlation coefficient (ICC), with Cronbach's alpha used to measure consistency among items on the scores on the checklists. Conclusions: Real-time and videotaped-based evaluations of resident performance of both technical and nontechnical skills during an HFMS septic shock scenario provided equally reliable methods of assessment.ACADEMIC EMERGENCY MEDICINE 2009; 16:887-893 ª

Prospective Comparison of Live Evaluation and Video Review in the Evaluation of Operator Performance in a Pediatric Emergency Airway Simulation

“…11,12 No study to date has evaluated the reliability of video compared to live assessment during pediatric rapid sequence intubation (pRSI). 13 We evaluated emergency medicine residents' performance during simulation of critical procedures to determine whether the reviewer actually needed to be present. Our specific aims were to demonstrate that (1) modification of a previously validated assessment tool provides reliable performance data in both live and video review environments and that (2) this tool is able to discriminate learner level of training.…”

Assessing Patient Care: Summary of the Breakout Group on Assessment of Observable Learner Performance

“…These searches were combined with the additional search terms "resident* or medical student*" (58,880) resulting in 414 and 267 final results, respectively. After reviewing for relevance, 76 articles remained.…”

“…35 Video assessments would allow multiple assessments of one learner's performance without requiring all faculty members to be present during the simulation session. Williams et al 58 have demonstrated that assessment of videotaped sessions have comparable inter-rater reliability when compared to real-time assessment.OCSEs. Objective structured clinical examinations are routinely used to evaluate multiple ACGME core competencies and are particularly useful for those that involve direct patient contact (data gathering, assimilation of data, and patient management).…”

“…35 Video assessments would allow multiple assessments of one learner's performance without requiring all faculty members to be present during the simulation session. Williams et al 58 have demonstrated that assessment of videotaped sessions have comparable inter-rater reliability when compared to real-time assessment.…”

Simulation in Emergency Medicine Training

“…In describing their model simulation curriculum for EM, McLaughlin et al 127 noted that they limited simulation-based assessment to formative evaluation. 137,141 Gordon et al 142 used the high-stakes assessment tool developed for the EM oral board examination. Issues surrounding the need for assessment instruments were thoroughly reviewed in the 2008 AEM Consensus Conference breakout session addressing the use of simulation in the assessment of emergency physicians.…”