Ensuring Competency in Open Aortic Aneurysm Repair – Development and Validation of a New Assessment Tool

Abstract

This paper presents validity evidence for a newly developed, procedure specific assessment tool, Open Aortic Aneurysm Repair Assessment of Technical Expertise (OPERATE), assessing important technical steps of open abdominal aortic aneurysm repair on a simulator, including a credible pass/fail score. The OPERATE can be used to provide structured feedback during simulation based training and to ensure basic open technical skills before supervised training on patients. Objective: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard. Methods: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting.All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score. Results: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 AE 12 and 25.6 AE 6, respectively (p ¼ .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed. Conclusion: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.

How often will midwives and obstetricians experience obstetric emergencies or high-risk deliveries: a national cross-sectional study

“…51 Rare events in these medical specialties also represent a challenge in terms of learning the required skills via the traditional apprenticeship model and ongoing clinical work, which is why simulation-based skills training is necessary. 52 Finally, the results from this study only provide a part of the overall picture of training and maintaining the competences in managing rare complications.…”

“… 51 Rare events in these medical specialties also represent a challenge in terms of learning the required skills via the traditional apprenticeship model and ongoing clinical work, which is why simulation-based skills training is necessary. 52 …”

Reply

“…Abbott et al 1 also advise against the 2017 European Society for Vascular Surgery (ESVS) guidelines, 5 which provide a class IIa recommendation for patients with 60% to 99% ACS in the presence of specific markers (eg, silent infarcts on computed tomography scan or magnetic resonance imaging, ACS progression, plaque hemorrhage, embolization on transcranial Doppler ultrasound). Where did the authors find the evidence to claim that markers proposed by the ESVS should not be used to justify carotid procedures in ACS patients with unstable plaque?…”

“…Where did the authors find the evidence to claim that markers proposed by the ESVS should not be used to justify carotid procedures in ACS patients with unstable plaque? Rebutting the 2017 ESVS guidelines 5 and claiming that all ACS patients should be managed with BMT alone 1 have not been proven.…”

“…Pursuant to this goal, our research group has recently conducted a study to develop and to gather validity evidence for a procedure-specific assessment tool that includes the most important technical steps in performing an open AAA repair on a simulator, the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE). 5 OPERATE showed strong validity evidence based on the five sources by Messick, including a credible pass-fail score to allow mastery learning (ie, training to a predefined level of proficiency). We propose that this tool be used in providing feedback during simulation-based training and for summative assessment before supervised training on patients.…”