This paper presents validity evidence for a newly developed, procedure specific assessment tool, Open Aortic Aneurysm Repair Assessment of Technical Expertise (OPERATE), assessing important technical steps of open abdominal aortic aneurysm repair on a simulator, including a credible pass/fail score. The OPERATE can be used to provide structured feedback during simulation based training and to ensure basic open technical skills before supervised training on patients. Objective: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard. Methods: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting.All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score. Results: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 AE 12 and 25.6 AE 6, respectively (p ¼ .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed. Conclusion: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.

No abstract

No abstract

Background and aim Recognition of structured training in endovascular aortic repair (EVAR) for vascular trainees is increasing. Nevertheless, how trainees can achieve sufficient skills in EVAR sizing and graft selection is sparsely described. The aim of this study was to investigate the effect of systematic training in basic EVAR sizing and graft selection on vascular surgery trainees using a validated assessment tool. Methods Sixteen vascular surgery trainees were included in an intensive 6-h hands-on workshop in aortic sizing and stent graft selection for EVAR with a trainer-to-trainee ratio of 1:2. After 1-h lecture, participants did 5 h of supervised training on increasingly complex cases. Finally, the participants were tested using a validated assessment tool. Results All participants were able to size the test-case and select a stent graft combination in 24:35 (13:30–48:20) min (median and range). The participants’ overall test scores (lower is better) were in median 17.9 (11.9–28.4). This did not differ from the scores of experienced EVAR operators 14.7 (11.7–25.2) (<200 EVAR’s) ( p = .32) but was inferior to the score of EVAR experts 11.2 (9.8 –18.7) (≥200 EVAR’s) ( p = .01). The sub-score for anatomical measurements was 10.6 (3.9–18.8) and comparable with the experienced group 9.7 (8.1–12.8) ( p = .83) but inferior to the expert operators 6.5 (5.2–10.2) ( p = .04). The sub-score for stent graft selection was 7.5 (4.9–14.1) and comparable with experienced operators scoring 4.5 (3.6–12.3) ( p = .09) but inferior to the expert operators score of 5.0 (3.6–8.4) ( p = .01). Conclusion This study presents the results of a standardised one-day basic EVAR sizing and graft selection workshop. Vascular surgery trainees with no prior EVAR experience learned to size and select stent grafts for a simple infra-renal AAA on par with experienced EVAR operators.

No abstract

No abstract

Background: The aims of the study were to develop an assessment tool in local anaesthetic thoracoscopy (LAT), investigate validity evidence, and establish a pass/fail standard.Methods: Validity evidence for the assessment tool was gathered using the unified Messick framework.The tool was developed by five experts in respiratory medicine and medical education. Doctors with varying experience performed two consecutive procedures in a standardized, simulation-based setting using a newly developed thorax/lung silicone model. Performances were video-recorded and assessed by four expert raters using the new tool. Contrasting groups' method was used to set a pass/fail standard.Results: Nine novices and 8 experienced participants were included, generating 34 recorded performances and 136 expert assessments. The tool had a high internal consistency (Cronbach's alpha =0.94) and high inter-rater reliability (Cronbach's alpha =0.91). The total item score significantly correlated with the global score (r s =0.86, P<0.001). Participants' first performance correlated to second performance (test-retest reliability) with a Pearson's r of 0.93, P<0.001. Generalisability (G) study showed a G-coefficient of 0.92 and decision (D) study estimated that one performance assessed by two raters or four performances assessed by one rater are needed to reach an acceptable reliability, i.e., G-coefficient >0.80. The tool was able to discriminate between the two groups in both performances: experienced mean score =30.8±4.2; novice mean score =15.8±2.3, P<0.001. Pass/fail standard was set at 22 points. Conclusions:The newly developed assessment tool showed solid evidence of validity and can be used to ensure competence in LAT.

No abstract