Standard Setting of Competency in Mastoidectomy for the Cross-Institutional Mastoidectomy Assessment Tool

Abstract

Objective: Competency-based surgical training involves progressive autonomy given to the trainee. This requires systematic and evidence-based assessment with well-defined standards of proficiency. The objective of this study is to develop standards for the cross-institutional mastoidectomy assessment tool to inform decisions regarding whether a resident demonstrates sufficient skill to perform a mastoidectomy with or without supervision. Methods: A panel of fellowship-trained content experts in mastoidectomy was surveyed in relation to the 16 items of the assessment tool to determine the skills needed for supervised and unsupervised surgery. We examined the consensus score to investigate the degree of agreement among respondents for each survey item as well as additional analyses to determine whether the reported skill level required for each survey item was significantly different for the supervised versus unsupervised level. Results: Ten panelists representing different US training programs responded. There was considerable consensus on cut-off scores for each item and trainee level between panelists, with moderate (0.62) to very high (0.95) consensus scores depending on assessment item. Further analyses demonstrated that the difference between supervised and unsupervised skill levels was significantly meaningful for all items. Finally, minimum-passing scores for each item was established. Conclusion: We defined performance standards for the cross-institutional mastoidectomy assessment tool using the Angoff method. These cut-off scores that can be used to determine when trainees can progress from performance under supervision to performance without supervision. This can be used to guide training in a competency-based training curriculum.

Standard Setting in Simulation-based Training of Surgical Procedures

“…In the category of item-centered standard setting methods, the Angoff method was used in 9 studies. 29,35,[53][54][55][56][57][58][59] This method allows content experts to make judgement on each individual item, then to determine if a borderline trainee who is just at the edge of minimal competence will pass that exact part of the task or checklist. We found the method to be applied with a number of variations, which were more or less in concordance with the original method.…”

3-D-Printed Models for Temporal Bone Training: A Validity Study

“…A previous study has established a cut-off score for virtual reality (VR) training of mastoidectomy using the modified WS; this cut-off score was found using the upper 10% of expert performances and set to 19.5 points of 26 points (38). Another study used a considerably different approach for setting a standard performance (the Angoff method) for a different mastoidectomy performance assessment tool (39). This emphasizes that standard setting is challenging and might be based on different methods.…”