Evaluation of adding item-response theory analysis for evaluation of the European Board of Ophthalmology Diploma examination

Abstract

Purpose: To investigate whether introduction of item-response theory (IRT) analysis, in parallel to the traditional' statistical analysis methods available for performance evaluation of multiple T/F items as used in the European Board of Ophthalmology Diploma (EBOD) examination, has proved beneficial, and secondly, to study whether the overall assessment performance of the current written part of EBOD is sufficiently high (KR-200.90) to be kept as examination format in future EBOD editions. Methods: Traditional' analysis methods for individual MCQ item performance comprise P-statistics, Rit-statistics and item discrimination, while overall reliability is evaluated through KR-20 for multiple T/F items. The additional set of statistical analysis methods for the evaluation of EBOD comprises mainly IRT analysis. These analysis techniques are used to monitor whether the introduction of negative marking for incorrect answers (since EBOD 2010) has a positive influence on the statistical performance of EBOD as a whole and its individual test items in particular. Results: Item-response theory analysis demonstrated that item performance parameters should not be evaluated individually, but should be related to one another. Before the introduction of negative marking, the overall EBOD reliability (KR-20) was good though with room for improvement (EBOD 2008: 0.81; EBOD 2009: 0.78). After the introduction of negative marking, the overall reliability of EBOD improved significantly (EBOD 2010: 0.92; EBOD 2011:0.91; EBOD 2012: 0.91). Conclusion: Although many statistical performance parameters are available to evaluate individual items, our study demonstrates that the overall reliability assessment remains the only crucial parameter to be evaluated allowing comparison. While individual item performance analysis is worthwhile to undertake as secondary analysis, drawing final conclusions seems to be more difficult. Performance parameters need to be related, as shown by IRT analysis. Therefore, IRT analysis has proved beneficial for the statistical analysis of EBOD. Introduction of negative marking has led to a significant increase in the reliability (KR-20>0.90), indicating that the current examination format can be kept for future EBOD examinations

Procedural aspects of the organization of the comprehensive European Board of Ophthalmology Diploma examination

“…EBO has been represented within UEMS-CESMA since November 2009, and the comprehensive EBOD examination was in 2016, the 3rd largest European postgraduate medical assessment with 619 candidates, after anaesthesiology with over 2,000 candidates, and cardiology with approximately 1,000 candidates [ 3 , 4 ]. However, when considering the initiative to promote in peer-reviewed and book publications, the EBOD examination is the leading specialty [ 2 , 6 , 7 , 8 , 9 ]. Until present, only the specialists sections of anesthesiology [ 5 ] and hand surgery [ 10 ] have published peer-reviewed articles about their European postgraduate medical assessments [ 3 , 4 ].…”

“…Part 1 of the comprehensive EBOD examination is a written paper and pencil test composed of 52 MCQs in a modified multiple independent true-false format (type X) [ 11 ] including an active “don’t know” option [ 2 , 6 - 9 ]. The reason for modification of the type X format relies in the use of negative marking for incorrect answers.…”

“…In order not to put too much pressure on the candidates sitting the comprehensive EBOD examination, the Examination Committee has decided to consider the “don’t know” option as a valid choice for candidates and decided not to penalize the candidate for this choice. A score of 0 will be obtained for “don’t know” answers, while a negative mark of -0.5 will be given to incorrect answers [ 2 , 8 ]. However, in type X MCQs with negative marking often a negative mark of -1 is given to wrong answers [ 10 ].…”

Can ‘Fellow of the European Board of Ophthalmology Subspecialty Diploma in Glaucoma,’ a subspecialty examination on glaucoma induce the qualification standard of glaucoma clinical practice in Europe?

“…The EBO has gained respect in the last few years, not only within Europe but also on a worldwide level, for harmonizing education and training in ophthalmology within UEMS (associated) countries. One of the means available to EBO is the organization of the comprehensive European Board of Ophthalmology Diploma (EBOD) examination, which has been organized on a yearly basis since 1995 [ 1 , 2 , 3 , 4 ]. The EBOD diploma and the related title of Fellow of the European Board of Ophthalmology (FEBO) obtained by candidates who have been successful at the comprehensive EBOD examination ensures a minimum standard of knowledge in comprehensive, general ophthalmology among specialists and residents who are in their final or fourth year of training within UEMS (associated) countries [ 5 ].…”

Introduction of subspecialty examinations by the European Board of Ophthalmology (EBO) in close collaboration with European Subspecialty Ophthalmological Societies: FEBO‐SA

“…One of the means available to EBO is the organization of the comprehensive European Board of Ophthalmology Diploma (EBOD) examination, which started in 1995 (Mathysen , ; Mathysen et al. ,b, ). The purpose of the comprehensive EBOD Examination is to ensure standardization of the knowledge level in general ophthalmology.…”