Evaluation of an Intervention to Improve Quality of Single-best Answer Multiple-choice Questions

Abstract

IntroductionDespite the ubiquity of single-best answer multiple-choice questions (MCQ) in assessments throughout medical education, question writers often receive little to no formal training, potentially decreasing the validity of assessments. While lengthy training opportunities in item writing exist, the availability of brief interventions is limited.MethodsWe developed and performed an initial validation of an item-quality assessment tool and measured the impact of a brief educational intervention on the quality of single-best answer MCQs.ResultsThe item-quality assessment tool demonstrated moderate internal structure evidence when applied to the 20 practice questions (κ=.671, p<.001) and excellent internal structure when applied to the true dataset (κ=0.904, p<.001). Quality scale scores for pre-intervention questions ranged from 2–6 with a mean ± standard deviation (SD) of 3.79 ± 1.23, while post-intervention scores ranged from 4–6 with a mean ± SD of 5.42 ± 0.69. The post-intervention scores were significantly higher than the pre-intervention scores, x2(1) =38, p <0.001.ConclusionOur study demonstrated short-term improvement in single-best answer MCQ writing quality after a brief, open-access lecture, as measured by a simple, novel, grading rubric with reasonable validity evidence.

Efficacy of Faculty Development Training Workshops (FDTWs) on Writing High-Quality Multiple-Choice Questions at Northern Border University (NBU) in the Kingdom of Saudi Arabia (KSA)

“…Through the participants’ feedback, the need to conduct further workshops can be understood. Scot et al [ 23 ] demonstrated how a brief intervention in the form of a lecture brought about short-term improvement in the quality of MCQ writing [ 23 ]. To ensure that this improvement is consistent and reliable, further faculty development workshops aimed at improving MCQ writing skills must have a proper structure and be organized at regular intervals [ 5 ].…”

Análisis de instrumentos evaluativos empleados en ciencias durante la pandemia: selección múltiple, indicadores y rendimiento

“…En cuanto a las limitaciones de este trabajo se encuentran el disponer de los resultados y la retroalimentación de los indicadores de las preguntas posteriormente a la aplicación del instrumento 32 , criterio que no pudo ser subsanado; sin embargo, una fortaleza es la posibilidad de mejorar la evaluación sistemáticamente a partir del extenso banco de preguntas examinado y del conocimiento sobre las preguntas que discriminan eficientemente frente a aquellas que presentan debilidad en este ítem 33,34 .…”

Generation of Medical Case-Based Multiple-Choice Questions

“…The purpose of this narrative review is to provide a thorough examination of the development of medical case-based MCQs, covering their origins and recent technology breakthroughs and discussing their importance, methodology, and potential future directions [5][6][7][8].…”

“…Over-Reliance on Single Best Answer: The practical situations encountered in the field of medicine rarely possess unequivocal resolutions. According to Scott et al (2018), the utilization of single-best-answer MCQs may occasionally result in an oversimplification of intricate clinical settings [8].…”