Item Analysis of Multiple Choice and Extended Matching Questions in the Final MBBS Medicine and Therapeutics Examination

Abstract

Background: Most universities around the world use the multiple-choice question (MCQ) examination format to evaluate medical education. However, the suitability and advantages of traditional MCQs and extended matching questions (EMQs) continue to be debated. Objectives: This study mainly aimed to perform a comprehensive comparative analysis of the performance of the EMQ and traditional MCQ formats in the final MBBS exit examination. Methods: We conducted an item analysis of 80 EMQs, and 200 MCQs administered to 532 examinees across the four campuses of the University of the West Indies during the final MBBS medicine and therapeutics examination of 2019. Exam performance measures included central tendency, item discrimination, reliability, item difficulty, and distractor efficacy. Results: For the 532 students who sat the exam, the highest, lowest, and mean (± SD) scores for the EMQs were 93, 41, and 69.0 (± 9.8), respectively; for the MCQs, the respective values were 82, 41, and 62.7 (± 7.4). The predictive value of the EMQ and MCQ grades individually in the overall failure was 0.67 (95% CI = 0.39, 0.87) and 0.89 (95% CI = 0.65, 0.98), respectively. KR-20 coefficients for the EMQs and MCQs ranged from 0.52 to 0.70 and 0.71 to 0.79, respectively. The proportion of questions with two or more functional distractors was consistently higher for the MCQs than for the EMQs in all four cohorts of students. Conclusions: The MCQs were more predictive of the overall failure and had higher inter-item reliability, making the MCQ format more suitable for high-stakes examinations.

Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy

“…These questions assess students’ ability to integrate information, reflecting real-world challenges, and shaping competent professionals. 14–16 …”

“…[10][11][12][13] Analyzing MCQs in medical education is crucial as it allows educators to assess the effectiveness of questions in testing higher-order thinking and clinical reasoning skills, ensuring assessments accurately reflect the skills required for medical practice. [14][15][16] While AI's capabilities in answering queries and simulating scenarios are noteworthy, the depth and breadth of its understanding, especially concerning multiplechoice questions (MCQs) of medical exams, still need to be thoroughly evaluated. 17 Several studies have demonstrated that ChatGPT outperforms medical students on MCQ items in board and licensing exams.…”

“…These questions assess students' ability to integrate information, reflecting real-world challenges, and shaping competent professionals. [14][15][16] A systematic analysis using Bloom's Taxonomy as a benchmark can offer a more structured and comprehensive understanding of AI's role in this domain. 22 Bloom's Taxonomy, a foundational educational framework, categorizes cognitive learning objectives into hierarchical levels, ranging from basic knowledge recall to complex evaluation and synthesis.…”