A mixed methods evaluation of the effect of confidence-based versus conventional multiple-choice questions on student performance and the learning journey

Abstract

Background Traditional single best answer multiple-choice questions (MCQs) are a proven and ubiquitous assessment tool. By their very nature, MCQs prompt students to guess a correct outcome when unsure of the answer, which may lead to a reduced ability to reliably assay student knowledge. Moreover, the traditional Single Best Answer Test (SBAT) offers binary feedback (correct or incorrect) and therefore offers no feedback or enhancement of the student learning journey. Confidence-based Answer Tests (CBATs) are designed to improve reliability because participants are not forced to guess where they cannot choose between two or more alternative answers which they may favour equally. CBATs enable students to reflect on their knowledge and better appreciate where their mastery of a particular subject may be weaker. Although CBATs can provide richer feedback to students and improve the learning journey, their use may be limited if they significantly alter student scores or grades, which may be viewed negatively. The aim of this study was to compare performance across these test paradigms, to investigate if there are any systematic biases present. Methods Thirty-four first-year optometry students and 10 lecturers undertook a test comprising 40 questions. Each question was completed using two specified test paradigms; for the first paradigm, they were allowed to weight their answers based on confidence (CBAT), and a single best answer (SBAT). Upon test completion, students undertook a survey comprising both Likert scale and open-ended responses regarding their experience and perspectives on the CBAT and SBAT multiple-choice test paradigms. These were analysed thematically. Results There was no significant difference between paradigms, with a median difference of 1.25% (p = 0.313, Kruskal-Wallis) in students and 3.33% (p = 0.437, Kruskal-Wallis) in staff. The survey indicated that students had no strong preference towards a particular method. Conclusions Since there was no significant difference between test paradigms, this validates implementation of the confidence-based paradigm as an equivalent and viable option for traditional MCQs but with the added potential benefit that, if coupled with reflective practice, can provide students with a richer learning experience. There is no inherent bias within one method over another.