Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study

Abstract

ObjectivesThe study aimed to compare candidate performance between traditional best-of-five single-best-answer (SBA) questions and very-short-answer (VSA) questions, in which candidates must generate their own answers of between one and five words. The primary objective was to determine if the mean positive cue rate for SBAs exceeded the null hypothesis guessing rate of 20%.DesignThis was a cross-sectional study undertaken in 2018.Setting20 medical schools in the UK.Participants1417 volunteer medical students preparing for their final undergraduate medicine examinations (total eligible population across all UK medical schools approximately 7500).InterventionsStudents completed a 50-question VSA test, followed immediately by the same test in SBA format, using a novel digital exam delivery platform which also facilitated rapid marking of VSAs.Main outcome measuresThe main outcome measure was the mean positive cue rate across SBAs: the percentage of students getting the SBA format of the question correct after getting the VSA format incorrect. Internal consistency, item discrimination and the pass rate using Cohen standard setting for VSAs and SBAs were also evaluated, and a cost analysis in terms of marking the VSA was performed.ResultsThe study was completed by 1417 students. Mean student scores were 21 percentage points higher for SBAs. The mean positive cue rate was 42.7% (95% CI 36.8% to 48.6%), one-sample t-test against ≤20%: t=7.53, p<0.001. Internal consistency was higher for VSAs than SBAs and the median item discrimination equivalent. The estimated marking cost was £2655 ($3500), with 24.5 hours of clinician time required (1.25 s per student per question).ConclusionsSBA questions can give a false impression of students’ competence. VSAs appear to have greater authenticity and can provide useful information regarding students’ cognitive errors, helping to improve learning as well as assessment. Electronic delivery and marking of VSAs is feasible and cost-effective.

Situational judgement test validity for selection: A systematic review and meta‐analysis

“…Thus, it is not possible to say whether removing such answering cues, for example, by using free text responses, would increase the validity of such assessments. Indeed, in this regard, there is some evidence that, at least for semantic knowledge tests, free text response format questions are generally experienced as more difficult compared to the equivalent SRQs 80 . With advances in natural language processing and machine learning comes the increasingly plausible possibility of automating, or semi‐automating, the scoring of such responses.…”

Question-Based Collaborative Learning for Constructive Curricular Alignment

“…Given our previous work on VSAQs being more representative of real-life practice than MCQs, it would be interesting to establish the level of learning assessed by this form of student-generated question, by collecting both quantitative and qualitative evaluative data. 13 We hope to expand this work by incorporating this collaborative activity into the students' clinical placements. The aim of this is to promote student engagement with the curriculum and provide further opportunities for students and faculty to identify any learning and teaching gaps.…”

Very short answer questions: a viable alternative to multiple choice questions

“…The demerits of MCQ are many, and there is strong backing for VSAQ, but the transition, as usual, is slow to come. An electronic VSA exam platform has been developed by the UK Medical Schools Council Assessment Alliance to complement their existing SBA platform, which is already widely used by medical schools throughout the UK [24]. We are encouraged by the finding by Sam et al [25] that VSAQ format is capable of high reliability, validity, discrimination, and authenticity, while SBAQ format was associated with significant cueing.…”