Comparison of Online Tests of Very Short Answer versus Single Best Answers for Medical Students in a Pharmacology Course over One Year

Abstract

Single best answers (single choice, SC) are the classical tools used in medical examinations on all levels of education. In contrast, very short answer (VSA) probably requires sound prior knowledge and deeper learning than SC, and VSA should make cueing and guessing impossible. Hence, in a basic pharmacology course, we wanted to compare the SC and VSA formats directly at the end of the course and one year later. Medical students (n = 211) were given a formative online test. Two groups were randomly formed (A and B). Participants in group A were first given fifteen single choices (one out of five) pharmacology questions and thereafter fifteen very short answer questions (open question which were to be answered online and semiautomatically assessed). Very similar questions with regard to learning objectives but in opposite order were given to group B. After one year, about half of students from group A were again given the very same questions (AA) or the opposite questions (AB). Likewise, group B was again tested with the opposite questions (BA) or the same (BB). The SC questions in groups A, AA, AB, B, BA, and BB were in sum easier to answer than the corresponding VSA questions. Repeating the test after one year with the same students increased retention of right answers by about 1.5 points. In summary, direct comparison questions in the VSA format are more difficult for our students to answer than questions in the SC format, conceivably because cueing and guessing are eliminated. Knowledge retention is present by repeating the very same examination format online. Retention of knowledge is higher when starting with VSA (group B) both for a subsequent SC format or a VSA format. These data would argue for more use of the VSA format at least in pharmacology examinations.

Assessment during Covid-19: quality assurance of an online open book formative examination for undergraduate medical students

“…The OBE items in this study were highly discriminatory between high performing and low performing students as 86.6% of the items were found to have moderate to high discrimination indices. This finding is in similarity with other studies which observed that tests with open-ended items can better discriminate between well-prepared students and marginal students, as they allow more possibility for differentiation in scores [ 21 , 30 , 31 ].…”

Comparison of Very Short Answer Questions and Multiple Choice Questions in Medical Students: Reliability, Discrimination, Acceptability and Effect on Knowledge Retention

“…Last, we cannot rule out COVID-19 related online teaching interference with the motivation and education of students [26-28]. Although our study remains inconclusive regarding knowledge retention, other studies have supported the positive effect of VSAQs on long-term retention, indicating higher retention in students taking an initial VSAQ test compared to a MCQ test [15-17], which is most likely explained by the greater level of retrieval difficulty demanded by VSAQs leading to better retention of information [25, 29].…”

“…Furthermore, transfer-appropriate-processing might apply here, which states that memory is enhanced when the initial encoding and later retrieval of the information are similar [24], because the information is encoded using VSAQs but retrieved in the PT using MCQs. However, other studies have shown information retention occurring regardless of the final test format [15,25]. Because students had not been sufficiently assessed with VSAQs before and therefore were not used to this question type, they might not have adjusted their learning strategy yet, even though a substantial number of students stated that they would change the way they studied if they were to be assessed with VSAQs.…”

“…The open-ended nature of the VSAQ reflects the clinical practice better and may prevent surface-level study approaches and cueing [1, 3, 4, 8]. In addition, VSAQs are better able to discriminate between students based on proficiency in the content knowledge [1-4, 13, 14] and may increase retention of knowledge [15-17].…”

Introducing Group Open-Book Exams as a Learning and Assessment Strategy in the Clinical Biochemistry Course for Medical Students

“…Neumann et al [ 32 ] reported good discrimination power between high- and low-performing students in the OBE. Our results confirmed their findings and showed a higher discrimination ability for CBEs.…”