Exploring Clinical Reasoning Strategies and Test-Taking Behaviors During Clinical Vignette Style Multiple-Choice Examinations: A Mixed Methods Study

Abstract

Background Clinical vignette multiple-choice questions (MCQs) are widely used in medical education, but clinical reasoning (CR) strategies employed when approaching these questions have not been well described. Objectives The aims of the study were (1) to identify CR strategies and test-taking (TT) behaviors of physician trainees while solving clinical vignette MCQs; and (2) to examine the relationships between CR strategies and behaviors, and performance on a high-stakes clinical vignette MCQ examination. Methods Thirteen postgraduate year–1 level trainees completed 6 clinical vignette MCQs using a think-aloud protocol. Thematic analysis employing elements of grounded theory was performed on data transcriptions to identify CR strategies and TT behaviors. Participants' CR strategies and TT behaviors were then compared with their US Medical Licensing Examination Step 2 Clinical Knowledge scores. Results Twelve CR strategies and TT behaviors were identified. Individuals with low performance on Step 2 Clinical Knowledge demonstrated increased premature closure and increased faulty knowledge, and showed comparatively less ruling out of alternatives or admission of knowledge deficits. High performers on Step 2 Clinical Knowledge demonstrated increased ruling out of alternatives and admission of knowledge deficits, and less premature closure, faulty knowledge, or closure prior to reading the alternatives. Conclusions Different patterns of CR strategies and TT behaviors may be used by high and low performers during high-stakes clinical vignette MCQ examinations.

Exploring examinee behaviours as validity evidence for multiple-choice question examinations

“…One other study, co‐authored by SJD and DT, explored reasoning using similar methods and found alike themes of identifying key features and non‐analytical reasoning, as well as some parallel test‐taking themes (i.e. process of elimination) among their participants . The presence of these findings in a sample composed of all postgraduate Year 1 trainees at different medical centres using different MCQ items and an alternative coding method is an important piece of evidence that the findings presented here are not merely an idiosyncratic occurrence.…”

“…With participants from several career stages, we limit the chances of ‘missing’ phenomena that may have been unique to participants at any one stage. The breadth of content, with several items from multiple content domains, is also unique compared with similar studies . These questions were notably of high quality.…”

“…Compared with other studies, our study uniquely includes participants from multiple levels of training . With participants from several career stages, we limit the chances of ‘missing’ phenomena that may have been unique to participants at any one stage.…”

“…Compared with other studies, our study uniquely includes participants from multiple levels of training. 27,[37][38][39][40] With participants from several career stages, we limit the chances of 'missing' phenomena that may have been unique to participants at any one stage. The breadth of content, with several items from multiple content domains, is also unique compared with similar studies.…”

“…The breadth of content, with several items from multiple content domains, is also unique compared with similar studies. [37][38][39][40] These questions were notably of high quality. Taken from the American College of Physicians (ACP) MKSAP 15 and MKSAP for Students 4 item banks, they were developed by expert question writers and went through peer review.…”