Does allowing access to electronic differential diagnosis support threaten the reliability of a licensing exam?

Abstract

IntroductionNewer electronic differential diagnosis supports (EDSs) are efficient and effective at improving diagnostic skill. Although these supports are encouraged in practice, they are prohibited in medical licensing examinations. The purpose of this study is to determine how using an EDS impacts examinees' results when answering clinical diagnosis questions.MethodThe authors recruited 100 medical students from McMaster University (Hamilton, Ontario) to answer 40 clinical diagnosis questions in a simulated examination in 2021. Of these, 50 were first‐year students and 50 were final‐year students. Participants from each year of study were randomised into one of two groups. During the survey, half of the students had access to Isabel (an EDS) and half did not. Differences were explored using analysis of variance (ANOVA), and reliability estimates were compared for each group.ResultsTest scores were higher for final‐year versus first‐year students (53 ± 13% versus 29 ± 10, p < 0.001) and higher with the use of EDS (44 ± 28% versus 36 ± 26%, p < 0.001). Students using the EDS took longer to complete the test (p < 0.001). Internal consistency reliability (Cronbach's alpha) increased with EDS use among final‐year students but was reduced among first‐year students, although the effect was not significant. A similar pattern was noted in item discrimination, which was significant.ConclusionEDS use during diagnostic licensing style questions was associated with modest improvements in performance, increased discrimination in senior students and increased testing time. Given that clinicians have access to EDS in routine clinical practice, allowing EDS use for diagnostic questions would maintain ecological validity of testing while preserving important psychometric test characteristics.

Rethinking assessment in response to generative artificial intelligence

“…The use of decision‐making support tools during assessments, such as electronic differential diagnosis in examinations, 1 is just the tip of the iceberg when it comes to how technology is currently changing assessment practice. We say this, because it appears we have reached a transformative stage in the development of artificial intelligence (AI).…”

“…Going forward, we see an important distinction between ‘assisted’ assessments and ‘unassisted’ assessments. The former is assessment that allows the candidate to draw on tools and resources, including textbooks, the Internet, decision‐making support tools 1 and now, generative AI models—in many ways, representative of real‐life clinical practice. Unassisted assessment, on the other hand, refers to times when we may wish to assess our students' and trainees' knowledge and understanding without access to such resources, such as for certification or summative contexts when the independent (and verifiable) demonstration of clinical knowledge and reasoning is a necessary element of the relevant competencies.…”

“…We need to think carefully about the kind of performance we want our assessments to elicit. As Sibbald et al's study shows, the use of electronic diagnosis software to ‘assist’ performance on exams does not necessarily undermine their psychometric properties 1 . But can we infer genuine understanding and ‘knows how’ (in Miller's terms) 5 from such assisted performance?…”