Three controversies over item disclosure in medical licensure examinations

Abstract

In response to views on public's right to know, there is growing attention to item disclosure – release of items, answer keys, and performance data to the public – in medical licensure examinations and their potential impact on the test's ability to measure competence and select qualified candidates. Recent debates on this issue have sparked legislative action internationally, including South Korea, with prior discussions among North American countries dating over three decades. The purpose of this study is to identify and analyze three issues associated with item disclosure in medical licensure examinations – 1) fairness and validity, 2) impact on passing levels, and 3) utility of item disclosure – by synthesizing existing literature in relation to standards in testing. Historically, the controversy over item disclosure has centered on fairness and validity. Proponents of item disclosure stress test takers’ right to know, while opponents argue from a validity perspective. Item disclosure may bias item characteristics, such as difficulty and discrimination, and has consequences on setting passing levels. To date, there has been limited research on the utility of item disclosure for large scale testing. These issues requires ongoing and careful consideration.

Differences Between Students’ and Teachers’ Fairness Perceptions: Exploring the Potential of a Self-Administered Questionnaire to Improve Teachers’ Assessment Practices

“…Note that in the following, we will focus on this procedural, hence pragmatic understanding of Fairness as a working definition. For broader and multifaceted conceptualizations of Fairness that question the utility of a single umbrella term and that explicitly include sociocultural aspects, we recommend comprehensive treatments of this topic by Gipps and Stobart (2009), Camilli (2013), Tierney (2013Tierney ( , 2014, and Rasooli et al (2018).…”

“…Thus, future studies might explore whether the Fairness Barometer and hence the measured concept of fairness can be expanded while still focusing only on aspects that students can reliably evaluate and teachers can change. We also want to stress that our working definition of fairness mostly focused on classroom justice theory and that broader conceptualizations are possible, and possibly-depending on the context-preferable (e.g., Gipps and Stobart, 2009;Camilli, 2013); Tierney (2013Tierney ( , 2014 and Rasooli et al (2018).…”

Does Repetition of the Same Test Questions in Consecutive Years Affect their Psychometric Indicators? – Five-year Analysis of In-house Exams at Medical University of Warsaw

“…Although there are research results on the influence of public availability of large banks of test questions (Gilmer, 1989;Park & Yang, 2015;Wagner-Menghin et al, 2013;Wood, 2009;Yang et al, 2018) on exam results, the conclusions from this type of research cannot be directly transferred to in-house exams, which are conducted on a much smaller scale than large country-wide standardized exams. Among teachers, there is a conviction that the use of the same MCQs for next year exam will contribute to an increase in its ease which will result in inflated scores.…”

“…Open banks of exam questions are published in numerous countries and every student can become familiar with their contents and thus prepare for the final exam or mid-term test (Considine et al, 2005;Hansen & Dexter, 1997;Park & Yang, 2015;Yang et al, 2018). It needs to be noted, however, that open task banks usually contain a few thousand of items (e.g.…”

Effects of test item disclosure on medical licensing examination

“…Maintaining the security of test items and scores is paramount if tests are to provide useful inferences on examinee performance (Cohen and Wollack 2006 ). However, publicly disclosing item-level information has historically faced challenges on grounds of fairness and validity (Bersoff 1981 ; Camilli 2006 ; Park and Yang 2015 ). Proponents of item disclosure emphasize the need for test takers to know the actual test items and answer keys in addition to other relevant performance data, based on the principle of fairness and public accountability.…”