The Effects of Test Disclosure on Equated Scores and Pass Rates

Abstract

This paper examines the effects of test item disclosure on resulting examinee equated scores and population passing rates. The equating model studied was the common-item nonequivalent-populations design under Tucker linear equating procedures. The research involved simulating disclosure by placing correct answers of "disclosed" items into response vectors of selected examinees. The degree of exposure the disclosed items received in the population was manipulated by varying the number of items disclosed and the number of examinee records receiving the correct answers. Other factors considered among the 10 experimental conditions included the characteristics of the disclosed items (difficulty of disclosed items and whether they were anchor or nonanchor test items) and the ability level of the subgroup receiving the disclosed items. Results suggest that effects of disclosure depend on the nature of the released items. Specific effects of disclosure on particular examinees are also discussed. Index terms: equated scores, licensing exams, passing rates, simulated disclosure, test disclosure. The issue of test disclosure has become the focus of the debate surrounding truth-in-testing legislation. Between 1977 and 1983, approximately 90 testing bills were introduced in 28 states; five bills were introduced in the U.S. House of Representatives (Greer, 1984a). Although only two states-California and New York-have enacted truth-in

Three controversies over item disclosure in medical licensure examinations

“…A test designed without an appropriate process of score equating will not factor in variables such as the difficulty of items and changes in test environment. Based on an analysis on the influence of item disclosure, Gilmer found that continuous disclosure of test items would result in higher pass rates regardless of examinees’ performance, which would lead to unfair benefits for those who share the previous test items ( 19 ).…”

Does Repetition of the Same Test Questions in Consecutive Years Affect their Psychometric Indicators? – Five-year Analysis of In-house Exams at Medical University of Warsaw

“…Although there are research results on the influence of public availability of large banks of test questions (Gilmer, 1989;Park & Yang, 2015;Wagner-Menghin et al, 2013;Wood, 2009;Yang et al, 2018) on exam results, the conclusions from this type of research cannot be directly transferred to in-house exams, which are conducted on a much smaller scale than large country-wide standardized exams. Among teachers, there is a conviction that the use of the same MCQs for next year exam will contribute to an increase in its ease which will result in inflated scores.…”

Effects of Misbehaving Common Items on Aggregate Scores and an Application of the Mantel-Haenszel Statistic in Test Equating

“…More importantly, even at modest exposure levels, IRT equated score distributions altered considerably. Under a traditional Tucker linear equating method, Gilmer (1989) found modest effects on the passing rates on a certification test due to simulated item disclosure.…”