A Latent Trait Look at Pretest-Posttest Validation of Criterion-referenced Test Items

Abstract

Since Cox and Vargas (1966) introduced their pretest-posttest validity index for criterion-referenced test items, a great number of additions and modifications havefollowed. All are based on the idea of gain scoring; that is, they are computedfrom the differences between proportions ofpretest andposttest item responses. Although the method is simple and generally considered as the prototype of criterion-referenced item analysis, it has many and serious disadvantages. Some of these go back to thefact that it leads to indices based on a dual test administration-and population-dependent item p values. Others have to do with the global information about the discriminating power that these indices provide, the implicit weighting they suppose, and the meaningless maximization of posttest scores they lead to. Analyzing the pretest-posttest methodfrom a latent trait point of view, it is proposed to replace indices like Cox and Vargas' Dpp by an evaluation of the item informationfunctionfor the mastery score. An empirical study was conducted to compare the differences in item selection between both methods.As in any other area of educational and psychological measurement, more attention has been paid to reliability than to validity aspects of criterion-referenced measurement. Several test parameters have been proposed and compared with their normreferenced counterparts, assessment methods have been introduced and examined using both real and simulated data, and the criterion-referenced reliability problem seems on its way to a great diversity of solutions (Hambleton & Novick, 1973; Huynh, 1976a Huynh, , 1976bLivingston, 1972;Marshall, 1975 That less powerful efforts have been made to tackle the validity problem may in part be due to a standpoint advocated by, for example, Millman (1974). According to this standpoint, criterion-referenced validity is the same as content validity and to establish this the construction of a well-defined domain of items is sufficient. Once an item is included in the domain no empirical information or item analysis can or Thanks are due to Fred N. Kerlinger, Gideon J. Mellenbergh, Robert F. van Naerssen, and Egbert Warries for their helpful comments; to Hans van Aalst, Fred Boesenkool, Kees Hellingman, Ton Heuvelman, Rien Steen, Niels Veldhuizen, Ronny Wierstra, and Theo Wubbels for participating in the empirical study and computational assistance; and to Paula Achterberg for typing the manuscript. 379WIM J. VAN These authors distinguish three approaches to the validity problem. The first is the aforementioned item form or item generation rule approach. In it, the fixed syntactical structure and variable elements of item sentences are used to define domains and-eventually with the aid of the computer-to sample items. Item validity is automatically guaranteed because the definition of the domain and the construction of the items are accomplished y the same set of rules. The second approach is a judgmental procedure in which content specialists are retained. The judgmental task may assume ...

Surrogate indices for item information functions in criterion-referenced language testing

“…Although traditional NRM statistics may be used, Popham (1978;1981) has noted that because CRTs frequently have nonnormal distributions, the interpretation of such indices may be difficult. There are CRT indices which involve examining differential performance by masters and nonmasters (Hudson and Lynch, 1984;Brown, 1991).…”

“…The suggested methods for CRT test item selection utilizing IRT generally involve the use of IIFs. The use of IIFs as a standard against which to judge CRT item discrimination measures has proved useful in educational measurement (van der Linden, 1981;Harwell, 1983;Harris and Subkoviak, 1986;Shannon and Cliver, 1987). Because it is sensitive to the difficulty level associated with a selected cut score or standard, the IIF can serve as a useful anchor in examining the utility of various CRT indices.…”

Evaluation of Learning Outcomes Through Multiple Choice Pre- and Post-Training Assessments

“…As a method to minimize guessing, a number of authors have suggested adding an 'I don't know' (IDK) option to the true-false answer choices in MCQ assessments (Sanderson, 1973;Newble et. al., 1979;Courtenay & Weidemann, 1985;Hammond, McIndoe, Sansome, & Spargo, 1998;van Mameren & van der Vleuten, 1999;Spears & Wilson, 2010). For example, van Mameren and van der Vleuten (1999) suggested the formula (total # correct answers)-(total # incorrect answers) for the score, with no penalty for IDK answers.…”

“…A majority of the research on the IDK option has been conducted in the context of True or False (T/F) type questions (Sanderson, 1973;Newble et al, 1979;Courtenay & Weidemann, 1985;Hammond et al, 1998;van Mameren & van der Vleuten, 1999;Spears & Wilson, 2010). The work by Newble et al (1979) included 19 multiple choice items in a post-test only assessment with an IDK option, but a gap in knowledge still exists on how the IDK option applies to MCQ with more than 2 options in a pre-/post-test assessment model.…”

Objective tests and latent trait theories

“…This description is necessarily over-simplified and obviously ignores many important related issues such as atypical students (Levine and Rubin [4]), longish tests (Wainer et al [5]), test-item bias (Shepard et al [6]), item context (Leary and Dorans [7]), generalizability (Izard [8]) and validation issues (van der Linden [9]). Nevertheless, it is hoped that the outline is sufficient for those who would like to know more to read the exposition of Choppin [10], Willmott and Fowles [11], and the detailed explanation of Wright and Stone.…”