Some Problems in Construction and Evaluation of Criterion‐Referenced Tests

Abstract

Henrysson S. & Wedman I. 1974. Some Problems in Construction and Evaluation of Criterion-Referenced Tests. Scand. J. educ. Res.,18, 1-12. The interest in criterion-referenced tests has increased in a very obvious way in recent years. There are, however, many problems associated with a criterion-referenced testing procedure. In the first part of this study the effect of the aims on the construction of the tests, the homogeneity of the items and the cutting-scores for what should be regarded as satisfactory knowledge are dealt with. Concerning item-analysis of criterion-referenced tests several discrimination measures have been proposed. Some of these are discussed in the second part together with empirical data showing intercorrelations between them. The report ends with some comments on future work with criterion-referenced tests. INTRODUCTIONIn recent years great interest has been shown in the forming of achievement tests in order to make them reflect the contents of the instruction. A distinction made here is the one between norm-referenced and criterion-referenced tests (NRT and CRT respectively; Glaser 1963). In NRT the result of every individual is compared to the result of a norm group. In CRT, however, the individual results are compared to previously denned and specified standards.CRT has been created to show what the pupils can do and has its origin in the criticism recently directed at the using of tests in education constructed to rank individuals (see, e.g., Nitko 1970), which is one of the initial points at the construction of NRT.More specificially, the growing interest in CRT is a result of several trends: 1) Criticism of NRT for not giving information about actual standards -what is learned. 2) A growing interest in evaluation -both of individuals and systems. Evaluation is based upon established aims and the tests should tell if these aims are reached or not. A relative comparison between students or groups of students is not enough.

A Latent Trait Look at Pretest-Posttest Validation of Criterion-Referenced Test Items

“…This is correct, since it relates differences in item responses to an external criterion, namely the (objective-based) instruction which has taken place in between. A discussion of the rationale underlying Cox and Vargas' proposal can be found in Cox (1971), Ozenne (1971), Edmonston and Randall (1972), Henrysson and Wedman (1973), Roudabush (1973), Wedman (1973, 1974a, 1974b, and in the papers by Hambleton and colleagues referred to in the preceding section. Nicolich, Nicolich, and Raph (1974) have proposed to base criterion-referenced item analysis on an item change score.…”

“…There are also validity coefficients and procedures based on three item administrations, one prior to the instruction and two afterwards (Brennan, 1974;Brennan & Stolurow, 1971;Ivens, 1972), and, analogous to the pretest-posttest method for item validation, Ozenne (1971) and Barta, Ahn, and Gastright (1976) propose coefficients and procedures for validating tests. Several empirical studies have investigated the properties of the above validity coefficients, both in relation to one another and to traditional item selection statistics (Cox & Vargas, 1966;Crehan, 1974;Gorth & Hambleton, 1972;Henrysson & Wedman, 1973;Herbig, 1976;Kosecoff & Klein, 1974;Wedman, 1974b). On the whole, it appears that pretest-posttest item validation results in a more or less different item selection than does traditional statistics computed from the posttest data.…”

“…This a priori approach (Popham, 1971) can mostly be found in connection with the idea that item domains are best constructed using item forms (Hively, 1974;Hively et al 1973;Hively, Patterson, & Page, 1968;Osburn, 1968). Others (e.g., Henrysson & Wedman [1973]; Wedman [1974a]; Smith [Note 11) take a more empirical view and argue that item writing nearly always involves some subjectivity and that, to eliminate the consequences of this, analysis of empirical item data is indispensable. In this paper we share this empirical approach.…”

“…Moreover, unlike the preceding coefficients, Harris' index can assume values outside the intervals [0, 1] or [-1, +1] which the preceding coefficients are restricted to. Instead of using a pretest-posttest design, the simultaneous use of separate groups of experts and nonexperts is sometimes used (see, e.g.,Henrysson & Wedman, 1973) Millman (1974). refers to the former as preinstructionpostinstruction (the same individual tested twice) and the latter as instructed-uninstructed criterion groups (different individuals in both groups).…”

A Latent Trait Look at Pretest-Posttest Validation of Criterion-referenced Test Items

“…This is correct, since it relates differences in item responses to an external criterion, namely the (objective-based) instruction which has taken place in between. A discussion of the rationale underlying Cox and Vargas' proposal can be found in Cox (1971), Ozenne (1971), Edmonston and Randall (1972), Henrysson and Wedman (1973), Roudabush (1973), Wedman (1973Wedman ( , 1974aWedman ( , 1974b, and in the papers by Hambleton and colleagues referred to in the preceding section. Nicolich, Nicolich, and Raph (1974) have proposed to base criterion-referenced item analysis on an item change score.…”