Passing Scores and Test Lengths for Domain-Referenced Measures

Abstract

In recent years much attention has been given to criterion-referenced measures which relate test performance to absolute standards rather than to the performance of others. Popham and Husek (1969) provide a readable account of the differences between such measures and the more traditional norm-referenced tests. The purpose of this paper is to synthesize some of the literature on establishing standards and determining the number of items needed in criterionreferenced measures.This paper is written from the following perspective. A domain (i.e., population) of dichotomously scorable test "items" is conceptualized. This population of items need not actually exist. What is important, though, is that it is described well enough so that a relatively high degree of agreement can be reached about which kinds of items are or are not members of the population. In practice, only a reasonably representative sample of items is required. 2 Theoretically, the items of a domain may be relatively heterogeneous in content, form, and difficulty. In practice, however, they should be measures of a limited number of skills and knowledges so This investigation was supported in part by the Instructional Objectives Exchange while the author was on leave from Cornell University. The reactions of Wells Hively, Paul Cieslak, and two referees to an earlier draft of this paper are gratefully acknowledged.2 In contrast to criterion, the term domain is reserved for the case where an item generation procedure is employed or a universe is postulated and the items used are considered to be a representative sample. Cronbach and his colleagues (1963, 1972) used the notion of a universe of test items in a theory of reliability. Osburn (1968) seems to have made the first in-print statement of the precise * definition of domain. One set of procedures for generating such a domain makes use of item forms. A thorough description of the techniques and examples may be found in Maxwell, Hively, Lundin, Rabehl, & Sension (1971) with further descriptions and extensions in Mussell (1969) and Rabehl (1971). Other item generating schemes have been proposed by Anderson (1972), Bormuth (1970), Guttman (see, e.g., Jordan, 1971), and to some extent by Scandura (see, e.g., Durnin & Scandura, 1971).A less formal way of conceptualizing a domain of items is to construct first an "amplified objective" (Popham, 1972) in which elements from content and method facets included in and excluded from the domain are identified. This procedure is being followed in the development of objectives-based tests in the basic skill areas published by the Instructional Objectives Exchange,

Decision Models for Use with Criterion-Referenced Tests

“…In this paper approaches to criterion-referenced testing based on a Neyman-Pearson approach (e.g., Fhan6r, 1974;Kriewall, 1972;Millman, 1973;Wilcox, 1976) have been disregarded. In these approaches an indifference zone instead of a true cutoff score is specified, and the cutoff score is found testing the hypothesis that the student is at the lower bound against the hypothesis that he/she is at the upper bound of this zone.…”

A Comparison of Domain Sampling Procedures for Test Construction

“…Hambleton (1984a) reviewed methods to determine the test length needed to reduce classification errors. Hambleton reviewed methods making use of the binomial model (Millman, 1972(Millman, , 1973, Bayesian methods (Novick & Lewis, 1974), an "indifference zone" (Wilcox, 1976), computer simulation methods (Eignor & Hambleton, 1979;Hambleton, Mills, & Simon, 1983) and item response theory (Bimbaum, 1968;Lord, 1980). The relationship between test length and estimates of the content domain 3.core is highly relevant to investigations of test content selection.…”

Validity

“…These include, as we have seen, specification of the construct domain of reference in terms of topical content, typical behaviors, and underlying processes. Also needed are test specifications regarding stimulus formats and response alternatives, -54 -administration conditions (such as examinee instructions or time limits), and criteria for item scoring (Cronbach, 1971;Millman, 1973 (APA,19(5) stipulates that an important task in test construction, as well as in test evaluation, is to determine the degree to which the format and response properties of the test items or tasks are germane to the specified domain. For example, a verbal paper-and-pencil test of machine operation would be less relevant to the specified performance domain than a hands-on manipulation test.…”