Evaluation Models for Criterion-Referenced Testing: Views regarding Mastery and Standard-Setting

Abstract

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org. This content downloaded from 161.45. In the dozen years since Glaser's (1963) seminal article on criterion-referencedtesting, the acceptance of the concept of mastery as an educational and, hence, evaluation goal has grown tremendously. A large number of articles have been published, curriculum programs have been devised that employ criterionreferenced testing, and yet writers still feel it necessary to define what a criterion-referenced test is. Furthermore, the various published definitions are by no means equivalent. One also observes a shift in the interests and background of the authors of papers over this period. In the Sixties, writers were primarily advocating the adoption of criterion-referenced testing from an educational or philosophical point of view in spite of the reservations of the classical measurement theorists, whereas in the Seventies a new generation of measurement specialists have begun to be involved, and the papers are much more mathematical. A number of mathematically-based techniques for deciding The comments of Professor Frederick B. Davis, University of Pennsylvania Graduate School of Education, and Dean D. Dax Taylor, Southern Illinois University School of Medicine, on an earlier draft of this paper are gratefully acknowledged. 133 This content downloaded from 161.45.205.103 on Mon, 23 Mar 2015 11:21:46 UTC All use subject to JSTOR Terms and Conditions REVIEW OF EDUCATIONAL RESEARCH on cutting scores and related issues such as test length have been published-for a particularly good review see Millman (1973)-but the authors' conceptualizationsof the educational task facing the learners has not always been clear. The purpose of this paper is to investigate the mastery models underpinning the techniques being proposed and to review the procedures suggested for setting the pass-fail point.The adoption of the criterion-referenced approach to evaluation very quickly raises two measurement issues that have relatively less importance in norm-referenced testing. These can be broadly stated as the issue of the definition of mastery, and the issue of a priori standards-closely intertwined but different problems. Rigorous exploration of these to date has been quite minimal. Perhaps the development of content aspects has been of greater urgency. However, this area is receiving increasing attention. Measurement specialists have turned their attention to criterion-referenced measurement, introducing the use of decision theory and Bayesian statistics.The evolving models are alike in requiring tight specification of content areas. Objectives are to be written in sufficient detail so that the form and content of the measurements are implicit in the educational objective...

The Evaluation of Competence

“…Ultimately, adoption of any set of standards reflects an evaluator's values, for which no quantitative procedure, however sophisticated, can serve as an adequate proxy (see Glass, 1978). Other reviews covering standard-setting procedures (Hambleton et al, 1978;Meskauskas, 1976;Millman, 1973) are more optimistic, although they fail to discuss the values issue explicitly. And given the disparaging remarks made earlier about norm-referenced standard-setting, which is nearly ubiquitous in modern competence evaluation, it seems that only modest guidance is available to those who seek a firm solution to the standards problem.…”

Coefficients for Tests from a Decision Theoretic Point of View

“…Much attention has been paid recently in psychometrics to dichotomous decision situations, particularly for mastery decisions in criterion-referenced measurement (Hambleton & Novick, 1973;Huyhn, 1976b;Meskauskas, 1976). Meskauskas (1976) distinguished between State and Continuum models for mastery decisions.…”

“…Meskauskas (1976) distinguished between State and Continuum models for mastery decisions. In State models the true score is considered an all or none variable, representing either mastery or non-mastery of the subject matter.…”

A Hierarchical IRT Model for Criterion-Referenced Measurement

“…In contrast with a continuum view on mastery, a state view on mastery (Meskauskas, 1976) distinguishes only two levels of competence, mastery and nonmastery. The state view assumes learning to be an all-or-none process, with mastery corresponding to the presence of ability.…”

“…mastery (Meskauskas, 1976) implied that students can be ordered along a continuously-distributed ability dimension. In line with the continuum view of mastery it can also be expected that the items corresponding to learning tasks of a certain domain can be ordered along the same continuum as a function of their difficulty.…”