Latent Class Scaling Analysis

ETS Research Report Series

2008

Rater behavior in essay grading can be viewed as a signal-detection task, in that raters attempt to discriminate between latent classes of essays, with the latent classes being defined by a scoring rubric. The present report examines basic aspects of an approach to constructed-response (CR) scoring via a latent-class signal-detection model. The model provides a psychological framework for CR scoring and includes rater parameters with a clear cognitive basis. Simulations are used to examine how well rater parameters and latent-class sizes are recovered as well as the accuracy of classification. The relation of rater parameters to agreement statistics and classification accuracy is examined. The effects of using a balanced, incomplete block design are compared to those for a fully crossed design. The model is applied to several ETS datasets.

Section: Resultsmentioning

confidence: 99%

Studies of a Latent‐class Signal‐detection Model for Constructed‐response Scoring

ETS Research Report Series

2008

“…The model is completed by embedding the SDT model into a restricted latent class model (see Dayton, 1998;DeCarlo, 2002),…”

Section: The Mixture 1 Sdt Modelmentioning

confidence: 99%

The mirror effect and mixture signal detection theory.

Journal of Experimental Psychology: Learning, Memory, and Cogni

2007

The mirror effect for word frequency refers to the finding that low-frequency words have higher hit rates and lower false alarm rates than high-frequency words. This result is typically interpreted in terms of conventional signal detection theory (SDT), in which case it indicates that the order of the underlying old item distributions mirrors the order of the new item distributions. However, when viewed in terms of a mixture version of SDT, the order of hits and false alarms does not necessarily imply the same order in the underlying distributions because of possible effects of mixing. A reversal in underlying distributions did not appear for fits of mixture SDT models to data from 4 experiments.Keywords: mirror effect, signal detection theory, mixture modelsThe mirror effect (Glanzer & Adams, 1985, 1990) is a general phenomenon in recognition memory that has been studied with a number of variables, most often with word frequency. In that case, the mirror effect refers to the finding that low-frequency words have higher hit rates and lower false alarm rates than highfrequency words. This result is usually interpreted in terms of (unequal variance) signal detection theory (SDT), in which case it implies that the order of the underlying old item distributions mirrors the order of the new item distributions, just like the observed hit and false alarm rates. Figure 1 illustrates the interpretation of the mirror effect in terms of conventional SDT; note that the relative spacing and relative variances of the normal distributions shown in the figure are consistent with those found below (with the exception that the distributions in Figure 1 are spaced further apart for visual clarity). The figure shows that the mirror effect consists of two fundamental aspects (Glanzer & Adams, 1985). The first is that the distribution for high-frequency new (HN) words is to the right of that for low-frequency new (LN) words. This suggests that HN words are more familiar than LN words, as expected because highfrequency words are more commonly encountered. A second aspect of the mirror effect is that the distribution for low-frequency old (LO) words is to the right of that for high-frequency old (HO) words. Thus, when one studies the words, the less familiar lowfrequency words are strengthened more than the high-frequency words, with the result that the order of old item distributions mirrors that for new item distributions (i.e., the order is reversed); note that the SDT distance parameter dЈ is therefore larger for low-frequency words than for high-frequency words.It should be recognized that in order for the mirror effect to hold, both aspects noted above must be present-that is, the HN word distribution must be to the right of the LN distribution and the LO distribution must be to the right of the HO distribution. Simply finding a larger value of dЈ for low-frequency words is not sufficient to demonstrate the mirror effect, as was noted by Glanzer and Adams (1990), who gave examples in which the mirror effect does not appear. In on...

“…Note that it is important to run the program repeatedly with different starting values, because local maxima are sometimes encountered; see Dayton (1998) for a general discussion of various aspects of latent class analysis and DeCarlo (in press) for discussion of a latent class extension of signal detection theory (SDT) and additional sample programs.…”

Section: Appendixmentioning

confidence: 99%

Signal detection theory with finite mixture distributions: Theoretical developments with applications to recognition memory.

2002

Psychological Review

131

203

An extension of signal detection theory (SDT) that incorporates mixtures of the underlying distributions is presented. The mixtures can be motivated by the idea that a presentation of a signal shifts the location of an underlying distribution only if the observer is attending to the signal; otherwise, the distribution is not shifted or is only partially shifted. Thus, trials with a signal presentation consist of a mixture of 2 (or more) latent classes of trials. Mixture SDT provides a general theoretical framework that offers a new perspective on a number of findings. For example, mixture SDT offers an alternative to the unequal variance signal detection model; it can also account for nonlinear normal receiver operating characteristic curves, as found in recent research.Signal detection theory (SDT) provides a theoretical framework that has been quite useful in psychology and other fields (see Gescheider, 1997;Macmillan & Creelman, 1991;Swets, 1996). A basic idea of SDT is that decisions about the presence or absence of an event are based on decision criteria and on perceptions of the event or nonevent, with the perceptions being represented by probability distributions on an underlying continuum. Thus, in its simplest form, the theory considers two basic aspects of detection-the underlying representations, which are interpreted as psychological distributions of some sort (e.g., of perception or familiarity), and a decision aspect, which involves the use of decision criteria to arrive at a response.The present article extends SDT by viewing detection as consisting of an additional process. The result is a simple and psychologically meaningful extension of SDT that can be applied to any area of research where SDT has been applied. The approach is illustrated with applications to research on recognition memory, where the additional process can be interpreted as attention. In particular, the basic idea is that presentation of a signal shifts the location of the underlying distribution only if the observer is attending to the signal; otherwise, the distribution is not shifted or is only partially shifted. As a result, trials with a signal presentation consist of a mixture of two (or more) latent classes of trials, which can be interpreted as being attended and nonattended trials (or as two different levels of processing of the stimuli). Apart from that, the theory is the same as in conventional SDT, in that the underlying representations are used together with response criteria to arrive at an observed response. I show, however, that this simple extension of SDT is quite powerful and can account for a variety of findings across several areas of research. For example, the mixture approach offers an alternative to the unequal variance signal detection model (Green & Swets, 1966), which has been the standard model for many years, and provides a different interpretation of normal receiver operating characteristic (ROC) curves with slopes less than unity; it can also account for nonlinear normal ROC curves, as found in r...