Decision noise: An explanation for observed violations of signal detection theory

Abstract

Signal detection theory (SDT) has become a prominent and useful tool for analyzing performance across a wide spectrum of psychological tasks, from single-cell recordings and perceptual discrimination to high-level categorization, medical decision making, and memory tasks. The utility of SDT comes from its clear and simple account of how detection or classification performance can be translated into psychological quantities, such as sensitivity and bias. Whether its use is appropriate for a specific application depends on a number of underlying assumptions, and even though these assumptions are rarely tested, SDT has proved useful enough that it is considered one of the great successes of cognitive psychology. Yet, SDT has also undergone criticism, which began to emerge when this theory was relatively young. Criticisms of SDTSDT assumes that percepts are noisy and give rise to overlapping perceptual distributions for signal and noise trials. In order to distinguish between signal and noise trials, the observer uses a decision criterion to classify the percepts. Signal responses are "hits" when they are correct and "false alarms" when they are incorrect; similarly, noise responses can be classified as "correct rejections" and "misses." Many criticisms of SDT have centered on how the observer places a decision criterion during a detection or classification task, and whether a deterministic criterion is used at all (see, e.g., Dorfman & Biderman, 1971;Dorfman, Saslow, & Simpson, 1975;Kac, 1969;Kubovy & Healy, 1977;Larkin, 1971).Clearly, when initially performing a signal detection task, 1 an observer may be unable to estimate stimulus distributions and payoff values accurately; thus, one might expect the placement of a decision criterion to improve with experience, approaching a static optimal criterion. Yet, some results suggest that even with extensive practice, responses can be suboptimal: There are numerous demonstrations of human probability micromatching in signal detection tasks (see, e.g., Dusoir, 1974;Lee, 1963;Thomas, 1973Thomas, , 1975 and other demonstrations that static decision criteria are not typically used (e.g., Healy & Kubovy, 1981;Lee & Janke, 1964;Lee & Zentall, 1966;Treisman & Williams, 1984). Despite the fact that models accounting for these dynamics are based on a fairly reasonable assumption (i.e., that the decision criterion should improve with experience), they have not enjoyed the success of classic SDT-probably because they add layers of complexity to the theory that are not easily accommodated or validated. Given that even the basic assumptions required by SDT are rarely tested, it is perhaps not surprising that tests of these additional factors happen even less frequently. 465Copyright 2008 Psychonomic Society, Inc. THEORETICAL AND REVIEW ARTICLESDecision noise: An explanation for observed violations of signal detection theory SHANE T. MUELLERIndiana University, Bloomington, Indiana AND CHRISTOPH T. WEIDEMANN University of Pennsylvania, Philadelphia, PennsylvaniaIn signal detection the...

Decision criteria do not shift: Commentary on Mueller and Weidemann (2008)

“…Three attempts to account for these empirical phenomena, without giving up on the central concepts of SDT, have been published: Treisman (2002), Kornbrot (2006), and Mueller and Weidemann (2008). Treisman's and Kornbrot's arguments were inadequate, in our view, for various reasons, the most obvious being that they did not show that any kind of signal detection model could actually fit the data we reported.…”

“…Mueller and Weidemann (2008) propose that these unexpected phenomena are not due to problems with the decision-criterion construct but are instead due to two compounded effects: instability of the decision criterion across trials, and even greater instability in the flanking criteria that determine which confidence rating will be reported. There Mueller and Weidemann's experiment. In an Addendum, we respond to Weidemann and Mueller's (2008) reply to this Comment. In many areas of perception and memory research, experimental phenomena that appear to have significant implications about perception or memory processes per se could actually be due to the effects of response biases.…”

“…ratings data, the ratio of decision-criterion noise to ratingcriteria noise must always fall within the small range of values needed to cause the likelihood-ratio function to approach the value 1 as confidence decreases, instead of any other value. Because the specific value of this ratio has no special meaning in Mueller and Weidemann's (2008) theory, their model would fit the data without offering a meaningful explanation of them. Second, if the incorrect predictions of SDT are merely due to the manner in which participants make confidence ratings, as Muel ler and Weidemann suggested, there is no reason to expect the same problematic results to be observed when, instead of the rating responses, participants make only the yesor-no detection response, and response time (RT) is substituted for the degree of subjective confidence in order to compute likelihood-ratio functions.…”

“…However, more than the ROC curve, or even the decision criterion itself, it is the likelihood-ratio function that determines whether a decision maker is biased in the manner envisaged by SDT. Before considering Mueller and Weidemann's (2008) hypothesis about decision noise, therefore, we first explain how the likelihood-ratio function and the notion of response bias are connected. We then show how ratings and RT data can be used to determine whether the likelihood-ratio function behaves in a manner consistent with SDT.…”

Toward a complete decision model of item and source recognition: A discrete-state approach

“…Thus, it could be argued that the pattern of bivariate correlations, the use of likelihood ratio bounds, and the addition of a noise distribution are distortions forced upon the signal detection framework to cope with a compression pattern that, in reality, largely reflects scale-usage phenomena. In line with this argument, the potentially distorting role of response processes is currently receiving increased attention in the signal detection literature (Benjamin, Diaz, & Wee, 2009;Mueller & Weidemann, 2008).…”