On the measurement of criterion noise in signal detection theory: Reply to Benjamin (2013).

Kellen, David; Klauer, Karl Christoph; Singmann, Henrik

doi:10.1037/a0033141

Cited by 5 publications

(7 citation statements)

References 34 publications

(67 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First and foremost, we agree that Ratcliff et al (2018) provided evidence that drift rate varies in some manner between trials in an experiment, which stands in contrast to recent neuroscientific proposals that drift rate remains identical across decisions in an experiment (e.g., Churchland, Kiani, & Shadlen, 2008; Ditterich, 2006a, 2006b; Drugowitsch, Moreno-Bote, Churchland, Shadlen, & Pouget, 2012; O’Connell, Shadlen, Wong-Lin, & Kelly, 2018). We also agree that understanding which sources of random variability are necessary to explain empirical data is an important question for all fields that use computational models (e.g., Bhatia & Loomes, 2017; Kellen, Klauer, & Singmann, 2012, 2013; Regenwetter & Robinson, 2017, 2019), which in the case of the diffusion model involves determining whether or not random between-trial variability parameters (e.g., Ratcliff, 1978; Ratcliff & Rouder, 1998; Ratcliff & Tuerlinckx, 2002) are (a) necessary for explaining empirical trends in choice and RT data, and (b) useful for improving our understanding of decision-making. However, we believe it is important to distinguish between the multiple types of between-trial variability, and that the lack of distinction in Ratcliff et al (2018) led to misleading conclusions.…”

supporting

confidence: 73%

Systematic and random sources of variability in perceptual decision-making: Comment on Ratcliff, Voskuilen, and McKoon (2018).

Evans¹,

Tillman²,

Wagenmakers³

2020

Psychological Review

View full text Add to dashboard Cite

A key assumption of models of human cognition is that there is variability in information processing. Evidence accumulation models (EAMs) commonly assume two broad variabilities in information processing: within-trial variability, which is thought to reflect moment-to-moment fluctuations in perceptual processes, and between-trial variability, which is thought to reflect variability in slower-changing processes like attention, or systematic variability between the stimuli on different trials. Recently, Ratcliff, Voskuilen, and McKoon (2018) claimed to "provide direct evidence that external noise is, in fact, required to explain the data from five simple two-choice decision tasks" (p. 33), suggesting that at least some portion of the between-trial variability in information processing is due to "noise". However, we argue that Ratcliff et al. (2018) failed to distinguish between two different potential sources of between-trial variability: random (i.e., "external noise") and systematic (e.g., item effects). Contrary to the claims of Ratcliff et al. (2018), we show that "external noise" is not required to explain their findings, as the same trends of data can be produced when only item effects are present. Furthermore, we contend that the concept of "noise" within cognitive models merely serves as a convenience parameter for sources of variability that we know exist, but are unable to account for. Therefore, we question the usefulness of experiments aimed at testing the general existence of "random" variability, and instead suggest that future research should attempt to replace the random variability terms within cognitive models with actual explanations of the process.

show abstract

supporting

confidence: 73%

Systematic and random sources of variability in perceptual decision-making: Comment on Ratcliff, Voskuilen, and McKoon (2018).

Evans¹,

Tillman²,

Wagenmakers³

2020

Psychological Review

View full text Add to dashboard Cite

show abstract

“…Moreover, many researchers now argue that decision criteria might even vary within a given participant (e.g., Benjamin, Diaz, & Wee, 2009; Mueller & Weidemann, 2008; cf. Kellen, Klauer, & Singmann, 2012, 2013). In other words, just as signal strength has a probabilistic distribution, so do decision criteria, even for within-subjects designs.…”

Section: Computational Evidence For Differential Filler Siphoningmentioning

confidence: 99%

Fair lineups are better than biased lineups and showups, but not because they increase underlying discriminability.

Smith¹,

Wells²,

Lindsay³

et al. 2017

Law and Human Behavior

109

View full text Add to dashboard Cite

Receiver Operating Characteristic (ROC) analysis has recently come in vogue for assessing the underlying discriminability and the applied utility of lineup procedures. Two primary assumptions underlie recommendations that ROC analysis be used to assess the applied utility of lineup procedures: (a) ROC analysis of lineups measures underlying discriminability, and (b) the procedure that produces superior underlying discriminability produces superior applied utility. These same assumptions underlie a recently derived diagnostic-feature detection theory, a theory of discriminability, intended to explain recent patterns observed in ROC comparisons of lineups. We demonstrate, however, that these assumptions are incorrect when ROC analysis is applied to lineups. We also demonstrate that a structural phenomenon of lineups, differential filler siphoning, and not the psychological phenomenon of diagnostic-feature detection, explains why lineups are superior to showups and why fair lineups are superior to biased lineups. In the process of our proofs, we show that computational simulations have assumed, unrealistically, that all witnesses share exactly the same decision criteria. When criterial variance is included in computational models, differential filler siphoning emerges. The result proves dissociation between ROC curves and underlying discriminability: Higher ROC curves for lineups than for showups and for fair than for biased lineups despite no increase in underlying discriminability. (PsycINFO Database Record

show abstract

“…It has become a pressing matter to address the issue of individual subject analysis. Scanning the current literature, the author found several such publications in the Journal of Psychological Review (Fific et al, 2010 ; Hills and Hertwig, 2012 ; Benjamin, 2013 ; Kellen et al, 2013b ; Turner et al, 2013 ), the leading edge in theoretical advances relevant to the problem of averaging data across subjects.…”

Section: Discussionmentioning

confidence: 99%

“…There is a rapidly increasing trend toward accounting for individual-specific cognitive operations in contrast to testing models based on universal cognitive operations. Accounting for individual differences is essential to assessing which model provides the best fit to experimental data (Broder and Schutz, 2009 ; Dube and Rotello, 2012 ; Kellen et al, 2013a , b ; Turner et al, 2013 ). Evidence for individual differences has been reported in judgment strategies (e.g., Hilbig, 2008 ; Regenwetter et al, 2009 ), and the analyses of individual data have been called for repeatedly when investigating fast and frugal heuristics (Gigerenzer and Brighton, 2009 ; Marewski et al, 2010 ).…”

Section: Introductionmentioning

confidence: 99%

Double jeopardy in inferring cognitive processes

Fifić

2014

Front. Psychol.

View full text Add to dashboard Cite

Inferences we make about underlying cognitive processes can be jeopardized in two ways due to problematic forms of aggregation. First, averaging across individuals is typically considered a very useful tool for removing random variability. The threat is that averaging across subjects leads to averaging across different cognitive strategies, thus harming our inferences. The second threat comes from the construction of inadequate research designs possessing a low diagnostic accuracy of cognitive processes. For that reason we introduced the systems factorial technology (SFT), which has primarily been designed to make inferences about underlying processing order (serial, parallel, coactive), stopping rule (terminating, exhaustive), and process dependency. SFT proposes that the minimal research design complexity to learn about n number of cognitive processes should be equal to 2n. In addition, SFT proposes that (a) each cognitive process should be controlled by a separate experimental factor, and (b) The saliency levels of all factors should be combined in a full factorial design. In the current study, the author cross combined the levels of jeopardies in a 2 × 2 analysis, leading to four different analysis conditions. The results indicate a decline in the diagnostic accuracy of inferences made about cognitive processes due to the presence of each jeopardy in isolation and when combined. The results warrant the development of more individual subject analyses and the utilization of full-factorial (SFT) experimental designs.

show abstract

On the measurement of criterion noise in signal detection theory: Reply to Benjamin (2013).

Cited by 5 publications

References 34 publications

Systematic and random sources of variability in perceptual decision-making: Comment on Ratcliff, Voskuilen, and McKoon (2018).

Systematic and random sources of variability in perceptual decision-making: Comment on Ratcliff, Voskuilen, and McKoon (2018).

Fair lineups are better than biased lineups and showups, but not because they increase underlying discriminability.

Double jeopardy in inferring cognitive processes

Contact Info

Product

Resources

About