The Hard–Easy Effect in Subjective Probability Calibration

Suantak, Liana; Bolger, Fergus; Ferrell, William R.

doi:10.1006/obhd.1996.0074

Cited by 136 publications

(75 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…quality of the evidence (for a similar argument see Ferrell & McGoey, 1980;Suantak et al, 1996). Overall accuracy.…”

Section: Accuracy Of Confidence Ratingsmentioning

confidence: 81%

“…This assumption is true for signal detection models of confidence (e.g., Budescu, Wallsten, & Au, 1997;Macmillan & Creelman, 2005;Suantak, Bolger, & Ferrell, 1996) as well as a majority of sequential sampling models of confidence (e.g., Heath, 1984;Link, 2003;Merkle & Van Zandt, 2006;Moreno-Bote, in press;Van Zandt, 2000b;Vickers, 1979Vickers, , 2001. We take a different course.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Two-stage dynamic signal detection: A theory of choice, decision time, and confidence.

Pleskac¹,

Busemeyer²

2010

Psychological Review

666

947

View full text Add to dashboard Cite

The 3 most often-used performance measures in the cognitive and decision sciences are choice, response or decision time, and confidence. We develop a random walk/diffusion theory-2-stage dynamic signal detection (2DSD) theory-that accounts for all 3 measures using a common underlying process. The model uses a drift diffusion process to account for choice and decision time. To estimate confidence, we assume that evidence continues to accumulate after the choice. Judges then interrupt the process to categorize the accumulated evidence into a confidence rating. The model explains all known interrelationships between the 3 indices of performance. Furthermore, the model also accounts for the distributions of each variable in both a perceptual and general knowledge task. The dynamic nature of the model also reveals the moderating effects of time pressure on the accuracy of choice and confidence. Finally, the model specifies the optimal solution for giving the fastest choice and confidence rating for a given level of choice and confidence accuracy. Judges are found to act in a manner consistent with the optimal solution when making confidence judgments.

show abstract

“…quality of the evidence (for a similar argument see Ferrell & McGoey, 1980;Suantak et al, 1996). Overall accuracy.…”

Section: Accuracy Of Confidence Ratingsmentioning

confidence: 81%

mentioning

confidence: 99%

Two-stage dynamic signal detection: A theory of choice, decision time, and confidence.

Pleskac¹,

Busemeyer²

2010

Psychological Review

666

947

View full text Add to dashboard Cite

show abstract

“…That is, when confidence is high, it's generally too high, and when it's low, it's generally too low-the typical pattern of miscalibration. Similarly, when accuracy is low for a set of questions it's likely to be lower than the available information would lead one to expect, and when accuracy is high it's likely to be higher than one would expect (Dawes & Mulford, 1996;Klayman et al, 1999;Suantak, Bolger, & Ferrell, 1996). This leads to overconfidence for "hard" question sets, and occasionally even underconfidence for "easy" question sets -the hardeasy effect.…”

Section: Overconfidence In Interval Estimatesmentioning

confidence: 99%

Overconfidence in Interval Estimates.

Soll

Klayman

2004

Journal of Experimental Psychology: Learning, Memory, and Cogni

348

345

View full text Add to dashboard Cite

Judges were asked to make numerical estimates (e.g., “In what year was the first flight of a hot air balloon?”). Judges provided high and low estimates such that they were X% sure that the correct answer lay between them. They exhibited substantial overconfidence: The correct answer fell inside their intervals much less than X% of the time. This contrasts with choices between 2 possible answers to a question, which showed much less overconfidence. The authors show that overconfidence in interval estimates can result from variability in setting interval widths. However, the main cause is that subjective intervals are systematically too narrow given the accuracy of one's information—sometimes only 40% as large as necessary to be well calibrated. The degree of overconfidence varies greatly depending on how intervals are elicited. There are also substantial differences among domains and between male and female judges. The authors discuss the possible psychological mechanisms underlying this pattern of findings.

show abstract

“…Confidence has been studied in humans using a variety of techniques (Balakrishnan 1999;García-Pérez andAlcalá-Quintana 2011, 2012;Garcia-Perez and Peli 2014;Hsu and Doble 2015;Okamoto 2012;Sawides et al 2013) including probability judgments (Baranski and Petrusic 1994;Björkman 1994;Ferrell 1995;Ferrell and McGoey 1980;Juslin et al 1998;Keren 1991;Lichtenstein et al 1982;Stankov 1998;Stankov et al 2012;Suantak et al 1996). In fact, as noted in a recent review (Grimaldi et al 2015), confidence probability judgments (i.e., confidence ratings provided using a nearly continuous scale between 0 and 100% or 50 and 100%) provide the most common assessment of confidence.…”

mentioning

confidence: 99%

“…One common use of confidence recordings is in "confidence calibration" studies where confidence is compared with actual performance, where a data set may be classified as "wellcalibrated" or classified as indicative of "overconfidence" or "underconfidence" (Baranski and Petrusic 1994;Björkman 1994;Ferrell and McGoey 1980;Juslin et al 1998;Keren 1991;Lichtenstein et al 1982;Stankov 1998;Suantak et al 1996). Specifically, imagine that a subject reported 90% confidence that a given motion was rightward for 10 separate trials at a given stimulus level.…”

mentioning

confidence: 99%

A quantitative confidence signal detection model: 1. Fitting psychometric functions

Merfeld

2016

Journal of Neurophysiology

View full text Add to dashboard Cite

Yi Y, Merfeld DM. A quantitative confidence signal detection model: 1. Fitting psychometric functions. J Neurophysiol 115: 1932J Neurophysiol 115: -1945J Neurophysiol 115: , 2016. First published January 13, 2016; doi:10.1152/jn.00318.2015.-Perceptual thresholds are commonly assayed in the laboratory and clinic. When precision and accuracy are required, thresholds are quantified by fitting a psychometric function to forced-choice data. The primary shortcoming of this approach is that it typically requires 100 trials or more to yield accurate (i.e., small bias) and precise (i.e., small variance) psychometric parameter estimates. We show that confidence probability judgments combined with a model of confidence can yield psychometric parameter estimates that are markedly more precise and/or markedly more efficient than conventional methods. Specifically, both human data and simulations show that including confidence probability judgments for just 20 trials can yield psychometric parameter estimates that match the precision of those obtained from 100 trials using conventional analyses. Such an efficiency advantage would be especially beneficial for tasks (e.g., taste, smell, and vestibular assays) that require more than a few seconds for each trial, but this potential benefit could accrue for many other tasks.thresholds; decision-making; confidence rating; confidence calibration MEASURING THRESHOLDS IS PROBABLY the most common psychophysical procedure in use today; applications range from experimental psychology to neuroscience to economics to engineering. Fitting psychometric functions using categorical data analyses (Agresti 1996) that describe the relationship between a stimulus characteristic (e.g., amplitude) and a subject's forced-choice categorical responses provides a standard approach used to estimate thresholds (Green and Swets 1966; Macmillan and Creelman 2005).A recent comprehensive analysis (Garcia-Perez and AlcalaQuintana 2005) concluded that only maximum likelihood methods should be used when accuracy and precision of psychometric function fit parameters is important and, furthermore, showed that more than 100 forced-choice trials are generally required to yield acceptable fit parameter estimates. Because such perceptual threshold tests are common and because many trials are needed to yield accurate and precise psychometric fits, studies spanning 50 yr (Garcia

show abstract

The Hard–Easy Effect in Subjective Probability Calibration

Cited by 136 publications

References 0 publications

Two-stage dynamic signal detection: A theory of choice, decision time, and confidence.

Two-stage dynamic signal detection: A theory of choice, decision time, and confidence.

Overconfidence in Interval Estimates.

A quantitative confidence signal detection model: 1. Fitting psychometric functions

Contact Info

Product

Resources

About