Confidence as a noisy decision reliability estimate

Abstract

Decisions vary in difficulty. Humans know this and typically report more confidence in easy than in difficult decisions. However, confidence reports do not perfectly track decision accuracy, but also reflect response biases and difficulty misjudgments. To isolate the quality of confidence reports, we developed a model of the decision-making process underlying choice-confidence data. In this model, confidence reflects a subject's estimate of the reliability of their decision. The quality of this estimate is limited by the subject's uncertainty about the uncertainty of the variable that informs their decision ("meta-uncertainty"). This model provides an accurate account of choice-confidence data across a broad range of perceptual and cognitive tasks, revealing that meta-uncertainty varies across subjects, is stable over time, generalizes across some domains, and can be manipulated experimentally. The model offers a parsimonious explanation for the computational processes that underlie and constrain the sense of confidence.

A low-dimensional approximation of optimal confidence

“…Our reliability parameter, α, can be interpreted as an individual's estimate of the precision of evidence. This interpretation is similar to the recently proposed concept of "meta-uncertainty", which is described as "the subject's uncertainty about the uncertainty of the variable that informs their decision" (58). In both the LDC model and Boundy-Singer et al's CASANDRE model, one's estimate of evidence reliability weighs how evidence is used to compute confidence.…”

On why we lack confidence in signal-detection-based analyses of confidence

“…Indeed, some researchers have advocated for performing SDT-based analyses of confidence using calculations that assume non-normal experiential distributions (e.g. Medha & Dobromir, 2021; Miyoshi et al, 2022; Winter & Peters, 2022), or which have a differently shaped distribution to describe confidence (Boundy-Singer et al, 2022) or the variance of confidence criteria (Shekhar & Rahnev, 2021). The problem with this approach is that the criticisms we have made about the impact of erroneously assuming a normally-shaped experiential distribution could apply equally to erroneously assuming any other distribution shape, and we doubt the precise shape of distributions will ever be known in sufficient detail to preclude the possibility that results might differ from ground truth due to the wrong shape(s) having been assumed in calculations.…”