Examining the robustness of the relationship between metacognitive efficiency and metacognitive bias

Abstract

We recently found a positive relationship between estimates of metacognitive efficiency and metacognitive bias. However, this relationship was only examined on a within-subject level and required binarizing the confidence scale, a technique that introduces methodological difficulties. Here we examined the robustness of the positive relationship between estimates of metacognitive efficiency and metacognitive bias by conducting two different types of analyses. First, we developed a new within-subject analysis technique where the original n-point confidence scale is transformed into two different (n-1)-point scales in a way that mimics a naturalistic change in confidence. Second, we examined the across-subject correlation between metacognitive efficiency and metacognitive bias. Importantly, for both types of analyses, we not only established the direction of the effect but also computed effect sizes. We applied both techniques to the data from three tasks from the Confidence Database (N > 400 in each). We found that both approaches revealed a small to medium positive relationship between metacognitive efficiency and metacognitive bias. These results demonstrate that the positive relationship between metacognitive efficiency and metacognitive bias is robust across several analysis techniques and datasets, and have important implications for future research.

Metacognitive improvement: Disentangling adaptive training from experimental confounds.

“…Future attempts to improve the quality of confidence ratings may be informed by recent findings regarding the definition of metacognitive noise (Shekhar & Rahnev, 2021a, 2021b; Xue et al, 2021), as a way to provide more information to participants regarding the qualitative nature of their metacognitive deficits. They could also rely on elicitation methods that encourage participants to report optimal confidence estimates, such as measuring participants’ willingness to trade a gamble based on the accuracy of their response against a lottery with known probabilities (Dienes & Seth, 2010; Massoni et al, 2014).…”

“…Despite pioneering experiments showing promising results (Adams & Adams, 1958;Sharp et al, 1988), to our knowledge, no recent remediation procedure based on feedback has been successful in improving the quality of confidence ratings (for a recent attempt based on singletrial feedback, see Rahnev, 2019, 2020). Future attempts to improve the quality of confidence ratings may be informed by recent findings regarding the definition of metacognitive noise (Shekhar & Rahnev, 2021a, 2021bXue et al, 2021), as a way to provide more information to participants regarding the qualitative nature of their metacognitive deficits. They could also rely on elicitation methods that encourage participants to report optimal confidence estimates, such as measuring participants' willingness to trade a gamble based on the accuracy of their response against a lottery with known probabilities (Dienes & Seth, 2010;Massoni et al, 2014).…”

“…Next, we assessed in an exploratory analysis which of the two confounds, incentives or confidence scale, was the main contributor of the confidence increase. This also relates to the question of metacognitive training, as Carpenter and colleagues reported that the increase in metacognitive efficiency was in fact mediated by the increase in metacognitive bias, and as an increase in confidence bias might result in an increase in metacognitive efficiency (Shekhar & Rahnev, 2021).…”

“…Indeed, precise confidence criteria might be easier to maintain across two levels on a full scale than four levels on a half-scale. In addition, as suggested in recent works (Shekhar & Rahnev, 2021a, 2021bXue et al, 2021) criteria for high confidence are noisier than criteria for low confidence and thus a merge of high confidence categories can artificially increase metacognitive efficiency.…”

Confidence as a noisy decision reliability estimate

“…It has long been known that humans and other animals can meaningfully introspect about the quality of their decisions and actions [5][6][7]31,48 . Quantifying this ability has remained a significant challenge, even for simple binary decision-making tasks 12,13,15,28,40,41 . The core problem is that observable choice-confidence data reflect metacognitive ability as well as task difficulty and response bias.…”