How to measure metacognition

Abstract

The ability to recognize one's own successful cognitive processing, in e.g., perceptual or memory tasks, is often referred to as metacognition. How should we quantitatively measure such ability? Here we focus on a class of measures that assess the correspondence between trial-by-trial accuracy and one's own confidence. In general, for healthy subjects endowed with metacognitive sensitivity, when one is confident, one is more likely to be correct. Thus, the degree of association between accuracy and confidence can be taken as a quantitative measure of metacognition. However, many studies use a statistical correlation coefficient (e.g., Pearson's r) or its variant to assess this degree of association, and such measures are susceptible to undesirable influences from factors such as response biases. Here we review other measures based on signal detection theory and receiver operating characteristics (ROC) analysis that are “bias free,” and relate these quantities to the calibration and discrimination measures developed in the probability estimation literature. We go on to distinguish between the related concepts of metacognitive bias (a difference in subjective confidence despite basic task performance remaining constant), metacognitive sensitivity (how good one is at distinguishing between one's own correct and incorrect judgments) and metacognitive efficiency (a subject's level of metacognitive sensitivity given a certain level of task performance). Finally, we discuss how these three concepts pose interesting questions for the study of metacognition and conscious awareness.

Changes in ventromedial prefrontal and insular cortex support the development of metamemory from childhood into adolescence

“…not affected by metacognitive bias, or an individual's tendency to report high confidence (18). Longitudinal improvements in metamemory resolution were investigated using a model with an intercept centered at 9.6 y and linear change over time.…”

“…Metamemory resolution was calculated as the area under the AUCROC2 collapsed across all conditions. This measure captures how well observers can discriminate between correct and incorrect responses in their confidence judgments and is estimated by plotting varying levels of cumulative confidence pairs for of correct vs. incorrect responses and calculating the area under the AUCROC2 (18). Longitudinal analyses were conducted using multilevel models implemented in lme4 (39) (a model description is provided in SI Materials and Methods).…”

“…A high-resolution, whole-brain, magnetizationprepared rapid gradient-echo scan was acquired (SI Materials and Methods), and reconstruction was performed using Freesurfer 5.3 following standard procedures (36). Cortical thickness was extracted from the following ROIs based on the Destrieux atlas (37): vmPFC, defined as the orbital gyrus, orbital sulcus, and gyrus rectus; insula, defined as the anterior insula sulcus and short insular gyrus; APFC, defined as the middle frontal sulcus containing a majority of the coordinates reported in association with metacognitive ability in adults (9,18); and dACC, corresponding to the middle anterior cingulate gyrus and sulcus.…”

Regional White Matter Variation Associated with Domain-specific Metacognitive Accuracy

“…Type II sensitivity refers to an individualʼs ability to discriminate between their own correct and incorrect responses, whereas Type I sensitivity refers to an individualʼs ability to discriminate between stimulus alternatives (i.e., their capacity to distinguish old items from new items in a recognition memory task; Higham, Perfect, & Bruno, 2009;Clarke, Birdsall, & Tanner, 1959). SDT approaches can quantify metacognitive accuracy independent of an observerʼs decision strategy or cognitive ability on the primary task, which have been shown to confound other methods of estimating metacognitive ability (Fleming & Lau, 2014;Maniscalco & Lau, 2012).…”

Evidence for embodied predictive coding: the anterior insula coordinates cortical processing of tactile deviancy

“…Indeed, even with a high level of detection performance and equivalent difficulty ratings across conditions, we observed considerable variability in self-rated difficulty to detect stimulus changes, suggesting that some participants experienced switches more vividly than others. However, to better elucidate the role of attentional control or metacognitive report bias (Fleming and Lau 2014) in this result, it remains an important step for future research to manipulating stimulus probability in the context of a tactile detection task with intensity and perhaps confidence ratings on every trial. Future investigations will help to dissociate the role of executive function and metacognitive awareness in the response to embodied prediction error.…”