How experimental procedures influence estimates of metacognitive ability

Abstract

It is becoming widely appreciated that higher stimulus sensitivity trivially increases estimates of metacognitive sensitivity. Therefore, meaningful comparisons of metacognitive ability across conditions and observers necessitates equating stimulus sensitivity. To achieve this, one common approach is to use a continuous staircase that runs throughout the duration of the experiment under the assumption that this procedure has no influence on the estimated metacognitive ability. Here we critically examine this assumption. Using previously published data, we find that, compared to using a single level of stimulus contrast, staircase techniques lead to inflated estimates of metacognitive ability across a wide variety of measures including area under the type 2 ROC curve, the confidence-accuracy correlation phi, meta-d’, meta-d’/d’, and meta-d’–d’. Further, this metacognitive inflation correlates with the degree of stimulus variability experienced by each subject. Finally, we show that the degree of stimulus variability in a staircase procedure may itself be driven by individual differences in metacognitive ability. These results suggest that studies using a staircase approach are likely to report inflated estimates of metacognitive ability. Further, we argue that similar inflation likely occurs in the presence of variability in task difficulty caused by other factors such as fluctuations in alertness or gradual improvement on the task. We offer practical solutions to these issues, both in the design and analysis of metacognition experiments.

Response-Related Signals Increase Confidence But Not Metacognitive Performance

“…It has been recently shown that the variability in the stimuli presented may lead to inflated estimates of metacognitive performance (Rahnev and Fleming, 2019). To assess whether this was a problem in our data, we ran two separate analyses.…”

“…Another limitation is the use of adaptive staircase procedures throughout the experiment. While maintaining task difficulty constant across trials, conditions, and participants is important to finely estimate metacognitive performance (Rahnev and Fleming, 2019), it also may hinder the relevance of sensorimotor signals as informative cues regarding the difficulty with which a decision was made (Kiani et al, 2014). Thus, a possibility is that sensorimotor signals are more potent cues for confidence estimates under fluctuating task difficulty.…”

Resilience of perceptual metacognition in a dual-task paradigm

“…A recent study (Rahnev & Fleming, 2019) showed that, by exposing observers to different levels of a stimulus, staircase methods produce inflated measures of metacognition when compared to constant stimuli methodologies. As we used separate staircases in our two experimental conditions, if stimulus variability was to be higher in the dual-task condition, this could potentially inflate participants' metacognitive efficiency compared to the single-task condition, thus providing a simple explanation for our result.…”

Response-related signals increase confidence but not metacognitive performance

“…Another limitation is the use of adaptive staircase procedures throughout the experiment. While maintaining task difficulty constant across trials, conditions, and participants is important to finely estimate metacognitive performance (Rahnev & Fleming, 2019), it also may hinder the relevance of sensorimotor signals as informative cues regarding the difficulty with which a decision was made (Kiani, Corthell, & Shadlen, 2014). Thus, a possibility is that sensorimotor signals are more potent cues for confidence estimates under fluctuating task difficulty.…”