The Confidence Database

Abstract

Understanding how people rate their confidence is critical for characterizing a wide range of perceptual, memory, motor, and cognitive processes. However, as in many other fields, progress has been slowed by the difficulty of collecting new data and the unavailability of existing data. To address this issue, we created a large database of confidence studies spanning a broad set of paradigms, participant populations, and fields of study. The data from each study are structured in a common, easy-to-use format that can be easily imported and analyzed in multiple software packages. Each dataset is further accompanied by an explanation regarding the nature of the collected data. At the time of publication, the Confidence Database (available at osf.io/s46pr) contained 145 datasets with data from over 8,700 participants and almost 4 million trials. The database will remain open for new submissions indefinitely and is expected to continue to grow. We show the usefulness of this large collection of datasets in four different analyses that provide precise estimation for several foundational confidence-related effects and lead to new findings that depend on the availability of large quantity of data. This Confidence Database will continue to enable new discoveries and can serve as a blueprint for similar databases in related fields.

Limited metacognitive access to one’s own facial expressions

“…Because confidence is known to correlate negatively with response times 75,76 (RT), we first explored a potential relationship between the two and asked whether RTs could have served as a proxy for performance. We ran a Bayesian linear regression model of participants' confidence ratings including the RT as a fixed effect and random intercepts for participant and facial expression, as well as a per-participant random slope for RT.…”

Confidence as a priority signal

“…If that 26 choice had been correct, the participant received 1 €, and 0 € otherwise. This payoff method 27 ensured participants kept a stable performance and did not relax their attention or 28 automatized their responses during some trials. 29 30…”

“…= 0.001, p = 0.022) 24 and in expected performance (p = 0.180) between the two tasks were included as predictors 25 in the same regression. Second, we compared this regression model with a simpler one that 26 only included the difference between tasks in expected performance and in actual accuracy, 27 and we found that the model including confidence explained the data significantly better than 28 the model without confidence (log-likelihood: full model = -2266.989, no confidence model = 29 -2529.497; χ²(1) = 525.015, p < 0.001). Third, in a different but similar approach, we could 30 predict priority from confidence (p < 0.001) in a regression in which we included as an offset 31 the prediction of priority by expected performance and actual accuracy.…”

“…In other circumstances, however, 27 we are free to decide in which order to complete our tasks. In such unconstrained situations, 28 humans may not perform their tasks in a random order, and they might instead exhibit some 29 systematic preference for doing one task before another. Here, we suggest that confidence 30 may play a role in this prioritization problem.…”

“…Then, we let them 27 choose the order in which they would face the same two tasks in a subsequent test phase. 28 In other words, unlike in our previous experiments where we measured priority from the 29 ordering of the two responses, in this new design we could measure priority as the choice of 30 engaging in one task before the other, very much like in the to-do list example. 31 32…”