Confidence Intervals on Implicit Association Test Scores Are Really Rather Large

Abstract

Implicit association test scores are presented as point estimates. Unusually for a psychological measure, individual scores are not tested for statistical significance in standard practice. This study estimates individual confidence intervals for a large dataset of IAT scores. The intervals are large. Only half of scores are significantly different from zero. This result raises theoretical concerns about standard interpretations of the IAT.

Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

“…By contrast, generative modeling facilitates development of theory-informed models of behavior and takes advantage of all the information available, thereby improving accuracy of inference even when using smaller samples of behavioral (or neural) data. Our results question conclusions drawn from previous studies on reliability of various behavioral and neural measuresparticularly those that relied on two-stage approaches (e.g., Chen et al, 2015;Elliott et al, 2020;Enkavi et al, 2019;Gawronski et al, 2017;Hedge et al, 2017;Klein, 2020;Noble et al, 2019).…”

“…Such findings have since been replicated and extended to wide range of self-control tasks (Enkavi et al, 2019). Other behavioral tasks that are used widely throughout the behavioral sciences, including the IAT, also show similarly low test-retest correlations (r = .01-.72; average of r ≈ .4) across versions and timepoints (Gawronski et al, 2016;Klein, 2020). In the brain sciences, similarly low intraclass correlation coefficients were found in a meta-analysis of 90 experiments (mean ICC=0.397), and poor reliability of activity in regions of interest of brain regions across 11 common tasks used within the Human Connectome Project and the Dunedin Study (ICCs=0.067-0.485; Elliott et al, 2020).…”

The Implicit Relational Assessment Procedure is not suitable for individual use

“…It is also worth noting that similar analyses of data from another implicit measure, the Implicit Association Test, suggests that the IRAP's estimation precision is substantially worse than the IAT's (IRAP MAP 95% CI width = 1.33, IAT MAP 95% CI width = 0.75: see Hussey, 2020;Klein, 2020).…”

Implicit measures of beliefs

“…Implicit measures have become enveloped in this measurement crisis. Recent research has shown that the IAT does not conform to the measurement model which has formerly been specified for it; whereas it has previously been argued that the IAT captures a distinct latent variable to explicit measures Dovidio et al, 2001), research increasingly suggests that the IAT and explicit measures likely capture the same construct and differ primarily in their measurement precision (Schimmack, 2019a;Connor & Evers, 2020;Klein, 2020;Dalege & van der Maas, 2020). Likewise, recent evidence also suggests that the AMP only conforms to its assumed measurement model when awareness of the influence of primes in the task is high (i.e., the measure is either implicit or valid for participants, but not both simultaneously; .…”

“…At times, the poor test-retest reliability of implicit measures relative to explicit measures is often dismissed under the premise that the construct captured by implicit measures is nosier than that captured by explicit measures Brownstein et al, 2020;Dalege & van der Maas, 2020). Yet given the recent evidence that implicit measures likely capture the same construct as explicit measures (Schimmack, 2019a), combined with the poor estimation of individual implicit measure scores (Klein, 2020;Hussey, 2020), it seems much more likely that the measurement procedures themselves are simply nosier than their explicit equivalents. This, then, represents a fundamental barrier to the validity of these measures.…”