The Implicit Relational Assessment Procedure is not suitable for individual use

Abstract

A meta-analysis suggested that the Implicit Relational Assessment Procedure (IRAP) has potential “as a tool for clinical assessment”. Here I present evidence to the contrary. Using all published and unpublished file-drawer data available to me, I bootstrapping 95% Confidence Intervals for each IRAP D score. Results demonstrate that Confidence Intervals are extremely wide: regardless of the estimated D score, the data is equally compatible with a ‘true’ score lying anywhere in the range of very negative to very positive. The IRAP is therefore not currently suitable for individual level use or assessment in research or applied settings.

The Implicit Relational Assessment Procedure demonstrates poor internal consistency and test-retest reliability: A meta-analysis

“…This poor reliability has direct negative implications for statistical power in past and future studies. Elsewhere, recent research has also suggested that the IRAP demonstrates very poor individual level estimation (Hussey, 2020). As such, in its current form, the IRAP likely has limited use as an assessment tool in either research or applied settings.…”

“…On the one hand, in their meta-analysis of criterion validity of clinically relevant IRAP studies, Vahey et al (2015) argued that the IRAP has potential as a tool for clinical assessment. However, on the other hand, concerns have been expressed about the IRAP's low reliability (Golijani-Moghaddam et al, 2013;Greenwald & Lai, 2020) and poor individual-level estimation (Hussey, 2020). This tension between reliability and validity, and the importance of precise measurement more generally, has received renewed attention within psychology in recent years due to concerns about the replicability and validity of our findings (Flake & Fried, 2019).…”

A systematic review of Null Hypothesis Significance Testing, sample sizes, and statistical power in research using the Implicit Relational Assessment Procedure

“…Given recent calls for scrutiny of the replicability of behavioral research (e.g., Hantula, 2019;Task Given that statistical power across a literature is a key determinant of the replicability of the findings in that literature, these results paint a worrying picture for the replicability of IRAP research. These concerns add to concerns vocalized elsewhere about the IRAP's reliability (Hussey, 2020;Hussey & Drake, 2020a), a method factor that confounds several common analyses of IRAP data (Hussey & Drake, 2020b), and the fact that most IRAP studies come from a very narrow range of individuals and labs, potentially impacting the replicability and generalizability of claims (Hussey, 2022). Researchers should therefore interpret the results and conclusions of published IRAP research with some caution, and be cautious in choosing to employ the IRAP in their own work.…”

The individual-level precision of implicit measures

“…Fortunately, an alternative method can be used which does not rely on access to test-retest coefficients: namely, by bootstrapping confidence intervals around individuals' scores. Hussey (2020) proposed this method to estimate implicit measure confidence intervals, specifically around scores on the Implicit Relational Assessment Procedure (IRAP) across 18 different domains. Although the method of estimation was slightly different to the previous two studies, results were similarly poor, indicating that the IRAP does not provide precise individual-level estimates.…”