Evaluating the Evidential Value of Empirically Supported Psychological Treatments (ESTs): A Meta-Scientific Review

Abstract

Empirically supported treatments (or therapies; ESTs) are the gold standard in therapeutic interventions for psychopathology. Based on a set of methodological and statistical criteria, the APA has assigned particular treatment-diagnosis combinations EST status and has further rated their empirical support as Strong, Modest, and/or Controversial. Emerging concerns about the replicability of research findings in clinical psychology highlight the need to critically examine the evidential value of EST research. We therefore conducted a meta-scientific review of the EST literature, using clinical trials reported in an existing online APA database of ESTs, and a set of novel evidential value metrics (i.e., rates of misreported statistics, statistical power, R-Index, and Bayes Factors). Our analyses indicated that power and replicability estimates were concerningly low across almost all ESTs, and individually, some ESTs scored poorly across multiple metrics, with Strong ESTs failing to continuously outperform their Modest counterparts. Lastly, we found evidence of improvements over time in statistical power within the EST literature, but not for the strength of evidence of EST efficacy. We describe the implications of our findings for practicing psychotherapists and offer recommendations for improving the evidential value of EST research moving forward.

The efficacy of psychotherapies and pharmacotherapies for mental disorders in adults: an umbrella review and meta‐analytic evaluation of recent meta‐analyses

“…For psychotherapy, no recent meta‐analysis fulfilled the inclusion criteria. The quality of studies was found to be low 137 . For pharmacotherapy, the largest meta‐analysis reported a small SMD in comparison with placebo (0.27, see Figure 2) 90 .…”

Introduction to the special issue on the efficacy and effectiveness of couple and family interventions: Evidence base update 2010–2019

“…Additionally, we asked authors to apply criteria adapted from Southam‐Gerow and Prinstein (2014) to evaluate the evidence base in this special issue. Concerns have been raised regarding the use of such evidence base guidelines because they rely primarily on statistical significance and only require two positive results for favorable evaluation; this is particularly relevant as the dangers of false‐positive findings are increasingly being recognized (e.g., Sakuluk et al, 2019; Tackett et al, 2017). We encourage couple and family intervention researchers to more thoroughly consider issues of replicability (e.g., Tackett et al, 2017) and potential solutions to these problems (e.g., Nosek et al, 2018) as we look to the next decade of research.…”

Comparative clinical trials in psychotherapy: Have large effects been replicated?

“…At the same time, our unrestricted search suggests that replication attempts are rare regardless of trial effect or sample size. A recent investigation (Sakaluk et al, 2019) supports this conclusion, reporting 'weak' evidence for replicability across empirically supported psychotherapies. Unfortunately, a limitation of this study was its classification system, which grouped heterogeneous treatments and treatment comparisons.…”