Cross-trial prediction in psychotherapy: External validation of the Personalized Advantage Index using machine learning in two Dutch randomized trials comparing CBT versus IPT for depression

Bronswijk, Suzanne C van; Bruijniks, Sanne J. E.; Lorenzo‐Luaces, Lorenzo; DeRubeis, Robert J.; Lemmens, Lotte H.J.M.; Peeters, Frenk; Huibers, Marcus J.H.

doi:10.1080/10503307.2020.1823029

Cited by 38 publications

(41 citation statements)

References 52 publications

(69 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Interestingly, empirical studies found effects that are comparable to the standard deviation of individual treatment effects. For example, after cross-validating a method that matches patients to either cognitive-behavioural therapy or interpersonal psychotherapy, Van Bronswijk et al 25 found that patients matched to their optimal treatment scored about 2 points or 9.1% lower on the BDI-II. Following a similar approach, Schwartz and colleagues 26 reported 14.2% lower symptom severity on the Brief Symptom Inventory for the subgroup of patients that were predicted to benefit most from either CBT or PDT, which corresponds to a small effect (Cohen's d = 0.33), which is corroborated by another study that found a difference of 2.6 points on the HDRS 27 .…”

Section: Discussionmentioning

confidence: 99%

Heterogeneity of treatment effects in trials on psychotherapy of depression

Kaiser¹,

Volkmann²,

Volkmann³

et al. 2020

Preprint

View full text Add to dashboard Cite

BackgroundPsychotherapy is an evidence-based treatment for depression, but its average effect is modest. Thus, identifying subgroups that respond especially well to psychotherapy is an important goal. This would allow maximizing the efficacy of interventions. However, the extent of treatment effect heterogeneity (TEH) has not yet been systematically investigated. A reliable, evidence-based estimate of this heterogeneity would allow a more accurate assessment of the potential effects of enhancement by personalization.MethodsUsing a large database of randomized-controlled trials on psychotherapy for depression in adults (k = 306), we performed a Bayesian variance ratio meta-regression. Based on the results, we determined the theoretically maximum possible extent of variability of individual outcomes. Exploratory subgroup analyses were conducted for different types of psychotherapy. We determined the extent of expected TEH given the evidence by employing an analytical approach.OutcomesWe found for the entire sample a 9% higher variance in the intervention groups compared to the control groups, indicating TEH in psychotherapy for depression. Depending on the depression scale used, this corresponds to a standard deviation of the individual treatment effect of 3-4 points. Subgroup analyses revealed that due to a large number of studies, the evidence for TEH is strongest in cognitive-behavioural therapy, while the largest TEH was observed in behavioural activation therapy.InterpretationOur results show that the treatment effect of psychotherapy for depression varies. This is a clear indication that patients benefit differently from different interventions. Clinical or statistical methods that select the optimal intervention for each patient are likely to increase the success of treatment.

show abstract

Section: Discussionmentioning

confidence: 99%

Heterogeneity of treatment effects in trials on psychotherapy of depression

Kaiser¹,

Volkmann²,

Volkmann³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Several studies since then have tried to predict which evi dencebased psychotherapy is most likely to benefit a specific patient 55,59 , including efforts to identify which of two (or more) psychotherapies may be most effective 60,61 , and whether a given patient is predicted to respond better to psychotherapy or medi cations 56 . A recent scoping review 62 identified a total of 44 studies that developed and tested a machine learning model in psycho therapy, but only seven of them reported on the feasibility of the tool.…”

Section: Psychotherapiesmentioning

confidence: 99%

“…PAIstyle approaches that calculate treatment by variable in teractions quickly lead to highdimensionality prediction analy ses that are prone to overfitting (or require very large sample sizes). Using data from two Dutch randomized trials, van Bron swijk et al 60 examined whether PAI models developed in one clinical trial dataset were able to successfully generalize to an independent dataset. Although the models performed statisti cally above chance in the trial used to train them, they did not generalize to the other clinical trial when predicting benefit for CBT versus interpersonal therapy (IPT) for depression.…”

Section: Psychotherapiesmentioning

confidence: 99%

“…Studies without external validation are at high risk of over confidence, as demonstrated by Van Bronswijk et al 60 , who de veloped and then tested a treatment selection model across two randomized controlled trials comparing CBT and IPT. They found that the estimated effect size for the benefit of receiving the modelrecommended treatment (generated through internal crossvalidation) shrunk by 77% when the model was tested us ing the second study's data (external validation).…”

Section: How Close We Have Come To Real-world Implementationmentioning

confidence: 99%

See 1 more Smart Citation

The promise of machine learning in predicting treatment outcomes in psychiatry

Bondar²,

et al. 2021

View full text Add to dashboard Cite

For many years, psychiatrists have tried to understand factors involved in response to medications or psychotherapies, in order to personalize their treatment choices. There is now a broad and growing interest in the idea that we can develop models to personalize treatment decisions using new statistical approaches from the field of machine learning and applying them to larger volumes of data. In this pursuit, there has been a paradigm shift away from experimental studies to confirm or refute specific hypotheses towards a focus on the overall explanatory power of a predictive model when tested on new, unseen datasets. In this paper, we review key studies using machine learning to predict treatment outcomes in psychiatry, ranging from medications and psychotherapies to digital interventions and neurobiological treatments. Next, we focus on some new sources of data that are being used for the development of predictive models based on machine learning, such as electronic health records, smartphone and social media data, and on the potential utility of data from genetics, electrophysiology, neuroimaging and cognitive testing. Finally, we discuss how far the field has come towards implementing prediction tools in real‐world clinical practice. Relatively few retrospective studies to‐date include appropriate external validation procedures, and there are even fewer prospective studies testing the clinical feasibility and effectiveness of predictive models. Applications of machine learning in psychiatry face some of the same ethical challenges posed by these techniques in other areas of medicine or computer science, which we discuss here. In short, machine learning is a nascent but important approach to improve the effectiveness of mental health care, and several prospective clinical studies suggest that it may be working already.

show abstract

“…In a subsequent publication (Bruijniks et al, 2020), we used machine-learning methods to conduct a similar analysis, still finding large effects of the PTR on those assigned to their optimal vs. non-optimal treatment (d =0.57). Nonetheless, when we applied that prediction model to data from another trial (van Bronswijk et al, 2020), the observed advantage of the multivariable moderator was small (d =0.16); the effect size shrank over 77% of its original size.…”

Section: Is It the Size Or Is It The Fit?mentioning

confidence: 99%

Personalized medicine and cognitive-behavioral therapies for depression: Small effects, big problems, and bigger data

Lorenzo‐Luaces¹,

Peipert²,

Jesús-Romero³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Cognitive-behavioral therapies (CBTs) are the most widely studied form psychotherapy for disorders like depression and anxiety. Nonetheless, there is heterogeneity in response to CBTs vs. other treatments. Researchers have become increasingly interested in using pre-treatment individual differences (i.e., moderators) to match patients to the most effective treatments for them. Several methods to combine multiple variables to create precision treatment rules (PTRs) that identify subgroups have been proposed. We review the rationale behind multivariable PTRs as well as studies that have used different PTRs. We identify conceptual and methodological issues in the literature. Multivariable treatment assignment is a promising avenue of research. Nonetheless, effect sizes appear to be small and most of the samples that have been used to study these questions have been grossly underpowered to detect small effects. We recommend researchers explore multivariable treatment selection strategies, particularly those resembling risk-stratification, in heterogeneous samples of patients undergoing low-intensity CBTs vs. realistic minimal controls.

show abstract

Cross-trial prediction in psychotherapy: External validation of the Personalized Advantage Index using machine learning in two Dutch randomized trials comparing CBT versus IPT for depression

Cited by 38 publications

References 52 publications

Heterogeneity of treatment effects in trials on psychotherapy of depression

Heterogeneity of treatment effects in trials on psychotherapy of depression

The promise of machine learning in predicting treatment outcomes in psychiatry

Personalized medicine and cognitive-behavioral therapies for depression: Small effects, big problems, and bigger data

Contact Info

Product

Resources

About