Psilocybin Therapy for Treatment Resistant Depression: Prediction of Clinical Outcome by Natural Language Processing

Abstract

Background: Therapeutic administration of psychedelic drugs has shown significant potential in historical accounts and in recent clinical trials in the treatment of depression and other mood disorders. A recent randomized double-blind phase-IIb study demonstrated the safety and efficacy of COMP360, COMPASS Pathways’ proprietary synthetic formulation of psilocybin, in participants with treatment resistant depression. While promising, the treatment works for a portion of the population and early prediction of outcome is a key objective.Methods: Transcripts were made from audio recordings of the psychological support session between participant and therapist one day post COMP360 administration. A zero-shot machine learning classifier based on the BART large language model was used to compute two-dimensional sentiment (valence and arousal) for the participant and therapist from the transcript. These scores, combined with the Emotional Breakthrough Index (EBI) and treatment arm were used to predict treatment outcome as measured by MADRS scores. Code and data are available at https://github.com/compasspathways/Sentiment2DResults: Two multinomial logistic regression models were fit to predict responder status at week 3 and through week 12. Cross-validation of these models resulted in 85% and 88% accuracy and AUC values of 88% and 85%.Conclusions: A machine learning algorithm using NLP and EBI accurately predicts long term patient response, allowing rapid prognostication of personalized response to psilocybin treatment and insight into therapeutic model optimization. Further research is required to understand if language data from earlier stages in the therapeutic process hold similar predictive power.

From a Large Language Model to Three-Dimensional Sentiment

“…Furthermore, the model design allows one to "upgrade" the model with a more powerful zero-shot classifier as technology develops, or simply with one that is updated as language evolves. Initial exploration suggests that the three-dimensional model VAC here is a meaningful improvement over the two-dimensional VA model used in Dougherty et al [14], even when only considering the valence and arousal dimensions. For example, given two sentences that one would expect to differ primarily in their confidence dimension such as "I am enraged!"…”

“…Furthermore, its ratings are interpretable, consistent and reproducible. Like the model used in Dougherty et al [14], the VAC is mature enough to be meaningfully applied to real-world data and problems. Its use of a large language model means that the immense amount of data and on which the LLM was trained and the patterns learned in the process can be brought to bear on computing sentiment in a way that is sensitive to nuances of language.…”

“…Note that the model architecture a generalization to three dimensions of the twodimensional sentiment model used in Dougherty et al [14]; notable modifications include the addition of an anchor at the origin, and the use of anchor lists of multiple classes.…”

“…The lists in §3.4 were compiled by exploring candidate words found using the term frequency-inverse document frequency (TF-IDF) technique applied to a collection of pre-scored sentences. Using the two-dimensional model was described in Dougherty et al [14], and a one-dimensional confidence model was developed internally at Compass Pathways [24], we scored a large collection of sentences that were spoken in real therapy sessions. Once scored, we collected large numbers of sentences for each affective state corresponding to our nine anchor points.…”

“…Several attempts at building such automated dimensional sentiment models have been reported [10,11,12,13], however these are trained directly and only on human sentiment rating datasets (see Mohammad [13] for a comprehensive review). A notable exception is the twodimensional valence-arousal (VA) sentiment model of Dougherty et al [14] which leverages large language models (LLMs) via zero-shot learning and thus successfully brings to bear the deep language patterns understood by LLMs as a result of their immense amount of training data. Moreover, the Dougherty et al VA model has proven useful in the real world by helping predict long-term therapeutic outcomes.…”