A Comparison of Natural Language Processing Methods for Automated Coding of Motivational Interviewing

Abstract

Motivational Interviewing (MI) is an efficacious treatment for substance use disorders and other problem behaviors. Studies on MI fidelity and mechanisms of change typically use human raters to code therapy sessions, which requires considerable time, training, and financial costs. Natural language processing techniques have recently been utilized for coding MI sessions using machine learning techniques, rather than human coders, and preliminary results have suggested these methods hold promise. The current study extends this previous work by introducing two natural language processing models for automatically coding MI sessions via computer. The two models differ in the way they semantically represent session content, utilizing either 1) simple discrete sentence features (DSF model) and 2) more complex recursive neural networks (RNN model). Utterance- and session-level predictions from these models were compared to ratings provided by human coders using a large sample of MI sessions (N = 341 sessions; 78,977 clinician and client talk turns) from 6 MI studies. Results show that the DSF model generally had slightly better performance compared to the RNN model. The DSF model had “good” or higher utterance-level agreement with human coders (Cohen’s kappa > 0.60) for open and closed questions, affirm, giving information, and follow/neutral (all therapist codes); considerably higher agreement was obtained for session-level indices, and many estimates were competitive with human-to-human agreement. However, there was poor agreement for client change talk, client sustain talk, and therapist MI-inconsistent behaviors. Natural language processing methods provide accurate representations of human derived behavioral codes and could offer substantial improvements to the efficiency and scale in which MI mechanisms of change research and fidelity monitoring are conducted.

A multivariate meta-analysis of motivational interviewing process and outcome.

“…Recent work has evidenced promising results via implementing technological advances in assessing MI adherence using automated speech recognition and natural language processing (for a review see Pace et al, 2016), which may increase our ability to collect and analyze data on MI process and outcome. Researchers have demonstrated the ability to use machine learning procedures to analyze large unstructured text corpora to identify semantic content (Steyvers & Griffiths, 2007), discriminate between different psychotherapies (Imel, Steyvers, & Atkins, 2014), automatically assign MI behavioral codes (Tanana, Hallgren, Imel, Atkins, & Srikumar, 2016), and assess MI empathy ratings on par with observer ratings (Xiao, Imel, Georgiou, Atkins, & Narayanan, 2015). These advances will allow us to scale up our ability to collect data, and improve our ability to understand MI process and outcome and answer key questions about how MI works.…”

A Deep Learning Approach to Modeling Empathy in Addiction Counseling

“…Tanana et al proposed using recursive neural networks paired with maximum entropy Markov models (MEMMs) to predict statements by clients about changing or maintaining their addictive behaviors [12]. Tanana et al also compared recursive neural network models to discrete sentence features and reported improved accuracy of predicting several utterance level behaviors when using the recursive neural network model [13].…”

Observing Dialogue in Therapy: Categorizing and Forecasting Behavioral Codes

“…The majority baseline illustrates the severity of the label imbalance problem. Xiao et al (2016), BiGRU generic , Can et al (2015) and Tanana et al (2016) We found that predicting using MLP(H n ) + MLP(v n ) performs better than just MLP(H n ).…”